2024-04-03T16:13:29.8717207Z Current runner version: '2.315.0' 2024-04-03T16:13:29.8726147Z Runner name: 'i-01dbe7bd2ee9fbb9d' 2024-04-03T16:13:29.8727183Z Runner group name: 'Default' 2024-04-03T16:13:29.8728380Z Machine name: 'ip-10-0-13-6' 2024-04-03T16:13:29.8734042Z ##[group]GITHUB_TOKEN Permissions 2024-04-03T16:13:29.8736685Z Actions: read 2024-04-03T16:13:29.8737326Z Checks: read 2024-04-03T16:13:29.8737976Z Contents: read 2024-04-03T16:13:29.8738587Z Deployments: read 2024-04-03T16:13:29.8739206Z Discussions: read 2024-04-03T16:13:29.8739794Z Issues: read 2024-04-03T16:13:29.8740372Z Metadata: read 2024-04-03T16:13:29.8740977Z Packages: read 2024-04-03T16:13:29.8741583Z Pages: read 2024-04-03T16:13:29.8742166Z PullRequests: read 2024-04-03T16:13:29.8742871Z RepositoryProjects: read 2024-04-03T16:13:29.8743568Z SecurityEvents: read 2024-04-03T16:13:29.8744208Z Statuses: read 2024-04-03T16:13:29.8744766Z ##[endgroup] 2024-04-03T16:13:29.8749776Z Secret source: None 2024-04-03T16:13:29.8750910Z Prepare workflow directory 2024-04-03T16:13:30.0090608Z Prepare all required actions 2024-04-03T16:13:30.0325891Z Getting action download info 2024-04-03T16:13:30.1995800Z Download action repository 'pytorch/test-infra@main' (SHA:f5543b9b2d7b7c7b8eccd6576b68fc0472b1fbde) 2024-04-03T16:13:30.6580149Z Download action repository 'pytorch/pytorch@main' (SHA:d895192e87f9a2c9ab3d6fab60f08855a08c3900) 2024-04-03T16:13:35.3017637Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-04-03T16:13:35.4598069Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-04-03T16:13:35.7820816Z Getting action download info 2024-04-03T16:13:35.9040408Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-04-03T16:13:36.1155793Z Getting action download info 2024-04-03T16:13:36.2081471Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-04-03T16:13:36.3758749Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/123154/merge (4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0) 2024-04-03T16:13:36.3762536Z ##[group] Inputs 2024-04-03T16:13:36.3763123Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-04-03T16:13:36.3765223Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-04-03T16:13:36.3767992Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:13:36.3769313Z sync-tag: 2024-04-03T16:13:36.3770498Z timeout-minutes: 360 2024-04-03T16:13:36.3770915Z use-gha: 2024-04-03T16:13:36.3771283Z dashboard-tag: 2024-04-03T16:13:36.3771699Z s3-bucket: gha-artifacts 2024-04-03T16:13:36.3772122Z aws-role-to-assume: 2024-04-03T16:13:36.3772543Z ##[endgroup] 2024-04-03T16:13:36.3773836Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:13:36.5039028Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-04-03T16:13:36.5039763Z with: 2024-04-03T16:13:36.5040528Z github-secret: *** 2024-04-03T16:13:36.5041711Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-04-03T16:13:36.5042995Z activate-with-label: false 2024-04-03T16:13:36.5043447Z label: with-ssh 2024-04-03T16:13:36.5043822Z remove-existing-keys: true 2024-04-03T16:13:36.5044259Z env: 2024-04-03T16:13:36.5044594Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:13:36.5045008Z ##[endgroup] 2024-04-03T16:13:36.6336673Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-04-03T16:13:37.0038482Z Grabbing public ssh keys from https://github.com/caic99.keys 2024-04-03T16:13:37.0747239Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-04-03T16:13:37.0769403Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-04-03T16:13:37.0815324Z Login using: ssh ec2-user@ec2-54-166-137-214.compute-1.amazonaws.com 2024-04-03T16:13:37.0816279Z All testing is done inside the container, to start an interactive session run: 2024-04-03T16:13:37.0817329Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-04-03T16:13:37.1149738Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-04-03T16:13:37.1150458Z with: 2024-04-03T16:13:37.1150802Z submodules: recursive 2024-04-03T16:13:37.1151223Z fetch-depth: 0 2024-04-03T16:13:37.1151564Z env: 2024-04-03T16:13:37.1151893Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:13:37.1152321Z ##[endgroup] 2024-04-03T16:13:37.1451205Z ##[group]Run retry () { 2024-04-03T16:13:37.1451715Z retry () { 2024-04-03T16:13:37.1452349Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-04-03T16:13:37.1453045Z } 2024-04-03T16:13:37.1453402Z echo "${GITHUB_WORKSPACE}" 2024-04-03T16:13:37.1453921Z if [ -z "${NO_SUDO}" ]; then 2024-04-03T16:13:37.1454518Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-04-03T16:13:37.1455067Z else 2024-04-03T16:13:37.1455488Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-04-03T16:13:37.1456016Z fi 2024-04-03T16:13:37.1456442Z mkdir "${GITHUB_WORKSPACE}" 2024-04-03T16:13:37.1486100Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:13:37.1486757Z env: 2024-04-03T16:13:37.1487099Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:13:37.1487508Z NO_SUDO: 2024-04-03T16:13:37.1487853Z ##[endgroup] 2024-04-03T16:13:37.1614555Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:13:37.1857480Z ##[group]Run malfet/checkout@silent-checkout 2024-04-03T16:13:37.1858045Z with: 2024-04-03T16:13:37.1858435Z ref: d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:13:37.1858998Z fetch-depth: 0 2024-04-03T16:13:37.1859577Z submodules: recursive 2024-04-03T16:13:37.1860008Z quiet-checkout: true 2024-04-03T16:13:37.1860425Z repository: pytorch/pytorch 2024-04-03T16:13:37.1861098Z token: *** 2024-04-03T16:13:37.1861463Z ssh-strict: true 2024-04-03T16:13:37.1861851Z persist-credentials: true 2024-04-03T16:13:37.1862297Z clean: true 2024-04-03T16:13:37.1862708Z sparse-checkout-cone-mode: true 2024-04-03T16:13:37.1863175Z lfs: false 2024-04-03T16:13:37.1863544Z set-safe-directory: true 2024-04-03T16:13:37.1863958Z env: 2024-04-03T16:13:37.1864270Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:13:37.1864700Z ##[endgroup] 2024-04-03T16:13:37.3175133Z Syncing repository: pytorch/pytorch 2024-04-03T16:13:37.3177707Z ##[group]Getting Git version info 2024-04-03T16:13:37.3178654Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-04-03T16:13:37.3179765Z [command]/usr/bin/git version 2024-04-03T16:13:37.3180214Z git version 2.40.1 2024-04-03T16:13:37.3181825Z ##[endgroup] 2024-04-03T16:13:37.3198159Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/fb108ab1-e115-4bae-8f7f-d1fa2cbfe547' before making global git config changes 2024-04-03T16:13:37.3199792Z Adding repository directory to the temporary git global config as a safe directory 2024-04-03T16:13:37.3203243Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:13:37.3247709Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-04-03T16:13:37.3254128Z ##[group]Initializing the repository 2024-04-03T16:13:37.3257832Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:13:37.3289207Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-04-03T16:13:37.3290684Z hint: is subject to change. To configure the initial branch name to use in all 2024-04-03T16:13:37.3292117Z hint: of your new repositories, which will suppress this warning, call: 2024-04-03T16:13:37.3293008Z hint: 2024-04-03T16:13:37.3293649Z hint: git config --global init.defaultBranch 2024-04-03T16:13:37.3294352Z hint: 2024-04-03T16:13:37.3295183Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-04-03T16:13:37.3296774Z hint: 'development'. The just-created branch can be renamed via this command: 2024-04-03T16:13:37.3297772Z hint: 2024-04-03T16:13:37.3298398Z hint: git branch -m 2024-04-03T16:13:37.3299400Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-04-03T16:13:37.3301873Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-04-03T16:13:37.3333862Z ##[endgroup] 2024-04-03T16:13:37.3334852Z ##[group]Disabling automatic garbage collection 2024-04-03T16:13:37.3337866Z [command]/usr/bin/git config --local gc.auto 0 2024-04-03T16:13:37.3371266Z ##[endgroup] 2024-04-03T16:13:37.3372108Z ##[group]Setting up auth 2024-04-03T16:13:37.3380514Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-04-03T16:13:37.3412023Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-04-03T16:13:37.3729266Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-04-03T16:13:37.3759034Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-04-03T16:13:37.4055937Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-04-03T16:13:37.4103246Z ##[endgroup] 2024-04-03T16:13:37.4104216Z ##[group]Fetching the repository 2024-04-03T16:13:37.4112242Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-04-03T16:13:42.3853206Z remote: Enumerating objects: 1123747 2024-04-03T16:13:42.3853991Z remote: Enumerating objects: 1124411, done. 2024-04-03T16:13:42.3855337Z remote: Counting objects: 0% (1/664) 2024-04-03T16:13:42.3856664Z remote: Counting objects: 1% (7/664) 2024-04-03T16:13:42.3858058Z remote: Counting objects: 2% (14/664) 2024-04-03T16:13:42.3859428Z remote: Counting objects: 3% (20/664) 2024-04-03T16:13:42.3860185Z remote: Counting objects: 4% (27/664) 2024-04-03T16:13:42.3861430Z remote: Counting objects: 5% (34/664) 2024-04-03T16:13:42.3862822Z remote: Counting objects: 6% (40/664) 2024-04-03T16:13:42.3864118Z remote: Counting objects: 7% (47/664) 2024-04-03T16:13:42.3865448Z remote: Counting objects: 8% (54/664) 2024-04-03T16:13:42.3866366Z remote: Counting objects: 9% (60/664) 2024-04-03T16:13:42.3866993Z remote: Counting objects: 10% (67/664) 2024-04-03T16:13:42.3867591Z remote: Counting objects: 11% (74/664) 2024-04-03T16:13:42.3868207Z remote: Counting objects: 12% (80/664) 2024-04-03T16:13:42.3868822Z remote: Counting objects: 13% (87/664) 2024-04-03T16:13:42.3869418Z remote: Counting objects: 14% (93/664) 2024-04-03T16:13:42.3870049Z remote: Counting objects: 15% (100/664) 2024-04-03T16:13:42.3870686Z remote: Counting objects: 16% (107/664) 2024-04-03T16:13:42.3871298Z remote: Counting objects: 17% (113/664) 2024-04-03T16:13:42.3871928Z remote: Counting objects: 18% (120/664) 2024-04-03T16:13:42.3872557Z remote: Counting objects: 19% (127/664) 2024-04-03T16:13:42.3873445Z remote: Counting objects: 20% (133/664) 2024-04-03T16:13:42.3874073Z remote: Counting objects: 21% (140/664) 2024-04-03T16:13:42.3874700Z remote: Counting objects: 22% (147/664) 2024-04-03T16:13:42.3875308Z remote: Counting objects: 23% (153/664) 2024-04-03T16:13:42.3875929Z remote: Counting objects: 24% (160/664) 2024-04-03T16:13:42.3876596Z remote: Counting objects: 25% (166/664) 2024-04-03T16:13:42.3877233Z remote: Counting objects: 26% (173/664) 2024-04-03T16:13:42.3877838Z remote: Counting objects: 27% (180/664) 2024-04-03T16:13:42.3878607Z remote: Counting objects: 28% (186/664) 2024-04-03T16:13:42.3879254Z remote: Counting objects: 29% (193/664) 2024-04-03T16:13:42.3879884Z remote: Counting objects: 30% (200/664) 2024-04-03T16:13:42.3880488Z remote: Counting objects: 31% (206/664) 2024-04-03T16:13:42.3881113Z remote: Counting objects: 32% (213/664) 2024-04-03T16:13:42.3881740Z remote: Counting objects: 33% (220/664) 2024-04-03T16:13:42.3882343Z remote: Counting objects: 34% (226/664) 2024-04-03T16:13:42.3882967Z remote: Counting objects: 35% (233/664) 2024-04-03T16:13:42.3883584Z remote: Counting objects: 36% (240/664) 2024-04-03T16:13:42.3884189Z remote: Counting objects: 37% (246/664) 2024-04-03T16:13:42.3884810Z remote: Counting objects: 38% (253/664) 2024-04-03T16:13:42.3885426Z remote: Counting objects: 39% (259/664) 2024-04-03T16:13:42.3886029Z remote: Counting objects: 40% (266/664) 2024-04-03T16:13:42.3886659Z remote: Counting objects: 41% (273/664) 2024-04-03T16:13:42.3887286Z remote: Counting objects: 42% (279/664) 2024-04-03T16:13:42.3887892Z remote: Counting objects: 43% (286/664) 2024-04-03T16:13:42.3888512Z remote: Counting objects: 44% (293/664) 2024-04-03T16:13:42.3889130Z remote: Counting objects: 45% (299/664) 2024-04-03T16:13:42.3889732Z remote: Counting objects: 46% (306/664) 2024-04-03T16:13:42.3890364Z remote: Counting objects: 47% (313/664) 2024-04-03T16:13:42.3890987Z remote: Counting objects: 48% (319/664) 2024-04-03T16:13:42.3891590Z remote: Counting objects: 49% (326/664) 2024-04-03T16:13:42.3892211Z remote: Counting objects: 50% (332/664) 2024-04-03T16:13:42.3892831Z remote: Counting objects: 51% (339/664) 2024-04-03T16:13:42.3893430Z remote: Counting objects: 52% (346/664) 2024-04-03T16:13:42.3894053Z remote: Counting objects: 53% (352/664) 2024-04-03T16:13:42.3894677Z remote: Counting objects: 54% (359/664) 2024-04-03T16:13:42.3895298Z remote: Counting objects: 55% (366/664) 2024-04-03T16:13:42.3895904Z remote: Counting objects: 56% (372/664) 2024-04-03T16:13:42.3896520Z remote: Counting objects: 57% (379/664) 2024-04-03T16:13:42.3897140Z remote: Counting objects: 58% (386/664) 2024-04-03T16:13:42.3897740Z remote: Counting objects: 59% (392/664) 2024-04-03T16:13:42.3898367Z remote: Counting objects: 60% (399/664) 2024-04-03T16:13:42.3898987Z remote: Counting objects: 61% (406/664) 2024-04-03T16:13:42.3899590Z remote: Counting objects: 62% (412/664) 2024-04-03T16:13:42.3900284Z remote: Counting objects: 63% (419/664) 2024-04-03T16:13:42.3900906Z remote: Counting objects: 64% (425/664) 2024-04-03T16:13:42.3901529Z remote: Counting objects: 65% (432/664) 2024-04-03T16:13:42.3902133Z remote: Counting objects: 66% (439/664) 2024-04-03T16:13:42.3902757Z remote: Counting objects: 67% (445/664) 2024-04-03T16:13:42.3903385Z remote: Counting objects: 68% (452/664) 2024-04-03T16:13:42.3903986Z remote: Counting objects: 69% (459/664) 2024-04-03T16:13:42.3904606Z remote: Counting objects: 70% (465/664) 2024-04-03T16:13:42.3905229Z remote: Counting objects: 71% (472/664) 2024-04-03T16:13:42.3906074Z remote: Counting objects: 72% (479/664) 2024-04-03T16:13:42.3906857Z remote: Counting objects: 73% (485/664) 2024-04-03T16:13:42.3907480Z remote: Counting objects: 74% (492/664) 2024-04-03T16:13:42.3908089Z remote: Counting objects: 75% (498/664) 2024-04-03T16:13:42.3908715Z remote: Counting objects: 76% (505/664) 2024-04-03T16:13:42.3909335Z remote: Counting objects: 77% (512/664) 2024-04-03T16:13:42.3909938Z remote: Counting objects: 78% (518/664) 2024-04-03T16:13:42.3910565Z remote: Counting objects: 79% (525/664) 2024-04-03T16:13:42.3911193Z remote: Counting objects: 80% (532/664) 2024-04-03T16:13:42.3911929Z remote: Counting objects: 81% (538/664) 2024-04-03T16:13:42.3912599Z remote: Counting objects: 82% (545/664) 2024-04-03T16:13:42.3913222Z remote: Counting objects: 83% (552/664) 2024-04-03T16:13:42.3913844Z remote: Counting objects: 84% (558/664) 2024-04-03T16:13:42.3914444Z remote: Counting objects: 85% (565/664) 2024-04-03T16:13:42.3915077Z remote: Counting objects: 86% (572/664) 2024-04-03T16:13:42.3915693Z remote: Counting objects: 87% (578/664) 2024-04-03T16:13:42.3916295Z remote: Counting objects: 88% (585/664) 2024-04-03T16:13:42.3916913Z remote: Counting objects: 89% (591/664) 2024-04-03T16:13:42.3917528Z remote: Counting objects: 90% (598/664) 2024-04-03T16:13:42.3918125Z remote: Counting objects: 91% (605/664) 2024-04-03T16:13:42.3918744Z remote: Counting objects: 92% (611/664) 2024-04-03T16:13:42.3919361Z remote: Counting objects: 93% (618/664) 2024-04-03T16:13:42.3919969Z remote: Counting objects: 94% (625/664) 2024-04-03T16:13:42.3920590Z remote: Counting objects: 95% (631/664) 2024-04-03T16:13:42.3921210Z remote: Counting objects: 96% (638/664) 2024-04-03T16:13:42.3921810Z remote: Counting objects: 97% (645/664) 2024-04-03T16:13:42.3922430Z remote: Counting objects: 98% (651/664) 2024-04-03T16:13:42.3923059Z remote: Counting objects: 99% (658/664) 2024-04-03T16:13:42.3923662Z remote: Counting objects: 100% (664/664) 2024-04-03T16:13:42.3924332Z remote: Counting objects: 100% (664/664), done. 2024-04-03T16:13:42.3925034Z remote: Compressing objects: 0% (1/369) 2024-04-03T16:13:42.3973338Z remote: Compressing objects: 1% (4/369) 2024-04-03T16:13:42.4032392Z remote: Compressing objects: 2% (8/369) 2024-04-03T16:13:42.4165243Z remote: Compressing objects: 3% (12/369) 2024-04-03T16:13:42.4248679Z remote: Compressing objects: 4% (15/369) 2024-04-03T16:13:42.4667510Z remote: Compressing objects: 5% (19/369) 2024-04-03T16:13:42.4761021Z remote: Compressing objects: 6% (23/369) 2024-04-03T16:13:42.4885772Z remote: Compressing objects: 7% (26/369) 2024-04-03T16:13:42.5012847Z remote: Compressing objects: 8% (30/369) 2024-04-03T16:13:42.5040867Z remote: Compressing objects: 9% (34/369) 2024-04-03T16:13:42.5095537Z remote: Compressing objects: 10% (37/369) 2024-04-03T16:13:42.5176987Z remote: Compressing objects: 11% (41/369) 2024-04-03T16:13:42.5188440Z remote: Compressing objects: 12% (45/369) 2024-04-03T16:13:42.5233574Z remote: Compressing objects: 13% (48/369) 2024-04-03T16:13:42.5242620Z remote: Compressing objects: 14% (52/369) 2024-04-03T16:13:42.5272511Z remote: Compressing objects: 15% (56/369) 2024-04-03T16:13:42.5287575Z remote: Compressing objects: 16% (60/369) 2024-04-03T16:13:42.5305453Z remote: Compressing objects: 17% (63/369) 2024-04-03T16:13:42.5311150Z remote: Compressing objects: 18% (67/369) 2024-04-03T16:13:42.5315177Z remote: Compressing objects: 19% (71/369) 2024-04-03T16:13:42.5321402Z remote: Compressing objects: 20% (74/369) 2024-04-03T16:13:42.5329035Z remote: Compressing objects: 21% (78/369) 2024-04-03T16:13:42.5339496Z remote: Compressing objects: 22% (82/369) 2024-04-03T16:13:42.5349625Z remote: Compressing objects: 23% (85/369) 2024-04-03T16:13:42.5350368Z remote: Compressing objects: 24% (89/369) 2024-04-03T16:13:42.5350874Z remote: Compressing objects: 25% (93/369) 2024-04-03T16:13:42.5355186Z remote: Compressing objects: 26% (96/369) 2024-04-03T16:13:42.5355934Z remote: Compressing objects: 27% (100/369) 2024-04-03T16:13:42.5356475Z remote: Compressing objects: 28% (104/369) 2024-04-03T16:13:42.5359552Z remote: Compressing objects: 29% (108/369) 2024-04-03T16:13:42.5362505Z remote: Compressing objects: 30% (111/369) 2024-04-03T16:13:42.5363594Z remote: Compressing objects: 31% (115/369) 2024-04-03T16:13:42.5364649Z remote: Compressing objects: 32% (119/369) 2024-04-03T16:13:42.5376506Z remote: Compressing objects: 33% (122/369) 2024-04-03T16:13:42.5381167Z remote: Compressing objects: 34% (126/369) 2024-04-03T16:13:42.5384009Z remote: Compressing objects: 35% (130/369) 2024-04-03T16:13:42.5389336Z remote: Compressing objects: 36% (133/369) 2024-04-03T16:13:42.5392301Z remote: Compressing objects: 37% (137/369) 2024-04-03T16:13:42.5394145Z remote: Compressing objects: 38% (141/369) 2024-04-03T16:13:42.5396355Z remote: Compressing objects: 39% (144/369) 2024-04-03T16:13:42.5399933Z remote: Compressing objects: 40% (148/369) 2024-04-03T16:13:42.5403614Z remote: Compressing objects: 41% (152/369) 2024-04-03T16:13:42.5411291Z remote: Compressing objects: 42% (155/369) 2024-04-03T16:13:42.5412482Z remote: Compressing objects: 43% (159/369) 2024-04-03T16:13:42.5414966Z remote: Compressing objects: 44% (163/369) 2024-04-03T16:13:42.5423348Z remote: Compressing objects: 45% (167/369) 2024-04-03T16:13:42.5424042Z remote: Compressing objects: 46% (170/369) 2024-04-03T16:13:42.5424634Z remote: Compressing objects: 47% (174/369) 2024-04-03T16:13:42.5425553Z remote: Compressing objects: 48% (178/369) 2024-04-03T16:13:42.5426416Z remote: Compressing objects: 49% (181/369) 2024-04-03T16:13:42.5428436Z remote: Compressing objects: 50% (185/369) 2024-04-03T16:13:42.5430085Z remote: Compressing objects: 51% (189/369) 2024-04-03T16:13:42.5430981Z remote: Compressing objects: 52% (192/369) 2024-04-03T16:13:42.5432291Z remote: Compressing objects: 53% (196/369) 2024-04-03T16:13:42.5433528Z remote: Compressing objects: 54% (200/369) 2024-04-03T16:13:42.5434491Z remote: Compressing objects: 55% (203/369) 2024-04-03T16:13:42.5435301Z remote: Compressing objects: 56% (207/369) 2024-04-03T16:13:42.5436685Z remote: Compressing objects: 57% (211/369) 2024-04-03T16:13:42.5437461Z remote: Compressing objects: 58% (215/369) 2024-04-03T16:13:42.5438932Z remote: Compressing objects: 59% (218/369) 2024-04-03T16:13:42.5439638Z remote: Compressing objects: 60% (222/369) 2024-04-03T16:13:42.5440343Z remote: Compressing objects: 61% (226/369) 2024-04-03T16:13:42.5441010Z remote: Compressing objects: 62% (229/369) 2024-04-03T16:13:42.5441650Z remote: Compressing objects: 63% (233/369) 2024-04-03T16:13:42.5442304Z remote: Compressing objects: 64% (237/369) 2024-04-03T16:13:42.5442873Z remote: Compressing objects: 65% (240/369) 2024-04-03T16:13:42.5443544Z remote: Compressing objects: 66% (244/369) 2024-04-03T16:13:42.5444210Z remote: Compressing objects: 67% (248/369) 2024-04-03T16:13:42.5444917Z remote: Compressing objects: 68% (251/369) 2024-04-03T16:13:42.5445577Z remote: Compressing objects: 69% (255/369) 2024-04-03T16:13:42.5446233Z remote: Compressing objects: 70% (259/369) 2024-04-03T16:13:42.5446844Z remote: Compressing objects: 71% (262/369) 2024-04-03T16:13:42.5447568Z remote: Compressing objects: 72% (266/369) 2024-04-03T16:13:42.5448399Z remote: Compressing objects: 73% (270/369) 2024-04-03T16:13:42.5449444Z remote: Compressing objects: 74% (274/369) 2024-04-03T16:13:42.5450612Z remote: Compressing objects: 75% (277/369) 2024-04-03T16:13:42.5452688Z remote: Compressing objects: 76% (281/369) 2024-04-03T16:13:42.5453651Z remote: Compressing objects: 77% (285/369) 2024-04-03T16:13:42.5454729Z remote: Compressing objects: 78% (288/369) 2024-04-03T16:13:42.5455449Z remote: Compressing objects: 79% (292/369) 2024-04-03T16:13:42.5456501Z remote: Compressing objects: 80% (296/369) 2024-04-03T16:13:42.5458301Z remote: Compressing objects: 81% (299/369) 2024-04-03T16:13:42.5458999Z remote: Compressing objects: 82% (303/369) 2024-04-03T16:13:42.5459875Z remote: Compressing objects: 83% (307/369) 2024-04-03T16:13:42.5460956Z remote: Compressing objects: 84% (310/369) 2024-04-03T16:13:42.5462207Z remote: Compressing objects: 85% (314/369) 2024-04-03T16:13:42.5463653Z remote: Compressing objects: 86% (318/369) 2024-04-03T16:13:42.5465652Z remote: Compressing objects: 87% (322/369) 2024-04-03T16:13:42.5467866Z remote: Compressing objects: 88% (325/369) 2024-04-03T16:13:42.5468784Z remote: Compressing objects: 89% (329/369) 2024-04-03T16:13:42.5469887Z remote: Compressing objects: 90% (333/369) 2024-04-03T16:13:42.5470890Z remote: Compressing objects: 91% (336/369) 2024-04-03T16:13:42.5471541Z remote: Compressing objects: 92% (340/369) 2024-04-03T16:13:42.5472217Z remote: Compressing objects: 93% (344/369) 2024-04-03T16:13:42.5472890Z remote: Compressing objects: 94% (347/369) 2024-04-03T16:13:42.5478024Z remote: Compressing objects: 95% (351/369) 2024-04-03T16:13:42.5478885Z remote: Compressing objects: 96% (355/369) 2024-04-03T16:13:42.5479446Z remote: Compressing objects: 97% (358/369) 2024-04-03T16:13:42.5480142Z remote: Compressing objects: 98% (362/369) 2024-04-03T16:13:42.5480870Z remote: Compressing objects: 99% (366/369) 2024-04-03T16:13:42.5481542Z remote: Compressing objects: 100% (369/369) 2024-04-03T16:13:42.5482120Z remote: Compressing objects: 100% (369/369), done. 2024-04-03T16:14:10.1051731Z remote: Total 1124411 (delta 364), reused 532 (delta 295), pack-reused 1123747 2024-04-03T16:14:35.4663220Z [command]/usr/bin/git rev-parse --verify --quiet d253bd7befc64137de913dbb8727b511f1052411^{object} 2024-04-03T16:14:35.4691604Z d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:14:35.4701401Z ##[endgroup] 2024-04-03T16:14:35.4702813Z ##[group]Determining the checkout info 2024-04-03T16:14:35.4704402Z ##[endgroup] 2024-04-03T16:14:35.4705979Z ##[group]Checking out the ref 2024-04-03T16:14:35.4708875Z [command]/usr/bin/git checkout --quiet --force d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:14:37.4142214Z ##[endgroup] 2024-04-03T16:14:37.4143182Z ##[group]Setting up auth for fetching submodules 2024-04-03T16:14:37.4149311Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-04-03T16:14:37.4202526Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-04-03T16:14:37.4233954Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-04-03T16:14:37.4266014Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-04-03T16:14:37.4294643Z ##[endgroup] 2024-04-03T16:14:37.4295288Z ##[group]Fetching submodules 2024-04-03T16:14:37.4299812Z [command]/usr/bin/git submodule sync --recursive 2024-04-03T16:14:37.4627979Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-04-03T16:14:37.4938715Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-04-03T16:14:37.4940990Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-04-03T16:14:37.4944657Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-04-03T16:14:37.4948264Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-04-03T16:14:37.4951953Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2024-04-03T16:14:37.4956354Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-04-03T16:14:37.4959113Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-04-03T16:14:37.4963525Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-04-03T16:14:37.4967525Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-04-03T16:14:37.4971653Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2024-04-03T16:14:37.4976164Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-04-03T16:14:37.4980496Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-04-03T16:14:37.4985018Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-04-03T16:14:37.4990042Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-04-03T16:14:37.4995018Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-04-03T16:14:37.4999952Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-04-03T16:14:37.5005067Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-04-03T16:14:37.5010829Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:14:37.5016022Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-04-03T16:14:37.5021745Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-04-03T16:14:37.5027731Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-04-03T16:14:37.5033871Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2024-04-03T16:14:37.5039724Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-04-03T16:14:37.5045753Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-04-03T16:14:37.5052019Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-04-03T16:14:37.5058313Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-04-03T16:14:37.5064857Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2024-04-03T16:14:37.5071941Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-04-03T16:14:37.5078427Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-04-03T16:14:37.5085501Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2024-04-03T16:14:37.5092460Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-04-03T16:14:37.5099570Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-04-03T16:14:37.5107233Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-04-03T16:14:37.5114913Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-04-03T16:14:37.5122127Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-04-03T16:14:37.5129876Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-04-03T16:14:37.5137354Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-04-03T16:14:37.5145111Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2024-04-03T16:14:37.5154587Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-04-03T16:14:37.5162623Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2024-04-03T16:14:37.5191368Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-04-03T16:14:37.7845263Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-04-03T16:14:37.9678351Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-04-03T16:14:38.1466309Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-04-03T16:14:38.3905924Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2024-04-03T16:14:38.6212794Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-04-03T16:14:40.7424593Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-04-03T16:14:51.3063584Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-04-03T16:14:51.7894995Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-04-03T16:14:52.5151512Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2024-04-03T16:14:54.0839466Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-04-03T16:14:55.5783357Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-04-03T16:15:12.4987342Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-04-03T16:15:19.6158748Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-04-03T16:15:20.8512915Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-04-03T16:15:22.7072625Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-04-03T16:15:24.0024057Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-04-03T16:15:24.1742852Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-04-03T16:15:24.6322730Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-04-03T16:15:24.9595916Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-04-03T16:15:26.1367680Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-04-03T16:15:26.4829566Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2024-04-03T16:15:26.6832213Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-04-03T16:15:26.9704925Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-04-03T16:15:28.6768658Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-04-03T16:15:29.3454652Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-04-03T16:15:29.9858102Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2024-04-03T16:15:30.3431621Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-04-03T16:15:37.6114676Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-04-03T16:15:39.9502822Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2024-04-03T16:15:40.4562860Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-04-03T16:15:40.6601872Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-04-03T16:15:49.8524871Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-04-03T16:15:50.0234956Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-04-03T16:15:50.2209921Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-04-03T16:15:51.2187554Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-04-03T16:15:51.5476665Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-04-03T16:15:52.2196295Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2024-04-03T16:15:54.6187584Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-04-03T16:15:55.0582954Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2024-04-03T16:15:57.6444041Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-04-03T16:15:57.6567248Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-04-03T16:15:57.6665803Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-04-03T16:15:57.6936849Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-04-03T16:15:57.7206938Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2024-04-03T16:15:57.7655334Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-04-03T16:15:58.9464575Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-04-03T16:15:58.9721322Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-04-03T16:15:59.0869629Z Submodule path 'third_party/cpuinfo': checked out 'd6860c477c99f1fce9e28eb206891af3c0e1a1d7' 2024-04-03T16:15:59.1277308Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2024-04-03T16:15:59.1598821Z Submodule path 'third_party/cudnn_frontend': checked out '150798fe976556078f443fdb059a1ff0361f58a2' 2024-04-03T16:15:59.8049019Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-04-03T16:16:00.1116803Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-04-03T16:16:00.1943463Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-04-03T16:16:00.1962124Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:16:00.1965274Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:16:00.1968941Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:16:00.1972188Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:16:00.1976586Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:16:00.2003481Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-04-03T16:16:01.2677797Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-04-03T16:16:01.9495876Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-04-03T16:16:04.8744486Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-04-03T16:16:06.0568644Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-04-03T16:16:06.4380569Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-04-03T16:16:06.5546973Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-04-03T16:16:07.0740399Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-04-03T16:16:07.1487738Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-04-03T16:16:07.1616015Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-04-03T16:16:07.2966268Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-04-03T16:16:07.3374405Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-04-03T16:16:07.3475670Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-04-03T16:16:07.3940464Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-04-03T16:16:07.4223935Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-04-03T16:16:07.4786106Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-04-03T16:16:07.4924862Z Submodule path 'third_party/ideep': checked out '8a6cc4e09dc509f04f83c085e38786b1fb44e14d' 2024-04-03T16:16:07.4941566Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-04-03T16:16:07.4967619Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-04-03T16:16:19.8905474Z Submodule path 'third_party/ideep/mkl-dnn': checked out '86e6af5974177e513fd3fee58425e1063e7f1361' 2024-04-03T16:16:19.9020524Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2024-04-03T16:16:19.9209443Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-04-03T16:16:20.0399479Z Submodule path 'third_party/kineto': checked out '8466a8b111b36dc725e6855d52a0b133d925a8e0' 2024-04-03T16:16:20.0418606Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:16:20.0421148Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:16:20.0425216Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:16:20.0452068Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-04-03T16:16:20.9569679Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-04-03T16:16:22.3070924Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-04-03T16:16:23.6495390Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-04-03T16:16:23.6512355Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:16:23.6515229Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:16:23.6519066Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:16:23.6522189Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:16:23.6526126Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:16:23.6529986Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:16:23.6533710Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:16:23.6537904Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:16:23.6565854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-04-03T16:16:24.8747197Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-04-03T16:16:25.2402505Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-04-03T16:16:26.5752387Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-04-03T16:16:26.9039057Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-04-03T16:16:27.2864638Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-04-03T16:16:28.4831430Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-04-03T16:16:35.9074298Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-04-03T16:16:36.3492793Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-04-03T16:16:36.3693875Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-04-03T16:16:36.4128652Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-04-03T16:16:36.4271132Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-04-03T16:16:36.4287931Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:16:36.4314361Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-04-03T16:16:36.7372581Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-04-03T16:16:36.7577462Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-04-03T16:16:36.8046788Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-04-03T16:16:36.9266640Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-04-03T16:16:36.9446379Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-04-03T16:16:36.9882836Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-04-03T16:16:37.0565044Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-04-03T16:16:37.0983380Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-04-03T16:16:37.1254841Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-04-03T16:16:37.1409872Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2024-04-03T16:16:37.2720033Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-04-03T16:16:37.7159526Z Submodule path 'third_party/onnx': checked out 'ccde5da81388ffa770ca98b64e07f803ad089414' 2024-04-03T16:16:37.7196409Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-04-03T16:16:37.7199613Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-04-03T16:16:37.7226829Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-04-03T16:16:38.1445955Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-04-03T16:16:39.2110592Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-04-03T16:16:39.2509070Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-04-03T16:16:39.2716413Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2024-04-03T16:16:39.2734683Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:16:39.2761147Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2024-04-03T16:16:41.9476393Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2024-04-03T16:16:41.9498595Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:16:41.9501555Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:16:41.9529973Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2024-04-03T16:16:42.4018939Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2024-04-03T16:16:43.4154780Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2024-04-03T16:16:43.5009108Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2024-04-03T16:16:43.5025492Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:16:43.5051763Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2024-04-03T16:16:43.7049805Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-04-03T16:16:43.7152744Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-04-03T16:16:44.0322928Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-04-03T16:16:44.0346503Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:16:44.0349928Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-04-03T16:16:44.0376517Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-04-03T16:16:44.4590863Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-04-03T16:16:45.6880696Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-04-03T16:16:45.7728626Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-04-03T16:16:45.7822698Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-04-03T16:16:45.7958204Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-04-03T16:16:45.8339663Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-04-03T16:16:45.8672381Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-04-03T16:16:45.9158922Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-04-03T16:16:46.0486902Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2024-04-03T16:16:46.0790833Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-04-03T16:16:46.0808606Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:16:46.0811309Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:16:46.0815066Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:16:46.0818233Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:16:46.0845078Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-04-03T16:16:47.3223410Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-04-03T16:16:47.5640517Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-04-03T16:16:49.9688492Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-04-03T16:16:51.0646155Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-04-03T16:16:51.0812966Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-04-03T16:16:51.1653087Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-04-03T16:16:51.1982387Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-04-03T16:16:51.2000150Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:16:51.2026633Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-04-03T16:16:51.3973047Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-04-03T16:16:51.5721874Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2024-04-03T16:16:51.5755101Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-04-03T16:16:51.6084071Z Entering 'android/libs/fbjni' 2024-04-03T16:16:51.6127948Z Entering 'third_party/FP16' 2024-04-03T16:16:51.6169791Z Entering 'third_party/FXdiv' 2024-04-03T16:16:51.6212285Z Entering 'third_party/NNPACK' 2024-04-03T16:16:51.6255040Z Entering 'third_party/QNNPACK' 2024-04-03T16:16:51.6297900Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:16:51.6339994Z Entering 'third_party/XNNPACK' 2024-04-03T16:16:51.6399514Z Entering 'third_party/benchmark' 2024-04-03T16:16:51.6441957Z Entering 'third_party/cpuinfo' 2024-04-03T16:16:51.6486486Z Entering 'third_party/cub' 2024-04-03T16:16:51.6528064Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:16:51.6570053Z Entering 'third_party/cutlass' 2024-04-03T16:16:51.6619616Z Entering 'third_party/eigen' 2024-04-03T16:16:51.6665267Z Entering 'third_party/fbgemm' 2024-04-03T16:16:51.6708182Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:16:51.6748302Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:16:51.6789236Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:16:51.6837568Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:16:51.6877350Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:16:51.6919594Z Entering 'third_party/flatbuffers' 2024-04-03T16:16:51.6964280Z Entering 'third_party/fmt' 2024-04-03T16:16:51.7005669Z Entering 'third_party/foxi' 2024-04-03T16:16:51.7047289Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:16:51.7090058Z Entering 'third_party/gloo' 2024-04-03T16:16:51.7131725Z Entering 'third_party/googletest' 2024-04-03T16:16:51.7174190Z Entering 'third_party/ideep' 2024-04-03T16:16:51.7214014Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:16:51.7263544Z Entering 'third_party/ios-cmake' 2024-04-03T16:16:51.7305438Z Entering 'third_party/ittapi' 2024-04-03T16:16:51.7347339Z Entering 'third_party/kineto' 2024-04-03T16:16:51.7389223Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:16:51.7430618Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:16:51.7472915Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:16:51.7514798Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:16:51.7555497Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:16:51.7595315Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:16:51.7638176Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:16:51.7678998Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:16:51.7719627Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:16:51.7763036Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:16:51.7807492Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:16:51.7847952Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:16:51.7890160Z Entering 'third_party/mimalloc' 2024-04-03T16:16:51.7933903Z Entering 'third_party/nccl/nccl' 2024-04-03T16:16:51.7975361Z Entering 'third_party/neon2sse' 2024-04-03T16:16:51.8016168Z Entering 'third_party/nlohmann' 2024-04-03T16:16:51.8059435Z Entering 'third_party/onnx' 2024-04-03T16:16:51.8117720Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:16:51.8158889Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:16:51.8202387Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:16:51.8243183Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:16:51.8289574Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:16:51.8330485Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:16:51.8370625Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:16:51.8415818Z Entering 'third_party/pocketfft' 2024-04-03T16:16:51.8457292Z Entering 'third_party/protobuf' 2024-04-03T16:16:51.8503417Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:16:51.8544777Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:16:51.8587468Z Entering 'third_party/psimd' 2024-04-03T16:16:51.8629681Z Entering 'third_party/pthreadpool' 2024-04-03T16:16:51.8671097Z Entering 'third_party/pybind11' 2024-04-03T16:16:51.8712688Z Entering 'third_party/python-peachpy' 2024-04-03T16:16:51.8753929Z Entering 'third_party/sleef' 2024-04-03T16:16:51.8794918Z Entering 'third_party/tbb' 2024-04-03T16:16:51.8838541Z Entering 'third_party/tensorpipe' 2024-04-03T16:16:51.8879835Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:16:51.8921451Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:16:51.8961553Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:16:51.9002998Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:16:51.9043672Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:16:51.9087719Z Entering 'third_party/zstd' 2024-04-03T16:16:51.9142612Z ##[endgroup] 2024-04-03T16:16:51.9143523Z ##[group]Persisting credentials for submodules 2024-04-03T16:16:51.9150966Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-04-03T16:16:51.9469000Z Entering 'android/libs/fbjni' 2024-04-03T16:16:51.9519879Z Entering 'third_party/FP16' 2024-04-03T16:16:51.9571726Z Entering 'third_party/FXdiv' 2024-04-03T16:16:51.9623159Z Entering 'third_party/NNPACK' 2024-04-03T16:16:51.9675487Z Entering 'third_party/QNNPACK' 2024-04-03T16:16:51.9727060Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:16:51.9779168Z Entering 'third_party/XNNPACK' 2024-04-03T16:16:51.9848529Z Entering 'third_party/benchmark' 2024-04-03T16:16:51.9899577Z Entering 'third_party/cpuinfo' 2024-04-03T16:16:51.9952512Z Entering 'third_party/cub' 2024-04-03T16:16:52.0003207Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:16:52.0054056Z Entering 'third_party/cutlass' 2024-04-03T16:16:52.0113114Z Entering 'third_party/eigen' 2024-04-03T16:16:52.0165843Z Entering 'third_party/fbgemm' 2024-04-03T16:16:52.0217423Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:16:52.0269290Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:16:52.0320706Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:16:52.0377683Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:16:52.0428899Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:16:52.0480755Z Entering 'third_party/flatbuffers' 2024-04-03T16:16:52.0534693Z Entering 'third_party/fmt' 2024-04-03T16:16:52.0587785Z Entering 'third_party/foxi' 2024-04-03T16:16:52.0638264Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:16:52.0689834Z Entering 'third_party/gloo' 2024-04-03T16:16:52.0741722Z Entering 'third_party/googletest' 2024-04-03T16:16:52.0794661Z Entering 'third_party/ideep' 2024-04-03T16:16:52.0845074Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:16:52.0905179Z Entering 'third_party/ios-cmake' 2024-04-03T16:16:52.0956648Z Entering 'third_party/ittapi' 2024-04-03T16:16:52.1007815Z Entering 'third_party/kineto' 2024-04-03T16:16:52.1059483Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:16:52.1110346Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:16:52.1162495Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:16:52.1212992Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:16:52.1264004Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:16:52.1314048Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:16:52.1365733Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:16:52.1416624Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:16:52.1468961Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:16:52.1521570Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:16:52.1573442Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:16:52.1625294Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:16:52.1677531Z Entering 'third_party/mimalloc' 2024-04-03T16:16:52.1730169Z Entering 'third_party/nccl/nccl' 2024-04-03T16:16:52.1782149Z Entering 'third_party/neon2sse' 2024-04-03T16:16:52.1834188Z Entering 'third_party/nlohmann' 2024-04-03T16:16:52.1886658Z Entering 'third_party/onnx' 2024-04-03T16:16:52.1955295Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:16:52.2006296Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:16:52.2059532Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:16:52.2111636Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:16:52.2166351Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:16:52.2217811Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:16:52.2268286Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:16:52.2324019Z Entering 'third_party/pocketfft' 2024-04-03T16:16:52.2374580Z Entering 'third_party/protobuf' 2024-04-03T16:16:52.2429448Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:16:52.2479258Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:16:52.2532284Z Entering 'third_party/psimd' 2024-04-03T16:16:52.2583314Z Entering 'third_party/pthreadpool' 2024-04-03T16:16:52.2634660Z Entering 'third_party/pybind11' 2024-04-03T16:16:52.2686350Z Entering 'third_party/python-peachpy' 2024-04-03T16:16:52.2737900Z Entering 'third_party/sleef' 2024-04-03T16:16:52.2791146Z Entering 'third_party/tbb' 2024-04-03T16:16:52.2844491Z Entering 'third_party/tensorpipe' 2024-04-03T16:16:52.2895552Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:16:52.2946719Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:16:52.2998207Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:16:52.3049218Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:16:52.3098235Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:16:52.3153019Z Entering 'third_party/zstd' 2024-04-03T16:16:52.3219439Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-04-03T16:16:52.3530954Z Entering 'android/libs/fbjni' 2024-04-03T16:16:52.3578459Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-04-03T16:16:52.3598111Z Entering 'third_party/FP16' 2024-04-03T16:16:52.3646456Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-04-03T16:16:52.3665052Z Entering 'third_party/FXdiv' 2024-04-03T16:16:52.3713727Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-04-03T16:16:52.3733008Z Entering 'third_party/NNPACK' 2024-04-03T16:16:52.3780648Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-04-03T16:16:52.3800428Z Entering 'third_party/QNNPACK' 2024-04-03T16:16:52.3847623Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2024-04-03T16:16:52.3867099Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:16:52.3914502Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-04-03T16:16:52.3933812Z Entering 'third_party/XNNPACK' 2024-04-03T16:16:52.3982101Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-04-03T16:16:52.4019063Z Entering 'third_party/benchmark' 2024-04-03T16:16:52.4066576Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:16:52.4085708Z Entering 'third_party/cpuinfo' 2024-04-03T16:16:52.4133798Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-04-03T16:16:52.4154299Z Entering 'third_party/cub' 2024-04-03T16:16:52.4202721Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2024-04-03T16:16:52.4222198Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:16:52.4270653Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-04-03T16:16:52.4289589Z Entering 'third_party/cutlass' 2024-04-03T16:16:52.4337303Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-04-03T16:16:52.4364486Z Entering 'third_party/eigen' 2024-04-03T16:16:52.4413073Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-04-03T16:16:52.4434858Z Entering 'third_party/fbgemm' 2024-04-03T16:16:52.4482815Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-04-03T16:16:52.4501884Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:16:52.4549495Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-04-03T16:16:52.4567472Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:16:52.4615215Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-04-03T16:16:52.4634055Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:16:52.4681786Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-04-03T16:16:52.4706755Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:16:52.4753953Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:16:52.4771684Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:16:52.4819132Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-04-03T16:16:52.4838992Z Entering 'third_party/flatbuffers' 2024-04-03T16:16:52.4886192Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-04-03T16:16:52.4908301Z Entering 'third_party/fmt' 2024-04-03T16:16:52.4955673Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-04-03T16:16:52.4974885Z Entering 'third_party/foxi' 2024-04-03T16:16:52.5023411Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-04-03T16:16:52.5043918Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:16:52.5091161Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-04-03T16:16:52.5111235Z Entering 'third_party/gloo' 2024-04-03T16:16:52.5158175Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-04-03T16:16:52.5177620Z Entering 'third_party/googletest' 2024-04-03T16:16:52.5224256Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:16:52.5243940Z Entering 'third_party/ideep' 2024-04-03T16:16:52.5291803Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-04-03T16:16:52.5310184Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:16:52.5356527Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-04-03T16:16:52.5383512Z Entering 'third_party/ios-cmake' 2024-04-03T16:16:52.5432215Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2024-04-03T16:16:52.5451360Z Entering 'third_party/ittapi' 2024-04-03T16:16:52.5498422Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-04-03T16:16:52.5518127Z Entering 'third_party/kineto' 2024-04-03T16:16:52.5566662Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-04-03T16:16:52.5585093Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:16:52.5634234Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-04-03T16:16:52.5652021Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:16:52.5700107Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-04-03T16:16:52.5719874Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:16:52.5767539Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-04-03T16:16:52.5786240Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:16:52.5833991Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-04-03T16:16:52.5852408Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:16:52.5900482Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-04-03T16:16:52.5917557Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:16:52.5966317Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-04-03T16:16:52.5985787Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:16:52.6034221Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-04-03T16:16:52.6052584Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:16:52.6100783Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:16:52.6120667Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:16:52.6168397Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-04-03T16:16:52.6187816Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:16:52.6236151Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-04-03T16:16:52.6255480Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:16:52.6302914Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-04-03T16:16:52.6331424Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:16:52.6368982Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-04-03T16:16:52.6388916Z Entering 'third_party/mimalloc' 2024-04-03T16:16:52.6436344Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-04-03T16:16:52.6455921Z Entering 'third_party/nccl/nccl' 2024-04-03T16:16:52.6503683Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-04-03T16:16:52.6522853Z Entering 'third_party/neon2sse' 2024-04-03T16:16:52.6571132Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2024-04-03T16:16:52.6590170Z Entering 'third_party/nlohmann' 2024-04-03T16:16:52.6637704Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-04-03T16:16:52.6658123Z Entering 'third_party/onnx' 2024-04-03T16:16:52.6706282Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-04-03T16:16:52.6741959Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:16:52.6790729Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:16:52.6809444Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:16:52.6858488Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:16:52.6880151Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:16:52.6928712Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2024-04-03T16:16:52.6946805Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:16:52.6995523Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2024-04-03T16:16:52.7018218Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:16:52.7067333Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:16:52.7086309Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:16:52.7135491Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:16:52.7153127Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:16:52.7202773Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-04-03T16:16:52.7226165Z Entering 'third_party/pocketfft' 2024-04-03T16:16:52.7274492Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-04-03T16:16:52.7293422Z Entering 'third_party/protobuf' 2024-04-03T16:16:52.7341203Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-04-03T16:16:52.7364068Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:16:52.7411761Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:16:52.7430994Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:16:52.7478782Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:16:52.7498774Z Entering 'third_party/psimd' 2024-04-03T16:16:52.7546962Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-04-03T16:16:52.7566462Z Entering 'third_party/pthreadpool' 2024-04-03T16:16:52.7614306Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-04-03T16:16:52.7634492Z Entering 'third_party/pybind11' 2024-04-03T16:16:52.7682206Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:16:52.7702608Z Entering 'third_party/python-peachpy' 2024-04-03T16:16:52.7751867Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-04-03T16:16:52.7771247Z Entering 'third_party/sleef' 2024-04-03T16:16:52.7818797Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-04-03T16:16:52.7838706Z Entering 'third_party/tbb' 2024-04-03T16:16:52.7885948Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2024-04-03T16:16:52.7907780Z Entering 'third_party/tensorpipe' 2024-04-03T16:16:52.7955621Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-04-03T16:16:52.7973978Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:16:52.8020856Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:16:52.8039248Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:16:52.8086075Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-04-03T16:16:52.8103663Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:16:52.8150444Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-04-03T16:16:52.8168363Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:16:52.8215684Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:16:52.8232904Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:16:52.8280154Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-04-03T16:16:52.8301723Z Entering 'third_party/zstd' 2024-04-03T16:16:52.8349499Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2024-04-03T16:16:52.9354508Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-04-03T16:16:52.9667398Z Entering 'android/libs/fbjni' 2024-04-03T16:16:52.9709232Z Entering 'third_party/FP16' 2024-04-03T16:16:52.9751303Z Entering 'third_party/FXdiv' 2024-04-03T16:16:52.9794745Z Entering 'third_party/NNPACK' 2024-04-03T16:16:52.9836272Z Entering 'third_party/QNNPACK' 2024-04-03T16:16:52.9878877Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:16:52.9920546Z Entering 'third_party/XNNPACK' 2024-04-03T16:16:52.9980004Z Entering 'third_party/benchmark' 2024-04-03T16:16:53.0022390Z Entering 'third_party/cpuinfo' 2024-04-03T16:16:53.0065868Z Entering 'third_party/cub' 2024-04-03T16:16:53.0108456Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:16:53.0151055Z Entering 'third_party/cutlass' 2024-04-03T16:16:53.0201959Z Entering 'third_party/eigen' 2024-04-03T16:16:53.0246476Z Entering 'third_party/fbgemm' 2024-04-03T16:16:53.0288388Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:16:53.0329062Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:16:53.0370306Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:16:53.0418612Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:16:53.0459258Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:16:53.0500980Z Entering 'third_party/flatbuffers' 2024-04-03T16:16:53.0546276Z Entering 'third_party/fmt' 2024-04-03T16:16:53.0588794Z Entering 'third_party/foxi' 2024-04-03T16:16:53.0630152Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:16:53.0673180Z Entering 'third_party/gloo' 2024-04-03T16:16:53.0715587Z Entering 'third_party/googletest' 2024-04-03T16:16:53.0758151Z Entering 'third_party/ideep' 2024-04-03T16:16:53.0798949Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:16:53.0847491Z Entering 'third_party/ios-cmake' 2024-04-03T16:16:53.0889808Z Entering 'third_party/ittapi' 2024-04-03T16:16:53.0932708Z Entering 'third_party/kineto' 2024-04-03T16:16:53.0974456Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:16:53.1015665Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:16:53.1058189Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:16:53.1099918Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:16:53.1142470Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:16:53.1183417Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:16:53.1227902Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:16:53.1269348Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:16:53.1310917Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:16:53.1353832Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:16:53.1395755Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:16:53.1436720Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:16:53.1478927Z Entering 'third_party/mimalloc' 2024-04-03T16:16:53.1521267Z Entering 'third_party/nccl/nccl' 2024-04-03T16:16:53.1563709Z Entering 'third_party/neon2sse' 2024-04-03T16:16:53.1605527Z Entering 'third_party/nlohmann' 2024-04-03T16:16:53.1648396Z Entering 'third_party/onnx' 2024-04-03T16:16:53.1707541Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:16:53.1749464Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:16:53.1793973Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:16:53.1834943Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:16:53.1879881Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:16:53.1920609Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:16:53.1959804Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:16:53.2005517Z Entering 'third_party/pocketfft' 2024-04-03T16:16:53.2047199Z Entering 'third_party/protobuf' 2024-04-03T16:16:53.2092589Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:16:53.2133081Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:16:53.2176512Z Entering 'third_party/psimd' 2024-04-03T16:16:53.2217602Z Entering 'third_party/pthreadpool' 2024-04-03T16:16:53.2260440Z Entering 'third_party/pybind11' 2024-04-03T16:16:53.2302193Z Entering 'third_party/python-peachpy' 2024-04-03T16:16:53.2343890Z Entering 'third_party/sleef' 2024-04-03T16:16:53.2384983Z Entering 'third_party/tbb' 2024-04-03T16:16:53.2429970Z Entering 'third_party/tensorpipe' 2024-04-03T16:16:53.2473441Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:16:53.2515311Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:16:53.2555177Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:16:53.2596260Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:16:53.2635518Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:16:53.2679223Z Entering 'third_party/zstd' 2024-04-03T16:16:53.2736100Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-04-03T16:16:53.3050519Z Entering 'android/libs/fbjni' 2024-04-03T16:16:53.3091425Z Entering 'third_party/FP16' 2024-04-03T16:16:53.3134100Z Entering 'third_party/FXdiv' 2024-04-03T16:16:53.3175259Z Entering 'third_party/NNPACK' 2024-04-03T16:16:53.3216648Z Entering 'third_party/QNNPACK' 2024-04-03T16:16:53.3258700Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:16:53.3301358Z Entering 'third_party/XNNPACK' 2024-04-03T16:16:53.3360489Z Entering 'third_party/benchmark' 2024-04-03T16:16:53.3402644Z Entering 'third_party/cpuinfo' 2024-04-03T16:16:53.3444731Z Entering 'third_party/cub' 2024-04-03T16:16:53.3487091Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:16:53.3528358Z Entering 'third_party/cutlass' 2024-04-03T16:16:53.3578287Z Entering 'third_party/eigen' 2024-04-03T16:16:53.3622144Z Entering 'third_party/fbgemm' 2024-04-03T16:16:53.3664025Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:16:53.3704895Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:16:53.3746134Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:16:53.3795478Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:16:53.3836253Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:16:53.3878529Z Entering 'third_party/flatbuffers' 2024-04-03T16:16:53.3923618Z Entering 'third_party/fmt' 2024-04-03T16:16:53.3965763Z Entering 'third_party/foxi' 2024-04-03T16:16:53.4007559Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:16:53.4049301Z Entering 'third_party/gloo' 2024-04-03T16:16:53.4091359Z Entering 'third_party/googletest' 2024-04-03T16:16:53.4133666Z Entering 'third_party/ideep' 2024-04-03T16:16:53.4174586Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:16:53.4222112Z Entering 'third_party/ios-cmake' 2024-04-03T16:16:53.4264176Z Entering 'third_party/ittapi' 2024-04-03T16:16:53.4305046Z Entering 'third_party/kineto' 2024-04-03T16:16:53.4346962Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:16:53.4387412Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:16:53.4430835Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:16:53.4472697Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:16:53.4514826Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:16:53.4555333Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:16:53.4599107Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:16:53.4640592Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:16:53.4681747Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:16:53.4724262Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:16:53.4767189Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:16:53.4807753Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:16:53.4850058Z Entering 'third_party/mimalloc' 2024-04-03T16:16:53.4892310Z Entering 'third_party/nccl/nccl' 2024-04-03T16:16:53.4934453Z Entering 'third_party/neon2sse' 2024-04-03T16:16:53.4976353Z Entering 'third_party/nlohmann' 2024-04-03T16:16:53.5018920Z Entering 'third_party/onnx' 2024-04-03T16:16:53.5079138Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:16:53.5119739Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:16:53.5162686Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:16:53.5204746Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:16:53.5249598Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:16:53.5290361Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:16:53.5330716Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:16:53.5376562Z Entering 'third_party/pocketfft' 2024-04-03T16:16:53.5418043Z Entering 'third_party/protobuf' 2024-04-03T16:16:53.5463231Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:16:53.5503447Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:16:53.5546129Z Entering 'third_party/psimd' 2024-04-03T16:16:53.5587292Z Entering 'third_party/pthreadpool' 2024-04-03T16:16:53.5629013Z Entering 'third_party/pybind11' 2024-04-03T16:16:53.5671577Z Entering 'third_party/python-peachpy' 2024-04-03T16:16:53.5712891Z Entering 'third_party/sleef' 2024-04-03T16:16:53.5755159Z Entering 'third_party/tbb' 2024-04-03T16:16:53.5798125Z Entering 'third_party/tensorpipe' 2024-04-03T16:16:53.5840006Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:16:53.5880377Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:16:53.5921209Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:16:53.5962617Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:16:53.6003097Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:16:53.6047354Z Entering 'third_party/zstd' 2024-04-03T16:16:53.6101139Z ##[endgroup] 2024-04-03T16:16:53.6141231Z [command]/usr/bin/git log -1 --format='%H' 2024-04-03T16:16:53.6167991Z 'd253bd7befc64137de913dbb8727b511f1052411' 2024-04-03T16:16:53.6389650Z Prepare all required actions 2024-04-03T16:16:53.6390263Z Getting action download info 2024-04-03T16:16:53.8163522Z ##[group]Run ./.github/actions/setup-linux 2024-04-03T16:16:53.8164041Z env: 2024-04-03T16:16:53.8164378Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:53.8164813Z ##[endgroup] 2024-04-03T16:16:53.8246381Z ##[group]Run set -euo pipefail 2024-04-03T16:16:53.8246908Z set -euo pipefail 2024-04-03T16:16:53.8247398Z function get_ec2_metadata() { 2024-04-03T16:16:53.8248053Z  # Pulled from instance metadata endpoint for EC2 2024-04-03T16:16:53.8249191Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-04-03T16:16:53.8250175Z  category=$1 2024-04-03T16:16:53.8250822Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-04-03T16:16:53.8251631Z  runner_name_str=i-01dbe7bd2ee9fbb9d 2024-04-03T16:16:53.8252351Z  if [[ $runner_name_str != *"gcp"* ]]; then 2024-04-03T16:16:53.8253193Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-04-03T16:16:53.8253925Z  else 2024-04-03T16:16:53.8254578Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-04-03T16:16:53.8255323Z  fi 2024-04-03T16:16:53.8255671Z } 2024-04-03T16:16:53.8256118Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-04-03T16:16:53.8256831Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-04-03T16:16:53.8257660Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-04-03T16:16:53.8258381Z echo "system info $(uname -a)" 2024-04-03T16:16:53.8267492Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:53.8268097Z env: 2024-04-03T16:16:53.8268438Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:53.8268873Z ##[endgroup] 2024-04-03T16:16:53.8376478Z ami-id: ami-0ce0c36d7a00b20e2 2024-04-03T16:16:53.8445458Z instance-id: i-01dbe7bd2ee9fbb9d 2024-04-03T16:16:53.8511091Z instance-type: g3.8xlarge 2024-04-03T16:16:53.8520288Z system info Linux ip-10-0-13-6.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-04-03T16:16:53.8547901Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-04-03T16:16:53.8549048Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-04-03T16:16:53.8557403Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:53.8558028Z env: 2024-04-03T16:16:53.8558363Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:53.8558772Z ##[endgroup] 2024-04-03T16:16:53.8641654Z ##[group]Run if systemctl is-active --quiet docker; then 2024-04-03T16:16:53.8642407Z if systemctl is-active --quiet docker; then 2024-04-03T16:16:53.8643065Z  echo "Docker daemon is running..."; 2024-04-03T16:16:53.8643652Z else 2024-04-03T16:16:53.8644272Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-04-03T16:16:53.8644985Z fi 2024-04-03T16:16:53.8652592Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:53.8653221Z env: 2024-04-03T16:16:53.8653554Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:53.8653964Z ##[endgroup] 2024-04-03T16:16:53.8701921Z Docker daemon is running... 2024-04-03T16:16:53.8780902Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-04-03T16:16:53.8781608Z with: 2024-04-03T16:16:53.8781929Z shell: bash 2024-04-03T16:16:53.8782274Z timeout_minutes: 5 2024-04-03T16:16:53.8782665Z max_attempts: 3 2024-04-03T16:16:53.8783046Z retry_wait_seconds: 30 2024-04-03T16:16:53.8785005Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-04-03T16:16:53.8787252Z polling_interval_seconds: 1 2024-04-03T16:16:53.8787727Z warning_on_retry: true 2024-04-03T16:16:53.8788157Z continue_on_error: false 2024-04-03T16:16:53.8788552Z env: 2024-04-03T16:16:53.8788879Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:53.8789318Z AWS_RETRY_MODE: standard 2024-04-03T16:16:53.8789728Z AWS_MAX_ATTEMPTS: 5 2024-04-03T16:16:53.8790148Z AWS_DEFAULT_REGION: us-east-1 2024-04-03T16:16:53.8790606Z ##[endgroup] 2024-04-03T16:16:54.8918546Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-04-03T16:16:54.8919635Z Configure a credential helper to remove this warning. See 2024-04-03T16:16:54.8921056Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-04-03T16:16:54.8921756Z 2024-04-03T16:16:54.8921911Z Login Succeeded 2024-04-03T16:16:54.9499526Z Command completed after 1 attempt(s). 2024-04-03T16:16:54.9555213Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-04-03T16:16:54.9556149Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-04-03T16:16:54.9556975Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-04-03T16:16:54.9565471Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:54.9566094Z env: 2024-04-03T16:16:54.9566410Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:54.9566837Z ##[endgroup] 2024-04-03T16:16:54.9644061Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T16:16:54.9645021Z # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T16:16:54.9645761Z # shellcheck disable=SC2046 2024-04-03T16:16:54.9646330Z docker stop $(docker ps -q) || true 2024-04-03T16:16:54.9646910Z # Prune all of the docker images 2024-04-03T16:16:54.9647508Z docker system prune -af 2024-04-03T16:16:54.9655438Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:54.9656084Z env: 2024-04-03T16:16:54.9656402Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:54.9656829Z ##[endgroup] 2024-04-03T16:16:55.0093083Z "docker stop" requires at least 1 argument. 2024-04-03T16:16:55.0093850Z See 'docker stop --help'. 2024-04-03T16:16:55.0094148Z 2024-04-03T16:16:55.0094307Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-04-03T16:16:55.0094784Z 2024-04-03T16:16:55.0094984Z Stop one or more running containers 2024-04-03T16:16:55.0348314Z Total reclaimed space: 0B 2024-04-03T16:16:55.0391010Z ##[group]Run set +e 2024-04-03T16:16:55.0391444Z set +e 2024-04-03T16:16:55.0391820Z set -x 2024-04-03T16:16:55.0392158Z  2024-04-03T16:16:55.0392552Z PT_DOMAIN=download.pytorch.org 2024-04-03T16:16:55.0393547Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-04-03T16:16:55.0394918Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-04-03T16:16:55.0395886Z # one is returned at random 2024-04-03T16:16:55.0396560Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-04-03T16:16:55.0397182Z  2024-04-03T16:16:55.0397560Z if [ -z "${RESOLVED_IP}" ]; then 2024-04-03T16:16:55.0398333Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-04-03T16:16:55.0399288Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-04-03T16:16:55.0399962Z  2024-04-03T16:16:55.0400355Z  if [ -z "${RESOLVED_IP}" ]; then 2024-04-03T16:16:55.0401042Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-04-03T16:16:55.0401654Z  exit 1 2024-04-03T16:16:55.0402036Z  fi 2024-04-03T16:16:55.0402378Z fi 2024-04-03T16:16:55.0402692Z  2024-04-03T16:16:55.0403130Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-04-03T16:16:55.0404132Z  # Clean up any old records first 2024-04-03T16:16:55.0404759Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-04-03T16:16:55.0405321Z fi 2024-04-03T16:16:55.0405654Z  2024-04-03T16:16:55.0406163Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-04-03T16:16:55.0406843Z cat /etc/hosts 2024-04-03T16:16:55.0415191Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:55.0415814Z env: 2024-04-03T16:16:55.0416131Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:55.0416556Z ##[endgroup] 2024-04-03T16:16:55.0441717Z + PT_DOMAIN=download.pytorch.org 2024-04-03T16:16:55.0447271Z ++ dig -4 +short download.pytorch.org 2024-04-03T16:16:55.0448124Z ++ tail -n1 2024-04-03T16:16:55.0793541Z + RESOLVED_IP=108.138.64.116 2024-04-03T16:16:55.0794074Z + '[' -z 108.138.64.116 ']' 2024-04-03T16:16:55.0794617Z + grep -r download.pytorch.org /etc/hosts 2024-04-03T16:16:55.0805869Z + echo '108.138.64.116 download.pytorch.org' 2024-04-03T16:16:55.0806493Z + sudo tee -a /etc/hosts 2024-04-03T16:16:55.0904897Z 108.138.64.116 download.pytorch.org 2024-04-03T16:16:55.0921133Z + cat /etc/hosts 2024-04-03T16:16:55.0927455Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-04-03T16:16:55.0942064Z ::1 localhost6 localhost6.localdomain6 2024-04-03T16:16:55.0942712Z 108.138.64.116 download.pytorch.org 2024-04-03T16:16:55.1132337Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-04-03T16:16:55.1133128Z with: 2024-04-03T16:16:55.1134329Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:55.1135722Z docker-build-dir: .ci/docker 2024-04-03T16:16:55.1136187Z working-directory: . 2024-04-03T16:16:55.1136754Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:55.1137419Z force-push: false 2024-04-03T16:16:55.1137802Z env: 2024-04-03T16:16:55.1138111Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:55.1138537Z ##[endgroup] 2024-04-03T16:16:55.1163394Z ##[group]Run set -ex 2024-04-03T16:16:55.1163828Z set -ex 2024-04-03T16:16:55.1164190Z  2024-04-03T16:16:55.1164869Z # If the docker build directory or the build script doesn't exist, the action will 2024-04-03T16:16:55.1166125Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-04-03T16:16:55.1167129Z # job could then download the pre-built image as usual 2024-04-03T16:16:55.1168056Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-04-03T16:16:55.1168904Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-04-03T16:16:55.1169700Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:16:55.1170385Z  2024-04-03T16:16:55.1171024Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-04-03T16:16:55.1171819Z  exit 0 2024-04-03T16:16:55.1172168Z else 2024-04-03T16:16:55.1172620Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-04-03T16:16:55.1173179Z fi 2024-04-03T16:16:55.1173492Z  2024-04-03T16:16:55.1174068Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-04-03T16:16:55.1175150Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-04-03T16:16:55.1176115Z  # use it as it is, but first let's extract the tag 2024-04-03T16:16:55.1177000Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-04-03T16:16:55.1177913Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:16:55.1178791Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:16:55.1179480Z else 2024-04-03T16:16:55.1180180Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-04-03T16:16:55.1181017Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:16:55.1182125Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:16:55.1183074Z fi 2024-04-03T16:16:55.1191115Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:55.1191746Z env: 2024-04-03T16:16:55.1192060Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:55.1192492Z REPO_NAME: pytorch 2024-04-03T16:16:55.1193750Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:55.1195088Z DOCKER_BUILD_DIR: .ci/docker 2024-04-03T16:16:55.1195888Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:55.1196617Z ##[endgroup] 2024-04-03T16:16:55.1261572Z + [[ ! -d .ci/docker ]] 2024-04-03T16:16:55.1262072Z + [[ ! -f .ci/docker/build.sh ]] 2024-04-03T16:16:55.1262555Z + echo skip=false 2024-04-03T16:16:55.1264651Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-04-03T16:16:55.1269696Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:55.1271103Z ++ awk -F '[:,]' '{print $2}' 2024-04-03T16:16:55.1280232Z + DOCKER_TAG=16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:55.1281058Z + echo docker-tag=16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:55.1282953Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:55.1318110Z ##[group]Run set +e 2024-04-03T16:16:55.1318560Z set +e 2024-04-03T16:16:55.1318924Z set -x 2024-04-03T16:16:55.1319273Z  2024-04-03T16:16:55.1319588Z login() { 2024-04-03T16:16:55.1320419Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-04-03T16:16:55.1321330Z } 2024-04-03T16:16:55.1321641Z  2024-04-03T16:16:55.1321985Z retry () { 2024-04-03T16:16:55.1322465Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-04-03T16:16:55.1323064Z } 2024-04-03T16:16:55.1323392Z  2024-04-03T16:16:55.1323769Z retry login "${DOCKER_REGISTRY}" 2024-04-03T16:16:55.1324259Z  2024-04-03T16:16:55.1324842Z # Check if image already exists, if it does then skip building it 2024-04-03T16:16:55.1325731Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-04-03T16:16:55.1326337Z  exit 0 2024-04-03T16:16:55.1326710Z fi 2024-04-03T16:16:55.1327038Z  2024-04-03T16:16:55.1327639Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-04-03T16:16:55.1328680Z # be empty. The default action would be to continue rebuild the image 2024-04-03T16:16:55.1329602Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-04-03T16:16:55.1330449Z  # if we're on the base branch then use the parent commit 2024-04-03T16:16:55.1331169Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-04-03T16:16:55.1331699Z else 2024-04-03T16:16:55.1332277Z  # otherwise we're on a PR, so use the most recent base commit 2024-04-03T16:16:55.1333118Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-04-03T16:16:55.1333742Z fi 2024-04-03T16:16:55.1334074Z  2024-04-03T16:16:55.1334437Z if [[ -z "${MERGE_BASE}" ]]; then 2024-04-03T16:16:55.1335066Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-04-03T16:16:55.1335844Z  2024-04-03T16:16:55.1336643Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-04-03T16:16:55.1337599Z  exit 0 2024-04-03T16:16:55.1337963Z fi 2024-04-03T16:16:55.1338274Z  2024-04-03T16:16:55.1338810Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-04-03T16:16:55.1340049Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-04-03T16:16:55.1341068Z  exit 1 2024-04-03T16:16:55.1341411Z fi 2024-04-03T16:16:55.1341736Z  2024-04-03T16:16:55.1342330Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-04-03T16:16:55.1343509Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-04-03T16:16:55.1344599Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-04-03T16:16:55.1346046Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-04-03T16:16:55.1347425Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-04-03T16:16:55.1348231Z fi 2024-04-03T16:16:55.1348553Z  2024-04-03T16:16:55.1349160Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-04-03T16:16:55.1356995Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:55.1357617Z env: 2024-04-03T16:16:55.1357935Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:55.1358367Z DOCKER_BUILD_DIR: .ci/docker 2024-04-03T16:16:55.1358934Z BASE_REVISION: 1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:16:55.1360358Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:55.1361780Z DOCKER_TAG: 16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:55.1362503Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:55.1363145Z ##[endgroup] 2024-04-03T16:16:55.1387752Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:55.1388458Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:55.1390184Z + aws ecr get-login-password --region us-east-1 2024-04-03T16:16:55.1391257Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:55.6158211Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-04-03T16:16:55.6160072Z Configure a credential helper to remove this warning. See 2024-04-03T16:16:55.6161245Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-04-03T16:16:55.6161956Z 2024-04-03T16:16:55.6162091Z Login Succeeded 2024-04-03T16:16:55.6174254Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:55.8592399Z { 2024-04-03T16:16:55.8593130Z "schemaVersion": 2, 2024-04-03T16:16:55.8593883Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-04-03T16:16:55.8594622Z "config": { 2024-04-03T16:16:55.8595237Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-04-03T16:16:55.8595928Z "size": 44693, 2024-04-03T16:16:55.8596618Z "digest": "sha256:f0aa5113e491b8f8ef3b51fc066bce798422ecc2bdeb4be54a8a4efa6914052e" 2024-04-03T16:16:55.8597443Z }, 2024-04-03T16:16:55.8597776Z "layers": [ 2024-04-03T16:16:55.8598096Z { 2024-04-03T16:16:55.8598648Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8599367Z "size": 28580681, 2024-04-03T16:16:55.8600066Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-04-03T16:16:55.8600857Z }, 2024-04-03T16:16:55.8601497Z { 2024-04-03T16:16:55.8602034Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8602750Z "size": 7943353, 2024-04-03T16:16:55.8603452Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-04-03T16:16:55.8604225Z }, 2024-04-03T16:16:55.8604530Z { 2024-04-03T16:16:55.8605089Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8605779Z "size": 56226389, 2024-04-03T16:16:55.8606480Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-04-03T16:16:55.8607269Z }, 2024-04-03T16:16:55.8607584Z { 2024-04-03T16:16:55.8608104Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8608823Z "size": 187, 2024-04-03T16:16:55.8609527Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-04-03T16:16:55.8610290Z }, 2024-04-03T16:16:55.8610605Z { 2024-04-03T16:16:55.8611164Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8612408Z "size": 6885, 2024-04-03T16:16:55.8613079Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-04-03T16:16:55.8613872Z }, 2024-04-03T16:16:55.8614380Z + exit 0 2024-04-03T16:16:55.8614664Z { 2024-04-03T16:16:55.8615433Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8616211Z "size": 1383662475, 2024-04-03T16:16:55.8616916Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-04-03T16:16:55.8617707Z }, 2024-04-03T16:16:55.8618006Z { 2024-04-03T16:16:55.8618514Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8619213Z "size": 62282, 2024-04-03T16:16:55.8619883Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-04-03T16:16:55.8620655Z }, 2024-04-03T16:16:55.8620939Z { 2024-04-03T16:16:55.8621497Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8622204Z "size": 1684, 2024-04-03T16:16:55.8622934Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-04-03T16:16:55.8623731Z }, 2024-04-03T16:16:55.8624036Z { 2024-04-03T16:16:55.8624558Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8625276Z "size": 1522, 2024-04-03T16:16:55.8626566Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-04-03T16:16:55.8627350Z }, 2024-04-03T16:16:55.8627656Z { 2024-04-03T16:16:55.8628204Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8628899Z "size": 2445884909, 2024-04-03T16:16:55.8629595Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-04-03T16:16:55.8630369Z }, 2024-04-03T16:16:55.8630652Z { 2024-04-03T16:16:55.8631196Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8631915Z "size": 85882, 2024-04-03T16:16:55.8632575Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-04-03T16:16:55.8633355Z }, 2024-04-03T16:16:55.8633660Z { 2024-04-03T16:16:55.8634183Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8634888Z "size": 1827, 2024-04-03T16:16:55.8635587Z "digest": "sha256:020f47d0fa1be3a2df8dd2e863841d351ab72d00c7df4a4a16879ceea1239f4b" 2024-04-03T16:16:55.8636365Z }, 2024-04-03T16:16:55.8636665Z { 2024-04-03T16:16:55.8637212Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8637899Z "size": 654809190, 2024-04-03T16:16:55.8638605Z "digest": "sha256:8b29859251bec1f67480cc3935ceb13aeb54305ec53bcb4ef1d4977fcd62e2b0" 2024-04-03T16:16:55.8639385Z }, 2024-04-03T16:16:55.8639665Z { 2024-04-03T16:16:55.8640206Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8641123Z "size": 544, 2024-04-03T16:16:55.8641766Z "digest": "sha256:0e1508d7c9ab1501f0c741f4729a53eb3758856d8761c5258f732506d73c594e" 2024-04-03T16:16:55.8642537Z }, 2024-04-03T16:16:55.8642839Z { 2024-04-03T16:16:55.8643360Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8644066Z "size": 1282, 2024-04-03T16:16:55.8644757Z "digest": "sha256:89f208ed177f5a57c0375b5e3eabab8772559309b532c759b1ce8ea6cf1b3e4b" 2024-04-03T16:16:55.8645521Z }, 2024-04-03T16:16:55.8645824Z { 2024-04-03T16:16:55.8646364Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8647045Z "size": 483, 2024-04-03T16:16:55.8647734Z "digest": "sha256:ddecf600b7ac2f21cd805972b7cb9b1305ec1846970aca6b50db2c833e5b38cb" 2024-04-03T16:16:55.8648529Z }, 2024-04-03T16:16:55.8648811Z { 2024-04-03T16:16:55.8649356Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8650066Z "size": 91705074, 2024-04-03T16:16:55.8650771Z "digest": "sha256:7965ac4a7eb7caf381dde0f1bb10190588c124b84a8b1e9c77adf5143e9dc4f9" 2024-04-03T16:16:55.8651558Z }, 2024-04-03T16:16:55.8651858Z { 2024-04-03T16:16:55.8652378Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8653080Z "size": 3255, 2024-04-03T16:16:55.8653882Z "digest": "sha256:6b9458cb1b822d8bda5cc039459fec8b5d9e46187880eddb9e529319467f189b" 2024-04-03T16:16:55.8654679Z }, 2024-04-03T16:16:55.8654983Z { 2024-04-03T16:16:55.8655525Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8656208Z "size": 1904, 2024-04-03T16:16:55.8656900Z "digest": "sha256:a40a390f581ad70eca936ae2fe3aeadb7e19642cd6211232d77be76ba9fa7dd2" 2024-04-03T16:16:55.8657695Z }, 2024-04-03T16:16:55.8657977Z { 2024-04-03T16:16:55.8658518Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8659222Z "size": 699, 2024-04-03T16:16:55.8659903Z "digest": "sha256:01d4a84baddbd86b6da32efdf2e97a1088a27be6c25e024af3e1bc20f1db320e" 2024-04-03T16:16:55.8660718Z }, 2024-04-03T16:16:55.8661021Z { 2024-04-03T16:16:55.8661541Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8662248Z "size": 2803779640, 2024-04-03T16:16:55.8662963Z "digest": "sha256:db8dcb001b22779a63f46677dafc9340d330e77f96e6dcb21cd1b1210a88c0ae" 2024-04-03T16:16:55.8663735Z }, 2024-04-03T16:16:55.8664037Z { 2024-04-03T16:16:55.8664582Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8665266Z "size": 380, 2024-04-03T16:16:55.8666427Z "digest": "sha256:3eaa6199e98399217ecd8fe5727b666c8273744f0518bc3e5b1ac140ce3d123c" 2024-04-03T16:16:55.8667227Z }, 2024-04-03T16:16:55.8667510Z { 2024-04-03T16:16:55.8668054Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8668754Z "size": 11856, 2024-04-03T16:16:55.8669420Z "digest": "sha256:dd253632212f47f8e65518f3f9c4a80baeddd5f04f7bf0eb3840a976c93b5276" 2024-04-03T16:16:55.8670220Z }, 2024-04-03T16:16:55.8670525Z { 2024-04-03T16:16:55.8671046Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8671742Z "size": 802, 2024-04-03T16:16:55.8672397Z "digest": "sha256:5e6597056347c056681d9213c22a37f96446cb7b0d9b6838b5e6101ac98c7e8a" 2024-04-03T16:16:55.8673136Z }, 2024-04-03T16:16:55.8673441Z { 2024-04-03T16:16:55.8673994Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8674682Z "size": 106, 2024-04-03T16:16:55.8675361Z "digest": "sha256:858ba61782a0452f8aa8a1cef9b4fd1f301e540347dff3ed187bf69086efb8cc" 2024-04-03T16:16:55.8676144Z }, 2024-04-03T16:16:55.8676426Z { 2024-04-03T16:16:55.8676970Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8677673Z "size": 503, 2024-04-03T16:16:55.8678317Z "digest": "sha256:9cb381d49846ff5212226715691aa2c6abc9987a95b92335e81a6d03d9c4f944" 2024-04-03T16:16:55.8679247Z }, 2024-04-03T16:16:55.8679552Z { 2024-04-03T16:16:55.8680080Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8680790Z "size": 121477305, 2024-04-03T16:16:55.8681487Z "digest": "sha256:610c26b5e9a03476b891eae042a708b3462a40bc118e1e5e472f936448896531" 2024-04-03T16:16:55.8682234Z }, 2024-04-03T16:16:55.8682537Z { 2024-04-03T16:16:55.8683091Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8683774Z "size": 109, 2024-04-03T16:16:55.8684445Z "digest": "sha256:03a7067427cf481c50c8195f331a42f0ec54f72e472ea1fdf170ad664c44530a" 2024-04-03T16:16:55.8685219Z }, 2024-04-03T16:16:55.8685502Z { 2024-04-03T16:16:55.8686045Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8686751Z "size": 557, 2024-04-03T16:16:55.8687411Z "digest": "sha256:b4f17b88d23ef498cfc2839ad0358588aaca3004f277c84f3ec7d68336482c1e" 2024-04-03T16:16:55.8688200Z }, 2024-04-03T16:16:55.8688517Z { 2024-04-03T16:16:55.8689036Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8689740Z "size": 1175558, 2024-04-03T16:16:55.8690438Z "digest": "sha256:bc6bfcbe5530b217c0792229e143c80718d420f57f5e5db203ecf7be083b4d29" 2024-04-03T16:16:55.8691226Z }, 2024-04-03T16:16:55.8691510Z { 2024-04-03T16:16:55.8692166Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8692881Z "size": 103, 2024-04-03T16:16:55.8693557Z "digest": "sha256:df877d3106b553016cfe184088c5aa43dc0e564cb21a18e430d08bbdf8277f3d" 2024-04-03T16:16:55.8694342Z }, 2024-04-03T16:16:55.8694625Z { 2024-04-03T16:16:55.8695168Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8695872Z "size": 1489, 2024-04-03T16:16:55.8696564Z "digest": "sha256:0fbc7e84a570fa00c6b12df3ffc46a2b4ff2786d9e5564886359fe90b556556d" 2024-04-03T16:16:55.8697563Z }, 2024-04-03T16:16:55.8697869Z { 2024-04-03T16:16:55.8698403Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8699106Z "size": 442747152, 2024-04-03T16:16:55.8699786Z "digest": "sha256:1570829375e6e129fa627aa95c6619f8006d73f425c85d72f4595e55a0f74e0f" 2024-04-03T16:16:55.8700541Z }, 2024-04-03T16:16:55.8700822Z { 2024-04-03T16:16:55.8701362Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8702068Z "size": 161, 2024-04-03T16:16:55.8702728Z "digest": "sha256:6281fee72d3f38f69e3d04981823f93b775ef06fd399fc2308e9d5c951ec7bc1" 2024-04-03T16:16:55.8703505Z }, 2024-04-03T16:16:55.8703809Z { 2024-04-03T16:16:55.8704330Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8705033Z "size": 564, 2024-04-03T16:16:55.8706045Z "digest": "sha256:d01dc0055b877335000d5b68ac1c5e71147255853e3de4edfa032d27feff6452" 2024-04-03T16:16:55.8706831Z }, 2024-04-03T16:16:55.8707134Z { 2024-04-03T16:16:55.8707676Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8708379Z "size": 33199039, 2024-04-03T16:16:55.8709082Z "digest": "sha256:2abb072c55408b8bf96c5a0a24c729cb31029e0d42ef0192a2eba0cab079933c" 2024-04-03T16:16:55.8709868Z }, 2024-04-03T16:16:55.8710148Z { 2024-04-03T16:16:55.8710688Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8711389Z "size": 104, 2024-04-03T16:16:55.8712040Z "digest": "sha256:1323b806817860cf618c3a24ee430b0ccdd26464c11bcc46f717b6cbf241820c" 2024-04-03T16:16:55.8712816Z }, 2024-04-03T16:16:55.8713117Z { 2024-04-03T16:16:55.8713638Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8714341Z "size": 424, 2024-04-03T16:16:55.8715042Z "digest": "sha256:473ae4beebc0bc7146babdb210cabec932238e00b4a3f74f9f0bf5d7bf577593" 2024-04-03T16:16:55.8715826Z }, 2024-04-03T16:16:55.8716130Z { 2024-04-03T16:16:55.8716663Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8717505Z "size": 20262067, 2024-04-03T16:16:55.8718211Z "digest": "sha256:289a48eb62b5823f2c7d86b1875b0edd766b78c9e0edd7e1d5f54e94f004f0dc" 2024-04-03T16:16:55.8718997Z }, 2024-04-03T16:16:55.8719280Z { 2024-04-03T16:16:55.8719817Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8720520Z "size": 439, 2024-04-03T16:16:55.8721196Z "digest": "sha256:6ac1d762661f0e7800aadeafce3ba5f8d3337ad5c27ff47253ac76bbb7895841" 2024-04-03T16:16:55.8721998Z }, 2024-04-03T16:16:55.8722301Z { 2024-04-03T16:16:55.8722870Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8723579Z "size": 699, 2024-04-03T16:16:55.8724271Z "digest": "sha256:01d4a84baddbd86b6da32efdf2e97a1088a27be6c25e024af3e1bc20f1db320e" 2024-04-03T16:16:55.8725055Z }, 2024-04-03T16:16:55.8725359Z { 2024-04-03T16:16:55.8725897Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8726581Z "size": 142, 2024-04-03T16:16:55.8727263Z "digest": "sha256:796d028fd584b098f569e5427fcf37e2248f308b02af6185104c4faeeef48b05" 2024-04-03T16:16:55.8728035Z }, 2024-04-03T16:16:55.8728315Z { 2024-04-03T16:16:55.8728853Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8729552Z "size": 136, 2024-04-03T16:16:55.8730338Z "digest": "sha256:a3890e7012d35f6e60a4f6bdd11bf6bba9b20824a9bc7e4795e3f620f8eecdd5" 2024-04-03T16:16:55.8731153Z }, 2024-04-03T16:16:55.8731456Z { 2024-04-03T16:16:55.8731975Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8732676Z "size": 32, 2024-04-03T16:16:55.8733355Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:16:55.8734123Z }, 2024-04-03T16:16:55.8734426Z { 2024-04-03T16:16:55.8734965Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8735646Z "size": 188, 2024-04-03T16:16:55.8736325Z "digest": "sha256:b3d241c09291e0fe13d64493f03dbee53b27579eaa163e4d569d67ed310775f0" 2024-04-03T16:16:55.8737122Z }, 2024-04-03T16:16:55.8737405Z { 2024-04-03T16:16:55.8737946Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8738649Z "size": 564, 2024-04-03T16:16:55.8739285Z "digest": "sha256:c584774f42519400f1d08734f82c70984be83c66222c2bcf7b6d6703895f5c79" 2024-04-03T16:16:55.8740048Z }, 2024-04-03T16:16:55.8740359Z { 2024-04-03T16:16:55.8740879Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8741591Z "size": 43165822, 2024-04-03T16:16:55.8742366Z "digest": "sha256:e4b3bc5cda0494786fe54c1838bc55bce19728edf9a927c05bd5884dcbd6e205" 2024-04-03T16:16:55.8743163Z }, 2024-04-03T16:16:55.8743444Z { 2024-04-03T16:16:55.8743985Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8744685Z "size": 106, 2024-04-03T16:16:55.8745363Z "digest": "sha256:8b7274bb07ecf3280dfc69b15646503e9f67d0cf05be86b04469183960f72e64" 2024-04-03T16:16:55.8746631Z }, 2024-04-03T16:16:55.8746941Z { 2024-04-03T16:16:55.8747465Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8748212Z "size": 1192, 2024-04-03T16:16:55.8748876Z "digest": "sha256:f2934ba32628883717478385639d4c5f0403a09363b8cf2bb73b328e3b8cc042" 2024-04-03T16:16:55.8749639Z }, 2024-04-03T16:16:55.8749920Z { 2024-04-03T16:16:55.8750473Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8751179Z "size": 699, 2024-04-03T16:16:55.8751868Z "digest": "sha256:01d4a84baddbd86b6da32efdf2e97a1088a27be6c25e024af3e1bc20f1db320e" 2024-04-03T16:16:55.8752670Z }, 2024-04-03T16:16:55.8752970Z { 2024-04-03T16:16:55.8753489Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8754194Z "size": 137, 2024-04-03T16:16:55.8754871Z "digest": "sha256:d1c6f5fd2ce358717d4f731eb8c70e2757ed206be14c1f9999718a81f29d2874" 2024-04-03T16:16:55.8755824Z }, 2024-04-03T16:16:55.8756124Z { 2024-04-03T16:16:55.8756664Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8757347Z "size": 119, 2024-04-03T16:16:55.8758028Z "digest": "sha256:9c3c29d675a6f3566456d0b187f18d1a5deac38949fea55e068dbb7bc69d8bb4" 2024-04-03T16:16:55.8758810Z }, 2024-04-03T16:16:55.8759089Z { 2024-04-03T16:16:55.8759637Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8760350Z "size": 1704547419, 2024-04-03T16:16:55.8761018Z "digest": "sha256:e400461a29326092b6477a61196ef84989ebf7c7aee931714fd37259cc862d8b" 2024-04-03T16:16:55.8761781Z }, 2024-04-03T16:16:55.8762081Z { 2024-04-03T16:16:55.8762598Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8763304Z "size": 172, 2024-04-03T16:16:55.8763989Z "digest": "sha256:f3b4b7bb5d910fb4f87f578cc5c2180774bdff4c30831c40bc5425b29658682c" 2024-04-03T16:16:55.8764754Z }, 2024-04-03T16:16:55.8765072Z { 2024-04-03T16:16:55.8765613Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8766299Z "size": 1840, 2024-04-03T16:16:55.8766990Z "digest": "sha256:d39766ef1e6b8b4b6e2ae3d7d9f23f104b3c0309eefbd8bb4e5d3e362937299a" 2024-04-03T16:16:55.8767789Z }, 2024-04-03T16:16:55.8768072Z { 2024-04-03T16:16:55.8768745Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8769480Z "size": 7529780, 2024-04-03T16:16:55.8770153Z "digest": "sha256:8ce169be4b1d2efb4c8dbe9188699e646f24d62988505b7bc263e4605d20ff32" 2024-04-03T16:16:55.8770940Z }, 2024-04-03T16:16:55.8771242Z { 2024-04-03T16:16:55.8771760Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8772461Z "size": 164, 2024-04-03T16:16:55.8773138Z "digest": "sha256:fe2cd77be091e130f7e397bb03a127255f90a88edb85e9b91957619adac1d7b7" 2024-04-03T16:16:55.8773901Z }, 2024-04-03T16:16:55.8774204Z { 2024-04-03T16:16:55.8774739Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8775433Z "size": 7938, 2024-04-03T16:16:55.8776136Z "digest": "sha256:2a5bd85dd325ef3ddf1e8d9bd46e473bae5adf46aa7df529b7d898b7c4a42add" 2024-04-03T16:16:55.8776942Z }, 2024-04-03T16:16:55.8777219Z { 2024-04-03T16:16:55.8777755Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8778474Z "size": 8070, 2024-04-03T16:16:55.8779139Z "digest": "sha256:d1190269f41e7ab64b2eadaeba8697af4731d9667e9a95ac49d95c717878351f" 2024-04-03T16:16:55.8779927Z }, 2024-04-03T16:16:55.8780229Z { 2024-04-03T16:16:55.8780746Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8781446Z "size": 302, 2024-04-03T16:16:55.8782135Z "digest": "sha256:e8752c122b392a996acd66fc47ae9c18fe3f4960cbeee8bb1eeea5ab3dfbd9c2" 2024-04-03T16:16:55.8782914Z }, 2024-04-03T16:16:55.8783214Z { 2024-04-03T16:16:55.8783756Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8784449Z "size": 7620337, 2024-04-03T16:16:55.8785156Z "digest": "sha256:1eaea607f89d48ad9275ec68ec91f10e1cb3dda60bb0189f2793cef6780ad02f" 2024-04-03T16:16:55.8786452Z }, 2024-04-03T16:16:55.8786739Z { 2024-04-03T16:16:55.8787286Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8787986Z "size": 108, 2024-04-03T16:16:55.8788640Z "digest": "sha256:9ef6078991a0e9655c1656e4b3f24cdf5377a1c1ad41e4418b4a5b912856183f" 2024-04-03T16:16:55.8789419Z }, 2024-04-03T16:16:55.8789720Z { 2024-04-03T16:16:55.8790240Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8790954Z "size": 54145458, 2024-04-03T16:16:55.8791666Z "digest": "sha256:c0dab7a64f9be557be292ba18570e24ade19f386e38b36b5dfe3a5a844ef38eb" 2024-04-03T16:16:55.8792441Z }, 2024-04-03T16:16:55.8792743Z { 2024-04-03T16:16:55.8793282Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8794143Z "size": 498, 2024-04-03T16:16:55.8794816Z "digest": "sha256:20e7673e255ed89c9eca5e8f08025a96bf06263edb006590d7ff6137cf90730f" 2024-04-03T16:16:55.8795778Z }, 2024-04-03T16:16:55.8796059Z { 2024-04-03T16:16:55.8796599Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8797302Z "size": 1498859701, 2024-04-03T16:16:55.8797983Z "digest": "sha256:865db895296b5ec9e81d86e205660c33454f259b5dd7ed88f9402cbd87699077" 2024-04-03T16:16:55.8798757Z }, 2024-04-03T16:16:55.8799058Z { 2024-04-03T16:16:55.8799575Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8800276Z "size": 106, 2024-04-03T16:16:55.8800958Z "digest": "sha256:effd5b6e0122c9c96f918d64d4e1b49efb8b08c08324eea3054750d8d2b37652" 2024-04-03T16:16:55.8801730Z }, 2024-04-03T16:16:55.8802031Z { 2024-04-03T16:16:55.8802570Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8803254Z "size": 444, 2024-04-03T16:16:55.8803929Z "digest": "sha256:6150e99963233ce451f0f0e5a16b96087b4924cdbb89bb7701503081c6ca5027" 2024-04-03T16:16:55.8804799Z }, 2024-04-03T16:16:55.8805081Z { 2024-04-03T16:16:55.8805628Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8806331Z "size": 36532201, 2024-04-03T16:16:55.8807123Z "digest": "sha256:7ac5ac4b15768fbc183814207bf88e8b69cf1655e4651c06267591dad5d842d5" 2024-04-03T16:16:55.8807930Z }, 2024-04-03T16:16:55.8808232Z { 2024-04-03T16:16:55.8808753Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8809457Z "size": 111, 2024-04-03T16:16:55.8810125Z "digest": "sha256:f31a22b80d01bb61f572065e872c8276ef92389cb05b1f1b6853fda52ce462a2" 2024-04-03T16:16:55.8810880Z }, 2024-04-03T16:16:55.8811181Z { 2024-04-03T16:16:55.8811722Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8812405Z "size": 32, 2024-04-03T16:16:55.8813088Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:16:55.8813885Z }, 2024-04-03T16:16:55.8814165Z { 2024-04-03T16:16:55.8814705Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8815399Z "size": 32, 2024-04-03T16:16:55.8816058Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:16:55.8816842Z }, 2024-04-03T16:16:55.8817146Z { 2024-04-03T16:16:55.8817668Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:55.8818368Z "size": 32, 2024-04-03T16:16:55.8819041Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:16:55.8819804Z } 2024-04-03T16:16:55.8820104Z ] 2024-04-03T16:16:55.8820400Z } 2024-04-03T16:16:55.8959278Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-04-03T16:16:55.8959948Z tag=${ECR_DOCKER_IMAGE##*/} 2024-04-03T16:16:55.8960640Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-04-03T16:16:55.8969720Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:55.8970353Z env: 2024-04-03T16:16:55.8970697Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:55.8971980Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:55.8973333Z ##[endgroup] 2024-04-03T16:16:55.9001952Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9-16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:55.9087156Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-04-03T16:16:55.9087906Z with: 2024-04-03T16:16:55.9089093Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:55.9090615Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:55.9091248Z env: 2024-04-03T16:16:55.9091766Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:55.9092206Z ##[endgroup] 2024-04-03T16:16:55.9115994Z ##[group]Run set -x 2024-04-03T16:16:55.9116410Z set -x 2024-04-03T16:16:55.9116772Z set +e 2024-04-03T16:16:55.9117130Z  2024-04-03T16:16:55.9117449Z login() { 2024-04-03T16:16:55.9118281Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-04-03T16:16:55.9119201Z } 2024-04-03T16:16:55.9119521Z  2024-04-03T16:16:55.9119916Z retry () { 2024-04-03T16:16:55.9120388Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-04-03T16:16:55.9120944Z } 2024-04-03T16:16:55.9121279Z  2024-04-03T16:16:55.9121645Z retry login "${DOCKER_REGISTRY}" 2024-04-03T16:16:55.9122162Z  2024-04-03T16:16:55.9122496Z set -e 2024-04-03T16:16:55.9123126Z # ignore output since only exit code is used for conditional 2024-04-03T16:16:55.9124028Z # only pull docker image if it's not available locally 2024-04-03T16:16:55.9124995Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-04-03T16:16:55.9125938Z  retry docker pull "${DOCKER_IMAGE}" 2024-04-03T16:16:55.9126463Z fi 2024-04-03T16:16:55.9134208Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:55.9134831Z env: 2024-04-03T16:16:55.9135172Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:55.9136435Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:55.9137919Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:55.9138565Z ##[endgroup] 2024-04-03T16:16:55.9162710Z + set +e 2024-04-03T16:16:55.9163348Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:55.9164160Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:55.9166426Z + aws ecr get-login-password --region us-east-1 2024-04-03T16:16:55.9167722Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:56.4113831Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-04-03T16:16:56.4114851Z Configure a credential helper to remove this warning. See 2024-04-03T16:16:56.4115871Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-04-03T16:16:56.4116546Z 2024-04-03T16:16:56.4116722Z Login Succeeded 2024-04-03T16:16:56.4128008Z + set -e 2024-04-03T16:16:56.4129420Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:56.4340444Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:56.4342828Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:56.6370620Z 16884143090f148aa56809ceba26eeb699b5dc49: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9 2024-04-03T16:16:56.6371637Z 7a2c55901189: Pulling fs layer 2024-04-03T16:16:56.6372107Z 09d415c238d7: Pulling fs layer 2024-04-03T16:16:56.6372599Z 9fe6e2e61518: Pulling fs layer 2024-04-03T16:16:56.6373140Z 41f16248e682: Pulling fs layer 2024-04-03T16:16:56.6374317Z 95d7b7817039: Pulling fs layer 2024-04-03T16:16:56.6374832Z 8f6c90485347: Pulling fs layer 2024-04-03T16:16:56.6375414Z ab17245097e4: Pulling fs layer 2024-04-03T16:16:56.6376075Z dfecd7e9912b: Pulling fs layer 2024-04-03T16:16:56.6376593Z 464a8f745445: Pulling fs layer 2024-04-03T16:16:56.6377090Z d67c111fa588: Pulling fs layer 2024-04-03T16:16:56.6377504Z 0723abde19c0: Pulling fs layer 2024-04-03T16:16:56.6377965Z 020f47d0fa1b: Pulling fs layer 2024-04-03T16:16:56.6378621Z 8b29859251be: Pulling fs layer 2024-04-03T16:16:56.6379088Z 41f16248e682: Waiting 2024-04-03T16:16:56.6379965Z 0e1508d7c9ab: Pulling fs layer 2024-04-03T16:16:56.6380471Z 89f208ed177f: Pulling fs layer 2024-04-03T16:16:56.6380933Z ddecf600b7ac: Pulling fs layer 2024-04-03T16:16:56.6381635Z 7965ac4a7eb7: Pulling fs layer 2024-04-03T16:16:56.6382084Z 95d7b7817039: Waiting 2024-04-03T16:16:56.6382647Z 6b9458cb1b82: Pulling fs layer 2024-04-03T16:16:56.6383292Z a40a390f581a: Pulling fs layer 2024-04-03T16:16:56.6383933Z 89f208ed177f: Waiting 2024-04-03T16:16:56.6384344Z 01d4a84baddb: Pulling fs layer 2024-04-03T16:16:56.6384705Z ddecf600b7ac: Waiting 2024-04-03T16:16:56.6385123Z db8dcb001b22: Pulling fs layer 2024-04-03T16:16:56.6385870Z dfecd7e9912b: Waiting 2024-04-03T16:16:56.6386389Z 3eaa6199e983: Pulling fs layer 2024-04-03T16:16:56.6386861Z dd253632212f: Pulling fs layer 2024-04-03T16:16:56.6387302Z 5e6597056347: Pulling fs layer 2024-04-03T16:16:56.6387758Z 858ba61782a0: Pulling fs layer 2024-04-03T16:16:56.6388212Z 7965ac4a7eb7: Waiting 2024-04-03T16:16:56.6388588Z 01d4a84baddb: Waiting 2024-04-03T16:16:56.6388995Z 9cb381d49846: Pulling fs layer 2024-04-03T16:16:56.6389444Z 6b9458cb1b82: Waiting 2024-04-03T16:16:56.6389856Z 610c26b5e9a0: Pulling fs layer 2024-04-03T16:16:56.6390274Z 3eaa6199e983: Waiting 2024-04-03T16:16:56.6390701Z a40a390f581a: Waiting 2024-04-03T16:16:56.6391091Z 03a7067427cf: Pulling fs layer 2024-04-03T16:16:56.6391518Z b4f17b88d23e: Pulling fs layer 2024-04-03T16:16:56.6392011Z 464a8f745445: Waiting 2024-04-03T16:16:56.6392355Z d67c111fa588: Waiting 2024-04-03T16:16:56.6392799Z bc6bfcbe5530: Pulling fs layer 2024-04-03T16:16:56.6393182Z db8dcb001b22: Waiting 2024-04-03T16:16:56.6393590Z df877d3106b5: Pulling fs layer 2024-04-03T16:16:56.6394051Z 0fbc7e84a570: Pulling fs layer 2024-04-03T16:16:56.6394488Z 1570829375e6: Pulling fs layer 2024-04-03T16:16:56.6394921Z 8b29859251be: Waiting 2024-04-03T16:16:56.6395298Z 03a7067427cf: Waiting 2024-04-03T16:16:56.6395675Z b4f17b88d23e: Waiting 2024-04-03T16:16:56.6396061Z df877d3106b5: Waiting 2024-04-03T16:16:56.6396465Z 6281fee72d3f: Pulling fs layer 2024-04-03T16:16:56.6396889Z 0fbc7e84a570: Waiting 2024-04-03T16:16:56.6397276Z 0723abde19c0: Waiting 2024-04-03T16:16:56.6397662Z 9cb381d49846: Waiting 2024-04-03T16:16:56.6398028Z 020f47d0fa1b: Waiting 2024-04-03T16:16:56.6398412Z 858ba61782a0: Waiting 2024-04-03T16:16:56.6398784Z 1570829375e6: Waiting 2024-04-03T16:16:56.6399165Z d01dc0055b87: Pulling fs layer 2024-04-03T16:16:56.6399629Z 2abb072c5540: Pulling fs layer 2024-04-03T16:16:56.6400078Z 6281fee72d3f: Waiting 2024-04-03T16:16:56.6400464Z 1323b8068178: Pulling fs layer 2024-04-03T16:16:56.6400936Z dd253632212f: Waiting 2024-04-03T16:16:56.6401353Z d01dc0055b87: Waiting 2024-04-03T16:16:56.6401725Z 473ae4beebc0: Pulling fs layer 2024-04-03T16:16:56.6402199Z 289a48eb62b5: Pulling fs layer 2024-04-03T16:16:56.6402613Z 1323b8068178: Waiting 2024-04-03T16:16:56.6402993Z 473ae4beebc0: Waiting 2024-04-03T16:16:56.6403394Z 610c26b5e9a0: Waiting 2024-04-03T16:16:56.6403830Z 6ac1d762661f: Pulling fs layer 2024-04-03T16:16:56.6404241Z 796d028fd584: Pulling fs layer 2024-04-03T16:16:56.6404687Z a3890e7012d3: Pulling fs layer 2024-04-03T16:16:56.6405148Z 4f4fb700ef54: Pulling fs layer 2024-04-03T16:16:56.6405574Z 796d028fd584: Waiting 2024-04-03T16:16:56.6406012Z b3d241c09291: Pulling fs layer 2024-04-03T16:16:56.6406409Z a3890e7012d3: Waiting 2024-04-03T16:16:56.6406773Z b3d241c09291: Waiting 2024-04-03T16:16:56.6407173Z c584774f4251: Pulling fs layer 2024-04-03T16:16:56.6407800Z e4b3bc5cda04: Pulling fs layer 2024-04-03T16:16:56.6408265Z 6ac1d762661f: Waiting 2024-04-03T16:16:56.6408675Z 8b7274bb07ec: Pulling fs layer 2024-04-03T16:16:56.6409140Z f2934ba32628: Pulling fs layer 2024-04-03T16:16:56.6409582Z d1c6f5fd2ce3: Pulling fs layer 2024-04-03T16:16:56.6410025Z f2934ba32628: Waiting 2024-04-03T16:16:56.6410426Z 9c3c29d675a6: Pulling fs layer 2024-04-03T16:16:56.6410868Z e400461a2932: Pulling fs layer 2024-04-03T16:16:56.6411450Z d1c6f5fd2ce3: Waiting 2024-04-03T16:16:56.6411886Z 9c3c29d675a6: Waiting 2024-04-03T16:16:56.6412238Z f3b4b7bb5d91: Pulling fs layer 2024-04-03T16:16:56.6412702Z d39766ef1e6b: Pulling fs layer 2024-04-03T16:16:56.6413144Z 289a48eb62b5: Waiting 2024-04-03T16:16:56.6413518Z f3b4b7bb5d91: Waiting 2024-04-03T16:16:56.6413921Z 8ce169be4b1d: Pulling fs layer 2024-04-03T16:16:56.6414426Z fe2cd77be091: Pulling fs layer 2024-04-03T16:16:56.6414817Z d39766ef1e6b: Waiting 2024-04-03T16:16:56.6415223Z 2a5bd85dd325: Pulling fs layer 2024-04-03T16:16:56.6415690Z d1190269f41e: Pulling fs layer 2024-04-03T16:16:56.6416133Z e8752c122b39: Pulling fs layer 2024-04-03T16:16:56.6416574Z d1190269f41e: Waiting 2024-04-03T16:16:56.6416980Z 1eaea607f89d: Pulling fs layer 2024-04-03T16:16:56.6417402Z e8752c122b39: Waiting 2024-04-03T16:16:56.6417841Z 9ef6078991a0: Pulling fs layer 2024-04-03T16:16:56.6418265Z c0dab7a64f9b: Pulling fs layer 2024-04-03T16:16:56.6418709Z 20e7673e255e: Pulling fs layer 2024-04-03T16:16:56.6419209Z c0dab7a64f9b: Waiting 2024-04-03T16:16:56.6419568Z 865db895296b: Pulling fs layer 2024-04-03T16:16:56.6419993Z 9ef6078991a0: Waiting 2024-04-03T16:16:56.6420401Z effd5b6e0122: Pulling fs layer 2024-04-03T16:16:56.6420845Z 20e7673e255e: Waiting 2024-04-03T16:16:56.6421227Z 6150e9996323: Pulling fs layer 2024-04-03T16:16:56.6421682Z 7ac5ac4b1576: Pulling fs layer 2024-04-03T16:16:56.6422142Z f31a22b80d01: Pulling fs layer 2024-04-03T16:16:56.6422568Z 7ac5ac4b1576: Waiting 2024-04-03T16:16:56.6423007Z fe2cd77be091: Waiting 2024-04-03T16:16:56.6423407Z f31a22b80d01: Waiting 2024-04-03T16:16:56.7996728Z 09d415c238d7: Verifying Checksum 2024-04-03T16:16:56.7997301Z 09d415c238d7: Download complete 2024-04-03T16:16:56.9277428Z 41f16248e682: Verifying Checksum 2024-04-03T16:16:56.9277964Z 41f16248e682: Download complete 2024-04-03T16:16:56.9910541Z 7a2c55901189: Verifying Checksum 2024-04-03T16:16:57.0203284Z 95d7b7817039: Download complete 2024-04-03T16:16:57.1115206Z ab17245097e4: Download complete 2024-04-03T16:16:57.2029809Z dfecd7e9912b: Download complete 2024-04-03T16:16:57.2984599Z 9fe6e2e61518: Verifying Checksum 2024-04-03T16:16:57.3482097Z 9fe6e2e61518: Download complete 2024-04-03T16:16:57.3502318Z 464a8f745445: Verifying Checksum 2024-04-03T16:16:57.3502848Z 464a8f745445: Download complete 2024-04-03T16:16:57.4555729Z 0723abde19c0: Verifying Checksum 2024-04-03T16:16:57.4556302Z 0723abde19c0: Download complete 2024-04-03T16:16:57.5429327Z 020f47d0fa1b: Verifying Checksum 2024-04-03T16:16:57.5429907Z 020f47d0fa1b: Download complete 2024-04-03T16:16:57.7681628Z 7a2c55901189: Pull complete 2024-04-03T16:16:58.0531952Z 09d415c238d7: Pull complete 2024-04-03T16:16:59.1011360Z 9fe6e2e61518: Pull complete 2024-04-03T16:16:59.2394680Z 41f16248e682: Pull complete 2024-04-03T16:16:59.3450600Z 95d7b7817039: Pull complete 2024-04-03T16:17:08.9425564Z 8b29859251be: Verifying Checksum 2024-04-03T16:17:08.9426770Z 8b29859251be: Download complete 2024-04-03T16:17:09.0576550Z 0e1508d7c9ab: Verifying Checksum 2024-04-03T16:17:09.0577141Z 0e1508d7c9ab: Download complete 2024-04-03T16:17:09.1391891Z 89f208ed177f: Verifying Checksum 2024-04-03T16:17:09.1392369Z 89f208ed177f: Download complete 2024-04-03T16:17:09.2535252Z ddecf600b7ac: Verifying Checksum 2024-04-03T16:17:09.2535777Z ddecf600b7ac: Download complete 2024-04-03T16:17:12.4802710Z 7965ac4a7eb7: Verifying Checksum 2024-04-03T16:17:12.4803308Z 7965ac4a7eb7: Download complete 2024-04-03T16:17:12.5695465Z 6b9458cb1b82: Verifying Checksum 2024-04-03T16:17:12.5695913Z 6b9458cb1b82: Download complete 2024-04-03T16:17:12.6572768Z a40a390f581a: Download complete 2024-04-03T16:17:12.7397629Z 01d4a84baddb: Download complete 2024-04-03T16:17:15.5834670Z 8f6c90485347: Download complete 2024-04-03T16:17:15.6771293Z 3eaa6199e983: Download complete 2024-04-03T16:17:15.7653424Z dd253632212f: Verifying Checksum 2024-04-03T16:17:15.7653995Z dd253632212f: Download complete 2024-04-03T16:17:15.8436555Z 5e6597056347: Verifying Checksum 2024-04-03T16:17:15.8437396Z 5e6597056347: Download complete 2024-04-03T16:17:15.9353972Z 858ba61782a0: Download complete 2024-04-03T16:17:16.0218663Z 9cb381d49846: Verifying Checksum 2024-04-03T16:17:16.0221104Z 9cb381d49846: Download complete 2024-04-03T16:17:19.1943398Z 610c26b5e9a0: Verifying Checksum 2024-04-03T16:17:19.1943955Z 610c26b5e9a0: Download complete 2024-04-03T16:17:19.2923529Z 03a7067427cf: Verifying Checksum 2024-04-03T16:17:19.2924319Z 03a7067427cf: Download complete 2024-04-03T16:17:19.3785016Z b4f17b88d23e: Verifying Checksum 2024-04-03T16:17:19.3785585Z b4f17b88d23e: Download complete 2024-04-03T16:17:19.5932723Z bc6bfcbe5530: Download complete 2024-04-03T16:17:19.6829589Z df877d3106b5: Verifying Checksum 2024-04-03T16:17:19.6830178Z df877d3106b5: Download complete 2024-04-03T16:17:19.7618350Z 0fbc7e84a570: Verifying Checksum 2024-04-03T16:17:19.7618902Z 0fbc7e84a570: Download complete 2024-04-03T16:17:30.3349607Z 1570829375e6: Verifying Checksum 2024-04-03T16:17:30.3350225Z 1570829375e6: Download complete 2024-04-03T16:17:30.4272284Z 6281fee72d3f: Verifying Checksum 2024-04-03T16:17:30.4273394Z 6281fee72d3f: Download complete 2024-04-03T16:17:30.5317344Z d01dc0055b87: Download complete 2024-04-03T16:17:31.3004079Z 2abb072c5540: Verifying Checksum 2024-04-03T16:17:31.3004700Z 2abb072c5540: Download complete 2024-04-03T16:17:31.3943218Z 1323b8068178: Verifying Checksum 2024-04-03T16:17:31.3943801Z 1323b8068178: Download complete 2024-04-03T16:17:31.4798619Z 473ae4beebc0: Verifying Checksum 2024-04-03T16:17:31.4799782Z 473ae4beebc0: Download complete 2024-04-03T16:17:31.9976047Z d67c111fa588: Verifying Checksum 2024-04-03T16:17:31.9976662Z d67c111fa588: Download complete 2024-04-03T16:17:32.0359591Z 289a48eb62b5: Verifying Checksum 2024-04-03T16:17:32.0360612Z 289a48eb62b5: Download complete 2024-04-03T16:17:32.1123505Z 796d028fd584: Verifying Checksum 2024-04-03T16:17:32.1124092Z 796d028fd584: Download complete 2024-04-03T16:17:32.1178897Z 6ac1d762661f: Verifying Checksum 2024-04-03T16:17:32.1179422Z 6ac1d762661f: Download complete 2024-04-03T16:17:32.1296467Z 4f4fb700ef54: Verifying Checksum 2024-04-03T16:17:32.1296961Z 4f4fb700ef54: Download complete 2024-04-03T16:17:32.2308899Z b3d241c09291: Verifying Checksum 2024-04-03T16:17:32.2309814Z b3d241c09291: Download complete 2024-04-03T16:17:32.2741524Z a3890e7012d3: Verifying Checksum 2024-04-03T16:17:32.2742097Z a3890e7012d3: Download complete 2024-04-03T16:17:32.3635030Z c584774f4251: Download complete 2024-04-03T16:17:32.4518893Z 8b7274bb07ec: Verifying Checksum 2024-04-03T16:17:32.4519535Z 8b7274bb07ec: Download complete 2024-04-03T16:17:32.5423751Z f2934ba32628: Download complete 2024-04-03T16:17:32.6247721Z d1c6f5fd2ce3: Verifying Checksum 2024-04-03T16:17:32.6248383Z d1c6f5fd2ce3: Download complete 2024-04-03T16:17:32.7326870Z 9c3c29d675a6: Verifying Checksum 2024-04-03T16:17:32.7327419Z 9c3c29d675a6: Download complete 2024-04-03T16:17:33.4491349Z e4b3bc5cda04: Verifying Checksum 2024-04-03T16:17:33.4491891Z e4b3bc5cda04: Download complete 2024-04-03T16:17:33.6120509Z f3b4b7bb5d91: Download complete 2024-04-03T16:17:33.6868319Z d39766ef1e6b: Download complete 2024-04-03T16:17:34.0520789Z 8ce169be4b1d: Verifying Checksum 2024-04-03T16:17:34.0521704Z 8ce169be4b1d: Download complete 2024-04-03T16:17:34.1340227Z fe2cd77be091: Verifying Checksum 2024-04-03T16:17:34.1341164Z fe2cd77be091: Download complete 2024-04-03T16:17:34.2753863Z 2a5bd85dd325: Download complete 2024-04-03T16:17:34.3630378Z d1190269f41e: Verifying Checksum 2024-04-03T16:17:34.3631466Z d1190269f41e: Download complete 2024-04-03T16:17:34.4393394Z e8752c122b39: Verifying Checksum 2024-04-03T16:17:34.4394672Z e8752c122b39: Download complete 2024-04-03T16:17:34.7699708Z 1eaea607f89d: Verifying Checksum 2024-04-03T16:17:34.7700677Z 1eaea607f89d: Download complete 2024-04-03T16:17:34.8521337Z 9ef6078991a0: Verifying Checksum 2024-04-03T16:17:34.8522414Z 9ef6078991a0: Download complete 2024-04-03T16:17:34.9294708Z 8f6c90485347: Pull complete 2024-04-03T16:17:36.6696470Z c0dab7a64f9b: Verifying Checksum 2024-04-03T16:17:36.6697350Z c0dab7a64f9b: Download complete 2024-04-03T16:17:36.7964067Z 20e7673e255e: Download complete 2024-04-03T16:17:36.8378022Z ab17245097e4: Pull complete 2024-04-03T16:17:38.7177804Z dfecd7e9912b: Pull complete 2024-04-03T16:17:40.5973417Z 464a8f745445: Pull complete 2024-04-03T16:18:12.9122381Z 865db895296b: Verifying Checksum 2024-04-03T16:18:12.9122973Z 865db895296b: Download complete 2024-04-03T16:18:12.9952582Z effd5b6e0122: Verifying Checksum 2024-04-03T16:18:12.9953137Z effd5b6e0122: Download complete 2024-04-03T16:18:13.0843656Z 6150e9996323: Verifying Checksum 2024-04-03T16:18:13.0844236Z 6150e9996323: Download complete 2024-04-03T16:18:14.0436721Z 7ac5ac4b1576: Verifying Checksum 2024-04-03T16:18:14.0437290Z 7ac5ac4b1576: Download complete 2024-04-03T16:18:14.1501686Z f31a22b80d01: Verifying Checksum 2024-04-03T16:18:14.1502266Z f31a22b80d01: Download complete 2024-04-03T16:18:14.8379127Z d67c111fa588: Pull complete 2024-04-03T16:18:16.3438840Z e400461a2932: Verifying Checksum 2024-04-03T16:18:16.3439463Z e400461a2932: Download complete 2024-04-03T16:18:16.7100198Z 0723abde19c0: Pull complete 2024-04-03T16:18:18.7394046Z 020f47d0fa1b: Pull complete 2024-04-03T16:18:20.2772283Z db8dcb001b22: Verifying Checksum 2024-04-03T16:18:20.2772829Z db8dcb001b22: Download complete 2024-04-03T16:18:32.0317316Z 8b29859251be: Pull complete 2024-04-03T16:18:34.3134307Z 0e1508d7c9ab: Pull complete 2024-04-03T16:18:36.2083345Z 89f208ed177f: Pull complete 2024-04-03T16:18:38.0569728Z ddecf600b7ac: Pull complete 2024-04-03T16:18:42.4795808Z 7965ac4a7eb7: Pull complete 2024-04-03T16:18:42.6238314Z 6b9458cb1b82: Pull complete 2024-04-03T16:18:42.7275280Z a40a390f581a: Pull complete 2024-04-03T16:18:42.8186105Z 01d4a84baddb: Pull complete 2024-04-03T16:19:28.8377928Z db8dcb001b22: Pull complete 2024-04-03T16:19:30.7033644Z 3eaa6199e983: Pull complete 2024-04-03T16:19:33.0275280Z dd253632212f: Pull complete 2024-04-03T16:19:34.8616177Z 5e6597056347: Pull complete 2024-04-03T16:19:36.7065656Z 858ba61782a0: Pull complete 2024-04-03T16:19:38.5770937Z 9cb381d49846: Pull complete 2024-04-03T16:19:44.3896311Z 610c26b5e9a0: Pull complete 2024-04-03T16:19:46.5365609Z 03a7067427cf: Pull complete 2024-04-03T16:19:48.7824018Z b4f17b88d23e: Pull complete 2024-04-03T16:19:51.9290705Z bc6bfcbe5530: Pull complete 2024-04-03T16:19:55.8229210Z df877d3106b5: Pull complete 2024-04-03T16:19:58.2984236Z 0fbc7e84a570: Pull complete 2024-04-03T16:20:05.6176709Z 1570829375e6: Pull complete 2024-04-03T16:20:05.7395219Z 6281fee72d3f: Pull complete 2024-04-03T16:20:05.8314291Z d01dc0055b87: Pull complete 2024-04-03T16:20:06.6643987Z 2abb072c5540: Pull complete 2024-04-03T16:20:06.7671092Z 1323b8068178: Pull complete 2024-04-03T16:20:06.8640798Z 473ae4beebc0: Pull complete 2024-04-03T16:20:07.2277452Z 289a48eb62b5: Pull complete 2024-04-03T16:20:07.3321665Z 6ac1d762661f: Pull complete 2024-04-03T16:20:07.5481329Z 796d028fd584: Pull complete 2024-04-03T16:20:07.6453417Z a3890e7012d3: Pull complete 2024-04-03T16:20:07.7240723Z 4f4fb700ef54: Pull complete 2024-04-03T16:20:07.8432353Z b3d241c09291: Pull complete 2024-04-03T16:20:07.9394799Z c584774f4251: Pull complete 2024-04-03T16:20:09.9399892Z e4b3bc5cda04: Pull complete 2024-04-03T16:20:12.9597857Z 8b7274bb07ec: Pull complete 2024-04-03T16:20:15.2247043Z f2934ba32628: Pull complete 2024-04-03T16:20:19.2594945Z d1c6f5fd2ce3: Pull complete 2024-04-03T16:20:21.3946367Z 9c3c29d675a6: Pull complete 2024-04-03T16:20:53.3693872Z e400461a2932: Pull complete 2024-04-03T16:20:53.4752919Z f3b4b7bb5d91: Pull complete 2024-04-03T16:20:53.5908678Z d39766ef1e6b: Pull complete 2024-04-03T16:20:53.8433864Z 8ce169be4b1d: Pull complete 2024-04-03T16:20:53.9349137Z fe2cd77be091: Pull complete 2024-04-03T16:20:54.0308839Z 2a5bd85dd325: Pull complete 2024-04-03T16:20:54.1278955Z d1190269f41e: Pull complete 2024-04-03T16:20:54.2160331Z e8752c122b39: Pull complete 2024-04-03T16:20:56.1625524Z 1eaea607f89d: Pull complete 2024-04-03T16:20:58.2744305Z 9ef6078991a0: Pull complete 2024-04-03T16:21:02.1868206Z c0dab7a64f9b: Pull complete 2024-04-03T16:21:04.0641004Z 20e7673e255e: Pull complete 2024-04-03T16:21:25.2093221Z 865db895296b: Pull complete 2024-04-03T16:21:27.2345282Z effd5b6e0122: Pull complete 2024-04-03T16:21:29.3807501Z 6150e9996323: Pull complete 2024-04-03T16:21:32.5086952Z 7ac5ac4b1576: Pull complete 2024-04-03T16:21:33.1836578Z f31a22b80d01: Pull complete 2024-04-03T16:21:33.4901733Z Digest: sha256:68e0ca2e38253fb3bd10c84d380560df0602da1b16bccdfd7d4dd7ac22529759 2024-04-03T16:21:33.4937561Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:21:33.4969010Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:21:33.5155816Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-04-03T16:21:33.5156512Z with: 2024-04-03T16:21:33.5156854Z driver-version: 535.54.03 2024-04-03T16:21:33.5157299Z env: 2024-04-03T16:21:33.5157614Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:21:33.5158042Z ##[endgroup] 2024-04-03T16:21:33.5212324Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-04-03T16:21:33.5213037Z with: 2024-04-03T16:21:33.5213352Z timeout_minutes: 10 2024-04-03T16:21:33.5213755Z max_attempts: 3 2024-04-03T16:21:33.5254370Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-04-03T16:21:33.5295296Z retry_wait_seconds: 10 2024-04-03T16:21:33.5295752Z polling_interval_seconds: 1 2024-04-03T16:21:33.5296219Z warning_on_retry: true 2024-04-03T16:21:33.5296634Z continue_on_error: false 2024-04-03T16:21:33.5297042Z env: 2024-04-03T16:21:33.5297370Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:21:33.5297789Z DRIVER_VERSION: 535.54.03 2024-04-03T16:21:33.5298219Z ##[endgroup] 2024-04-03T16:21:33.6059169Z == Installing nvidia driver NVIDIA-Linux-x86_64-535.54.03.run == 2024-04-03T16:21:33.6061370Z + pre_install_nvidia_driver_amzn2 2024-04-03T16:21:33.6062019Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-04-03T16:21:34.1228716Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-04-03T16:21:34.1806085Z No Match for argument: nvidia-driver-latest-dkms 2024-04-03T16:21:34.2174303Z No Packages marked for removal 2024-04-03T16:21:34.2328997Z + install_nvidia_driver_common 2024-04-03T16:21:34.2332933Z + echo 'Before installing NVIDIA driver' 2024-04-03T16:21:34.2333808Z + lspci 2024-04-03T16:21:34.2335747Z Before installing NVIDIA driver 2024-04-03T16:21:34.2551820Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-04-03T16:21:34.2552850Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-04-03T16:21:34.2553877Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-04-03T16:21:34.2554836Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-04-03T16:21:34.2555710Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-04-03T16:21:34.2556588Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-04-03T16:21:34.2557597Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-04-03T16:21:34.2558992Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-04-03T16:21:34.2560052Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-04-03T16:21:34.2560802Z + lsmod 2024-04-03T16:21:34.2573576Z Module Size Used by 2024-04-03T16:21:34.2574110Z backlight 16384 0 2024-04-03T16:21:34.2574636Z xt_conntrack 16384 1 2024-04-03T16:21:34.2575102Z ipt_MASQUERADE 16384 1 2024-04-03T16:21:34.2575663Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-04-03T16:21:34.2576254Z nf_conntrack_netlink 49152 0 2024-04-03T16:21:34.2576804Z nfnetlink 16384 2 nf_conntrack_netlink 2024-04-03T16:21:34.2577576Z xfrm_user 45056 1 2024-04-03T16:21:34.2578091Z xfrm_algo 16384 1 xfrm_user 2024-04-03T16:21:34.2578660Z iptable_nat 16384 1 2024-04-03T16:21:34.2579506Z nf_conntrack_ipv4 16384 3 2024-04-03T16:21:34.2580646Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-04-03T16:21:34.2581772Z nf_nat_ipv4 16384 1 iptable_nat 2024-04-03T16:21:34.2582384Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-04-03T16:21:34.2583644Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-04-03T16:21:34.2584691Z xt_addrtype 16384 2 2024-04-03T16:21:34.2585147Z iptable_filter 16384 1 2024-04-03T16:21:34.2586079Z br_netfilter 24576 0 2024-04-03T16:21:34.2586716Z bridge 172032 1 br_netfilter 2024-04-03T16:21:34.2587251Z stp 16384 1 bridge 2024-04-03T16:21:34.2587795Z llc 16384 2 bridge,stp 2024-04-03T16:21:34.2588325Z overlay 86016 0 2024-04-03T16:21:34.2588735Z sunrpc 393216 1 2024-04-03T16:21:34.2589194Z dm_mirror 28672 0 2024-04-03T16:21:34.2589702Z dm_region_hash 20480 1 dm_mirror 2024-04-03T16:21:34.2590321Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-04-03T16:21:34.2590947Z dm_mod 143360 2 dm_log,dm_mirror 2024-04-03T16:21:34.2591508Z dax 69632 1 dm_mod 2024-04-03T16:21:34.2592006Z sb_edac 24576 0 2024-04-03T16:21:34.2592452Z crc32_pclmul 16384 0 2024-04-03T16:21:34.2592925Z ghash_clmulni_intel 16384 0 2024-04-03T16:21:34.2593410Z pcbc 16384 0 2024-04-03T16:21:34.2593853Z aesni_intel 188416 0 2024-04-03T16:21:34.2594346Z aes_x86_64 20480 1 aesni_intel 2024-04-03T16:21:34.2594906Z crypto_simd 16384 1 aesni_intel 2024-04-03T16:21:34.2595456Z glue_helper 16384 1 aesni_intel 2024-04-03T16:21:34.2596159Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-04-03T16:21:34.2596848Z mousedev 24576 0 2024-04-03T16:21:34.2597297Z psmouse 32768 0 2024-04-03T16:21:34.2597778Z pcc_cpufreq 16384 0 2024-04-03T16:21:34.2598240Z evdev 20480 3 2024-04-03T16:21:34.2598677Z button 16384 0 2024-04-03T16:21:34.2599129Z ena 139264 0 2024-04-03T16:21:34.2599589Z ptp 20480 1 ena 2024-04-03T16:21:34.2600064Z pps_core 20480 1 ptp 2024-04-03T16:21:34.2600555Z ata_piix 36864 0 2024-04-03T16:21:34.2601019Z xen_blkfront 49152 2 2024-04-03T16:21:34.2601483Z libata 266240 1 ata_piix 2024-04-03T16:21:34.2602043Z crc32c_intel 24576 0 2024-04-03T16:21:34.2602523Z scsi_mod 245760 1 libata 2024-04-03T16:21:34.2603026Z autofs4 49152 2 2024-04-03T16:21:34.2603459Z + modinfo nvidia 2024-04-03T16:21:34.2604307Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-04-03T16:21:34.2605199Z firmware: nvidia/535.54.03/gsp_tu10x.bin 2024-04-03T16:21:34.2606008Z firmware: nvidia/535.54.03/gsp_ga10x.bin 2024-04-03T16:21:34.2606642Z alias: char-major-195-* 2024-04-03T16:21:34.2607124Z version: 535.54.03 2024-04-03T16:21:34.2607692Z supported: external 2024-04-03T16:21:34.2608131Z license: NVIDIA 2024-04-03T16:21:34.2608571Z srcversion: EA9C7EF32617E104C8240C4 2024-04-03T16:21:34.2609263Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-04-03T16:21:34.2609895Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-04-03T16:21:34.2610511Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-04-03T16:21:34.2611109Z depends: i2c-core,drm 2024-04-03T16:21:34.2611564Z retpoline: Y 2024-04-03T16:21:34.2611946Z name: nvidia 2024-04-03T16:21:34.2612766Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-04-03T16:21:34.2613629Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-04-03T16:21:34.2614453Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-04-03T16:21:34.2615201Z parm: NVreg_ResmanDebugLevel:int 2024-04-03T16:21:34.2615760Z parm: NVreg_RmLogonRC:int 2024-04-03T16:21:34.2616300Z parm: NVreg_ModifyDeviceFiles:int 2024-04-03T16:21:34.2616857Z parm: NVreg_DeviceFileUID:int 2024-04-03T16:21:34.2617407Z parm: NVreg_DeviceFileGID:int 2024-04-03T16:21:34.2617956Z parm: NVreg_DeviceFileMode:int 2024-04-03T16:21:34.2618605Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-04-03T16:21:34.2619325Z parm: NVreg_UsePageAttributeTable:int 2024-04-03T16:21:34.2619929Z parm: NVreg_EnablePCIeGen3:int 2024-04-03T16:21:34.2620455Z parm: NVreg_EnableMSI:int 2024-04-03T16:21:34.2620977Z parm: NVreg_TCEBypassMode:int 2024-04-03T16:21:34.2621558Z parm: NVreg_EnableStreamMemOPs:int 2024-04-03T16:21:34.2622213Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-04-03T16:21:34.2622950Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-04-03T16:21:34.2623673Z parm: NVreg_EnableS0ixPowerManagement:int 2024-04-03T16:21:34.2624433Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-04-03T16:21:34.2625202Z parm: NVreg_DynamicPowerManagement:int 2024-04-03T16:21:34.2626348Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-04-03T16:21:34.2627092Z parm: NVreg_EnableGpuFirmware:int 2024-04-03T16:21:34.2627711Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-04-03T16:21:34.2628383Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-04-03T16:21:34.2629076Z parm: NVreg_EnableUserNUMAManagement:int 2024-04-03T16:21:34.2629685Z parm: NVreg_MemoryPoolSize:int 2024-04-03T16:21:34.2630274Z parm: NVreg_KMallocHeapMaxSize:int 2024-04-03T16:21:34.2630886Z parm: NVreg_VMallocHeapMaxSize:int 2024-04-03T16:21:34.2631467Z parm: NVreg_IgnoreMMIOCheck:int 2024-04-03T16:21:34.2632026Z parm: NVreg_NvLinkDisable:int 2024-04-03T16:21:34.2632672Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-04-03T16:21:34.2633321Z parm: NVreg_RegisterPCIDriver:int 2024-04-03T16:21:34.2633921Z parm: NVreg_EnableResizableBar:int 2024-04-03T16:21:34.2634531Z parm: NVreg_EnableDbgBreakpoint:int 2024-04-03T16:21:34.2635112Z parm: NVreg_RegistryDwords:charp 2024-04-03T16:21:34.2635737Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-04-03T16:21:34.2636345Z parm: NVreg_RmMsg:charp 2024-04-03T16:21:34.2636843Z parm: NVreg_GpuBlacklist:charp 2024-04-03T16:21:34.2637427Z parm: NVreg_TemporaryFilePath:charp 2024-04-03T16:21:34.2638019Z parm: NVreg_ExcludedGpus:charp 2024-04-03T16:21:34.2638628Z parm: NVreg_DmaRemapPeerMmio:int 2024-04-03T16:21:34.2639234Z parm: NVreg_RmNvlinkBandwidth:charp 2024-04-03T16:21:34.2639817Z parm: rm_firmware_active:charp 2024-04-03T16:21:34.2640317Z + HAS_NVIDIA_DRIVER=0 2024-04-03T16:21:34.2640975Z ++ command -v nvidia-smi 2024-04-03T16:21:34.2641469Z + '[' -x /usr/bin/nvidia-smi ']' 2024-04-03T16:21:34.2641910Z + set +e 2024-04-03T16:21:34.2642542Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-04-03T16:21:37.8379530Z + INSTALLED_DRIVER_VERSION=535.54.03 2024-04-03T16:21:37.8380809Z + NVIDIA_SMI_STATUS=0 2024-04-03T16:21:37.8381950Z + '[' 0 -ne 0 ']' 2024-04-03T16:21:37.8382512Z + '[' 535.54.03 '!=' 535.54.03 ']' 2024-04-03T16:21:37.8382997Z + HAS_NVIDIA_DRIVER=1 2024-04-03T16:21:37.8384255Z + echo 'NVIDIA driver (535.54.03) has already been installed. Skipping NVIDIA driver installation' 2024-04-03T16:21:37.8385918Z + set -e 2024-04-03T16:21:37.8386304Z + '[' 1 -eq 0 ']' 2024-04-03T16:21:37.8387360Z NVIDIA driver (535.54.03) has already been installed. Skipping NVIDIA driver installation 2024-04-03T16:21:37.8388264Z + post_install_nvidia_driver_common 2024-04-03T16:21:37.8388779Z + sudo modprobe nvidia 2024-04-03T16:21:37.8510968Z + echo 'After installing NVIDIA driver' 2024-04-03T16:21:37.8511982Z + lspci 2024-04-03T16:21:37.8513149Z After installing NVIDIA driver 2024-04-03T16:21:37.8707776Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-04-03T16:21:37.8709122Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-04-03T16:21:37.8710133Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-04-03T16:21:37.8711065Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-04-03T16:21:37.8711880Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-04-03T16:21:37.8712749Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-04-03T16:21:37.8713785Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-04-03T16:21:37.8714869Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-04-03T16:21:37.8715917Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-04-03T16:21:37.8716664Z + lsmod 2024-04-03T16:21:37.8729725Z Module Size Used by 2024-04-03T16:21:37.8730356Z nvidia_uvm 1466368 0 2024-04-03T16:21:37.8730811Z nvidia 56422400 1 nvidia_uvm 2024-04-03T16:21:37.8731397Z drm 421888 1 nvidia 2024-04-03T16:21:37.8731932Z i2c_core 77824 2 nvidia,drm 2024-04-03T16:21:37.8732430Z backlight 16384 0 2024-04-03T16:21:37.8732860Z xt_conntrack 16384 1 2024-04-03T16:21:37.8733303Z ipt_MASQUERADE 16384 1 2024-04-03T16:21:37.8733855Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-04-03T16:21:37.8734430Z nf_conntrack_netlink 49152 0 2024-04-03T16:21:37.8735009Z nfnetlink 16384 2 nf_conntrack_netlink 2024-04-03T16:21:37.8735638Z xfrm_user 45056 1 2024-04-03T16:21:37.8736113Z xfrm_algo 16384 1 xfrm_user 2024-04-03T16:21:37.8736635Z iptable_nat 16384 1 2024-04-03T16:21:37.8737129Z nf_conntrack_ipv4 16384 3 2024-04-03T16:21:37.8737654Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-04-03T16:21:37.8738257Z nf_nat_ipv4 16384 1 iptable_nat 2024-04-03T16:21:37.8738972Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-04-03T16:21:37.8740269Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-04-03T16:21:37.8741359Z xt_addrtype 16384 2 2024-04-03T16:21:37.8741835Z iptable_filter 16384 1 2024-04-03T16:21:37.8742299Z br_netfilter 24576 0 2024-04-03T16:21:37.8742818Z bridge 172032 1 br_netfilter 2024-04-03T16:21:37.8743372Z stp 16384 1 bridge 2024-04-03T16:21:37.8743906Z llc 16384 2 bridge,stp 2024-04-03T16:21:37.8744326Z overlay 86016 0 2024-04-03T16:21:37.8744880Z sunrpc 393216 1 2024-04-03T16:21:37.8745341Z dm_mirror 28672 0 2024-04-03T16:21:37.8746405Z dm_region_hash 20480 1 dm_mirror 2024-04-03T16:21:37.8747118Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-04-03T16:21:37.8747656Z dm_mod 143360 2 dm_log,dm_mirror 2024-04-03T16:21:37.8748200Z dax 69632 1 dm_mod 2024-04-03T16:21:37.8748696Z sb_edac 24576 0 2024-04-03T16:21:37.8749159Z crc32_pclmul 16384 0 2024-04-03T16:21:37.8749616Z ghash_clmulni_intel 16384 0 2024-04-03T16:21:37.8750092Z pcbc 16384 0 2024-04-03T16:21:37.8750552Z aesni_intel 188416 0 2024-04-03T16:21:37.8751029Z aes_x86_64 20480 1 aesni_intel 2024-04-03T16:21:37.8751591Z crypto_simd 16384 1 aesni_intel 2024-04-03T16:21:37.8752303Z glue_helper 16384 1 aesni_intel 2024-04-03T16:21:37.8753021Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-04-03T16:21:37.8753713Z mousedev 24576 0 2024-04-03T16:21:37.8754192Z psmouse 32768 0 2024-04-03T16:21:37.8754739Z pcc_cpufreq 16384 0 2024-04-03T16:21:37.8755137Z evdev 20480 3 2024-04-03T16:21:37.8755648Z button 16384 0 2024-04-03T16:21:37.8756177Z ena 139264 0 2024-04-03T16:21:37.8756582Z ptp 20480 1 ena 2024-04-03T16:21:37.8757079Z pps_core 20480 1 ptp 2024-04-03T16:21:37.8757688Z ata_piix 36864 0 2024-04-03T16:21:37.8758157Z xen_blkfront 49152 2 2024-04-03T16:21:37.8758641Z libata 266240 1 ata_piix 2024-04-03T16:21:37.8759140Z crc32c_intel 24576 0 2024-04-03T16:21:37.8759612Z scsi_mod 245760 1 libata 2024-04-03T16:21:37.8760109Z autofs4 49152 2 2024-04-03T16:21:37.8760549Z + modinfo nvidia 2024-04-03T16:21:37.8761446Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-04-03T16:21:37.8762329Z firmware: nvidia/535.54.03/gsp_tu10x.bin 2024-04-03T16:21:37.8762936Z firmware: nvidia/535.54.03/gsp_ga10x.bin 2024-04-03T16:21:37.8763560Z alias: char-major-195-* 2024-04-03T16:21:37.8764040Z version: 535.54.03 2024-04-03T16:21:37.8764455Z supported: external 2024-04-03T16:21:37.8764878Z license: NVIDIA 2024-04-03T16:21:37.8765313Z srcversion: EA9C7EF32617E104C8240C4 2024-04-03T16:21:37.8765891Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-04-03T16:21:37.8766518Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-04-03T16:21:37.8767141Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-04-03T16:21:37.8767740Z depends: i2c-core,drm 2024-04-03T16:21:37.8768188Z retpoline: Y 2024-04-03T16:21:37.8768564Z name: nvidia 2024-04-03T16:21:37.8769251Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-04-03T16:21:37.8770083Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-04-03T16:21:37.8770905Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-04-03T16:21:37.8771674Z parm: NVreg_ResmanDebugLevel:int 2024-04-03T16:21:37.8772214Z parm: NVreg_RmLogonRC:int 2024-04-03T16:21:37.8772752Z parm: NVreg_ModifyDeviceFiles:int 2024-04-03T16:21:37.8773330Z parm: NVreg_DeviceFileUID:int 2024-04-03T16:21:37.8773856Z parm: NVreg_DeviceFileGID:int 2024-04-03T16:21:37.8774410Z parm: NVreg_DeviceFileMode:int 2024-04-03T16:21:37.8775079Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-04-03T16:21:37.8775778Z parm: NVreg_UsePageAttributeTable:int 2024-04-03T16:21:37.8776382Z parm: NVreg_EnablePCIeGen3:int 2024-04-03T16:21:37.8776927Z parm: NVreg_EnableMSI:int 2024-04-03T16:21:37.8777434Z parm: NVreg_TCEBypassMode:int 2024-04-03T16:21:37.8778010Z parm: NVreg_EnableStreamMemOPs:int 2024-04-03T16:21:37.8778681Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-04-03T16:21:37.8779576Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-04-03T16:21:37.8780285Z parm: NVreg_EnableS0ixPowerManagement:int 2024-04-03T16:21:37.8781068Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-04-03T16:21:37.8781812Z parm: NVreg_DynamicPowerManagement:int 2024-04-03T16:21:37.8782589Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-04-03T16:21:37.8783347Z parm: NVreg_EnableGpuFirmware:int 2024-04-03T16:21:37.8783948Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-04-03T16:21:37.8784624Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-04-03T16:21:37.8785319Z parm: NVreg_EnableUserNUMAManagement:int 2024-04-03T16:21:37.8786256Z parm: NVreg_MemoryPoolSize:int 2024-04-03T16:21:37.8786972Z parm: NVreg_KMallocHeapMaxSize:int 2024-04-03T16:21:37.8787612Z parm: NVreg_VMallocHeapMaxSize:int 2024-04-03T16:21:37.8788189Z parm: NVreg_IgnoreMMIOCheck:int 2024-04-03T16:21:37.8788760Z parm: NVreg_NvLinkDisable:int 2024-04-03T16:21:37.8789399Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-04-03T16:21:37.8790045Z parm: NVreg_RegisterPCIDriver:int 2024-04-03T16:21:37.8790648Z parm: NVreg_EnableResizableBar:int 2024-04-03T16:21:37.8791264Z parm: NVreg_EnableDbgBreakpoint:int 2024-04-03T16:21:37.8791867Z parm: NVreg_RegistryDwords:charp 2024-04-03T16:21:37.8792476Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-04-03T16:21:37.8793075Z parm: NVreg_RmMsg:charp 2024-04-03T16:21:37.8793586Z parm: NVreg_GpuBlacklist:charp 2024-04-03T16:21:37.8794155Z parm: NVreg_TemporaryFilePath:charp 2024-04-03T16:21:37.8794742Z parm: NVreg_ExcludedGpus:charp 2024-04-03T16:21:37.8795320Z parm: NVreg_DmaRemapPeerMmio:int 2024-04-03T16:21:37.8795911Z parm: NVreg_RmNvlinkBandwidth:charp 2024-04-03T16:21:37.8796494Z parm: rm_firmware_active:charp 2024-04-03T16:21:37.8797005Z + set +e 2024-04-03T16:21:37.8797373Z + nvidia-smi 2024-04-03T16:21:41.6195299Z Wed Apr 3 16:21:41 2024 2024-04-03T16:21:41.6196591Z +---------------------------------------------------------------------------------------+ 2024-04-03T16:21:41.6197655Z | NVIDIA-SMI 535.54.03 Driver Version: 535.54.03 CUDA Version: 12.2 | 2024-04-03T16:21:41.6198634Z |-----------------------------------------+----------------------+----------------------+ 2024-04-03T16:21:41.6199639Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-04-03T16:21:41.6200755Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-04-03T16:21:41.6201659Z | | | MIG M. | 2024-04-03T16:21:41.6202331Z |=========================================+======================+======================| 2024-04-03T16:21:41.6241937Z | 0 Tesla M60 Off | 00000000:00:1D.0 Off | 0 | 2024-04-03T16:21:41.6243236Z | N/A 24C P0 37W / 150W | 0MiB / 7680MiB | 0% Default | 2024-04-03T16:21:41.6244030Z | | | N/A | 2024-04-03T16:21:41.6244882Z +-----------------------------------------+----------------------+----------------------+ 2024-04-03T16:21:41.6288338Z | 1 Tesla M60 Off | 00000000:00:1E.0 Off | 0 | 2024-04-03T16:21:41.6289913Z | N/A 30C P0 37W / 150W | 0MiB / 7680MiB | 100% Default | 2024-04-03T16:21:41.6290700Z | | | N/A | 2024-04-03T16:21:41.6291555Z +-----------------------------------------+----------------------+----------------------+ 2024-04-03T16:21:41.6292297Z 2024-04-03T16:21:41.6294432Z +---------------------------------------------------------------------------------------+ 2024-04-03T16:21:41.6295229Z | Processes: | 2024-04-03T16:21:41.6296122Z | GPU GI CI PID Type Process name GPU Memory | 2024-04-03T16:21:41.6297002Z | ID ID Usage | 2024-04-03T16:21:41.6297702Z |=======================================================================================| 2024-04-03T16:21:41.6298532Z | No running processes found | 2024-04-03T16:21:41.6299688Z +---------------------------------------------------------------------------------------+ 2024-04-03T16:21:42.1635118Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-04-03T16:21:44.2050537Z Tesla M60 2024-04-03T16:21:45.0371886Z + NVIDIA_SMI_STATUS=0 2024-04-03T16:21:45.0373099Z + '[' 0 -eq 0 ']' 2024-04-03T16:21:45.0373613Z + echo 'INFO: Ignoring allowed status 0' 2024-04-03T16:21:45.0374131Z + set -e 2024-04-03T16:21:45.0374493Z INFO: Ignoring allowed status 0 2024-04-03T16:21:45.0378194Z == Installing nvidia container toolkit for amzn2 == 2024-04-03T16:21:45.0381749Z + sudo yum install -y yum-utils 2024-04-03T16:21:45.5559912Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-04-03T16:21:47.2602248Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-04-03T16:21:47.2603156Z Nothing to do 2024-04-03T16:21:47.4126172Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-04-03T16:21:47.9352028Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-04-03T16:21:47.9655099Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-04-03T16:21:47.9656835Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-04-03T16:21:47.9658096Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-04-03T16:21:47.9800313Z + sudo yum install -y nvidia-docker2 2024-04-03T16:21:48.4790760Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-04-03T16:21:50.1778215Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-04-03T16:21:50.1778995Z Nothing to do 2024-04-03T16:21:50.3285260Z + sudo systemctl restart docker 2024-04-03T16:21:57.6286407Z Command completed after 1 attempt(s). 2024-04-03T16:21:57.6398210Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-04-03T16:21:57.6399250Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-04-03T16:21:57.6400139Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-04-03T16:21:57.6400985Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:21:57.6409851Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:21:57.6410515Z env: 2024-04-03T16:21:57.6410871Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:21:57.6411436Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:21:57.6412056Z ##[endgroup] 2024-04-03T16:21:58.2985050Z Defaulting to user installation because normal site-packages is not writeable 2024-04-03T16:21:58.7506315Z Collecting psutil==5.9.1 2024-04-03T16:21:58.7745825Z Downloading psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-04-03T16:21:58.8322092Z Collecting nvidia-ml-py==11.525.84 2024-04-03T16:21:58.8378034Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-04-03T16:21:58.8865548Z Installing collected packages: psutil, nvidia-ml-py 2024-04-03T16:21:59.0763881Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-04-03T16:21:59.2956232Z Prepare all required actions 2024-04-03T16:21:59.2956899Z Getting action download info 2024-04-03T16:21:59.4293989Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-04-03T16:21:59.6465043Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-04-03T16:21:59.8016607Z ##[group]Run ./.github/actions/download-build-artifacts 2024-04-03T16:21:59.8017215Z with: 2024-04-03T16:21:59.8017585Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-04-03T16:21:59.8018134Z s3-bucket: gha-artifacts 2024-04-03T16:21:59.8018546Z env: 2024-04-03T16:21:59.8018866Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:21:59.8019430Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:21:59.8020035Z ##[endgroup] 2024-04-03T16:21:59.8063971Z ##[group]Run seemethere/download-artifact-s3@v4 2024-04-03T16:21:59.8064510Z with: 2024-04-03T16:21:59.8064958Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-04-03T16:21:59.8065508Z s3-bucket: gha-artifacts 2024-04-03T16:21:59.8066298Z region: us-east-1 2024-04-03T16:21:59.8066681Z env: 2024-04-03T16:21:59.8066990Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:21:59.8067546Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:21:59.8068147Z ##[endgroup] 2024-04-03T16:22:00.4168852Z (node:55275) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-04-03T16:22:00.4170637Z 2024-04-03T16:22:00.4171362Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-04-03T16:22:00.4172845Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-04-03T16:22:00.4174869Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-04-03T16:22:00.5411130Z Found 1 objects with prefix pytorch/pytorch/8536950919/linux-focal-cuda11.8-py3.10-gcc9/ 2024-04-03T16:22:00.5412470Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-04-03T16:22:23.7082819Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-04-03T16:22:23.7093805Z Artifact download has finished successfully 2024-04-03T16:22:23.7299053Z ##[group]Run unzip -o artifacts.zip 2024-04-03T16:22:23.7299601Z unzip -o artifacts.zip 2024-04-03T16:22:23.7308661Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:22:23.7309289Z env: 2024-04-03T16:22:23.7309633Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:23.7310181Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:23.7310796Z ##[endgroup] 2024-04-03T16:22:23.7373162Z Archive: artifacts.zip 2024-04-03T16:22:23.7375166Z creating: dist/ 2024-04-03T16:22:26.1675545Z inflating: dist/torch-2.4.0a0+gitd253bd7-cp310-cp310-linux_x86_64.whl 2024-04-03T16:22:26.1676400Z creating: build/custom_test_artifacts/ 2024-04-03T16:22:26.1677146Z creating: build/custom_test_artifacts/custom-op-build/ 2024-04-03T16:22:26.1678163Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-04-03T16:22:26.1679458Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-04-03T16:22:26.1686263Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-04-03T16:22:26.1687482Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-04-03T16:22:26.1688683Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-04-03T16:22:26.1690420Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-04-03T16:22:26.1691685Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-04-03T16:22:26.1693117Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-04-03T16:22:26.1694802Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-04-03T16:22:26.1696138Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-04-03T16:22:26.1697633Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-04-03T16:22:26.1699131Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-04-03T16:22:26.1700618Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-04-03T16:22:26.1702099Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-04-03T16:22:26.1703569Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-04-03T16:22:26.1705090Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-04-03T16:22:26.1707045Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-04-03T16:22:26.1708391Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-04-03T16:22:26.1709701Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-04-03T16:22:26.1764415Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-04-03T16:22:26.1766279Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-04-03T16:22:26.1768165Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-04-03T16:22:26.1770105Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-04-03T16:22:26.1771999Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-04-03T16:22:26.1773802Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-04-03T16:22:26.1775627Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-04-03T16:22:26.1777451Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-04-03T16:22:26.1779248Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-04-03T16:22:26.1820826Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-04-03T16:22:26.1871198Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-04-03T16:22:26.1873017Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-04-03T16:22:26.1874676Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-04-03T16:22:26.1876306Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-04-03T16:22:26.1877895Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-04-03T16:22:26.1879503Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-04-03T16:22:26.1881044Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-04-03T16:22:26.1882623Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-04-03T16:22:26.1954847Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-04-03T16:22:26.2030838Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-04-03T16:22:26.2032570Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-04-03T16:22:26.2033860Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-04-03T16:22:26.2034979Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-04-03T16:22:26.2036151Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-04-03T16:22:26.2037332Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-04-03T16:22:26.2038668Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-04-03T16:22:26.2040162Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-04-03T16:22:26.2041625Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-04-03T16:22:26.2042971Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-04-03T16:22:26.2044375Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-04-03T16:22:26.2045769Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-04-03T16:22:26.2047145Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-04-03T16:22:26.2048537Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-04-03T16:22:26.2049915Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-04-03T16:22:26.2065633Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-04-03T16:22:26.2232674Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-04-03T16:22:26.2233986Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-04-03T16:22:26.2235401Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-04-03T16:22:26.2236961Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-04-03T16:22:26.2238478Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-04-03T16:22:26.2239895Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-04-03T16:22:26.2241399Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-04-03T16:22:26.2242869Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-04-03T16:22:26.2244343Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-04-03T16:22:26.2245826Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-04-03T16:22:26.2247273Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-04-03T16:22:26.2266022Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-04-03T16:22:26.2369372Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-04-03T16:22:26.2370925Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-03T16:22:26.2372327Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-04-03T16:22:26.2373751Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-04-03T16:22:26.2374964Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-04-03T16:22:26.2376281Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-04-03T16:22:26.2377430Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-04-03T16:22:26.2378526Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-04-03T16:22:26.2379532Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-04-03T16:22:26.2380575Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-04-03T16:22:26.2518711Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-04-03T16:22:26.2596643Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-04-03T16:22:26.2597591Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-04-03T16:22:26.2598469Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-04-03T16:22:26.2599524Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-04-03T16:22:26.2606791Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-04-03T16:22:26.2607988Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-04-03T16:22:26.2609165Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-04-03T16:22:26.2610424Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-04-03T16:22:26.2611667Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-04-03T16:22:26.2613079Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-04-03T16:22:26.2614478Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-04-03T16:22:26.2615773Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-04-03T16:22:26.2617047Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-04-03T16:22:26.2618518Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-04-03T16:22:26.2620009Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-04-03T16:22:26.2621470Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-04-03T16:22:26.2622935Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-04-03T16:22:26.2624407Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-04-03T16:22:26.2626270Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-04-03T16:22:26.2627693Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-04-03T16:22:26.2628978Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-04-03T16:22:26.2684361Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-04-03T16:22:26.2686209Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-04-03T16:22:26.2688067Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-04-03T16:22:26.2690595Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-04-03T16:22:26.2692582Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-04-03T16:22:26.2694547Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-04-03T16:22:26.2696488Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-04-03T16:22:26.2698292Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-04-03T16:22:26.2700076Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-04-03T16:22:26.2740862Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-04-03T16:22:26.2791184Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-04-03T16:22:26.2792993Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-04-03T16:22:26.2794634Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-04-03T16:22:26.2796256Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-04-03T16:22:26.2797834Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-04-03T16:22:26.2799424Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-04-03T16:22:26.2800948Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-04-03T16:22:26.2802517Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-04-03T16:22:26.2874884Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-04-03T16:22:26.2951034Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-04-03T16:22:26.2952563Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-04-03T16:22:26.2953836Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-04-03T16:22:26.2954937Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-04-03T16:22:26.2956078Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-04-03T16:22:26.2957291Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-04-03T16:22:26.2958664Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-04-03T16:22:26.2960197Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-04-03T16:22:26.2961669Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-04-03T16:22:26.2963052Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-04-03T16:22:26.2964506Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-04-03T16:22:26.2965943Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-04-03T16:22:26.2967363Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-04-03T16:22:26.2968794Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-04-03T16:22:26.2970213Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-04-03T16:22:26.2986826Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-04-03T16:22:26.3066803Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-04-03T16:22:26.3068385Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-03T16:22:26.3069966Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-04-03T16:22:26.3071197Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-04-03T16:22:26.3072350Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-04-03T16:22:26.3073492Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-04-03T16:22:26.3074772Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-04-03T16:22:26.3076032Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-04-03T16:22:26.3077028Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-04-03T16:22:26.3078054Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-04-03T16:22:26.3137728Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-04-03T16:22:26.3138707Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-04-03T16:22:26.3139676Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-04-03T16:22:26.3140811Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-04-03T16:22:26.3148146Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-04-03T16:22:26.3149425Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-04-03T16:22:26.3150716Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-04-03T16:22:26.3152074Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-04-03T16:22:26.3153415Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-04-03T16:22:26.3154925Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-04-03T16:22:26.3156450Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-04-03T16:22:26.3157797Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-04-03T16:22:26.3159157Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-04-03T16:22:26.3160731Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-04-03T16:22:26.3162325Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-04-03T16:22:26.3163859Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-04-03T16:22:26.3165430Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-04-03T16:22:26.3167027Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-04-03T16:22:26.3168623Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-04-03T16:22:26.3170027Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-04-03T16:22:26.3171408Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-04-03T16:22:26.3225037Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-04-03T16:22:26.3227205Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-04-03T16:22:26.3229341Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-04-03T16:22:26.3231542Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-04-03T16:22:26.3233526Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-04-03T16:22:26.3235395Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-04-03T16:22:26.3237291Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-04-03T16:22:26.3239206Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-04-03T16:22:26.3241158Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-04-03T16:22:26.3281668Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-04-03T16:22:26.3331507Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-04-03T16:22:26.3333397Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-04-03T16:22:26.3335160Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-04-03T16:22:26.3336856Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-04-03T16:22:26.3338509Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-04-03T16:22:26.3340207Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-04-03T16:22:26.3341889Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-04-03T16:22:26.3343552Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-04-03T16:22:26.3414632Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-04-03T16:22:26.3490915Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-04-03T16:22:26.3492541Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-04-03T16:22:26.3493891Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-04-03T16:22:26.3495085Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-04-03T16:22:26.3496344Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-04-03T16:22:26.3497655Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-04-03T16:22:26.3499100Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-04-03T16:22:26.3500739Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-04-03T16:22:26.3502310Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-04-03T16:22:26.3503797Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-04-03T16:22:26.3505320Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-04-03T16:22:26.3507285Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-04-03T16:22:26.3508863Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-04-03T16:22:26.3510520Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-04-03T16:22:26.3512044Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-04-03T16:22:26.3513666Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-04-03T16:22:26.3661660Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-04-03T16:22:26.3663162Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-04-03T16:22:26.3664706Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-04-03T16:22:26.3666668Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-04-03T16:22:26.3668343Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-04-03T16:22:26.3669884Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-04-03T16:22:26.3671491Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-04-03T16:22:26.3673105Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-04-03T16:22:26.3674728Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-04-03T16:22:26.3676325Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-04-03T16:22:26.3677933Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-04-03T16:22:26.3694312Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-04-03T16:22:26.3763016Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-04-03T16:22:26.3764686Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-03T16:22:26.3766173Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-04-03T16:22:26.3767508Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-04-03T16:22:26.3768730Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-04-03T16:22:26.3769970Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-04-03T16:22:26.3771216Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-04-03T16:22:26.3772400Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-04-03T16:22:26.3773452Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-04-03T16:22:26.3774559Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-04-03T16:22:26.3900278Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-04-03T16:22:26.3953315Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-04-03T16:22:26.3954112Z creating: build/lib/ 2024-04-03T16:22:26.3965300Z inflating: build/lib/libpthreadpool.a 2024-04-03T16:22:26.3966237Z inflating: build/lib/libclog.a 2024-04-03T16:22:26.3969227Z inflating: build/lib/libnnpack_reference_layers.a 2024-04-03T16:22:26.3978812Z inflating: build/lib/libcpuinfo_internals.a 2024-04-03T16:22:26.3989117Z inflating: build/lib/libcpuinfo.a 2024-04-03T16:22:26.4099588Z inflating: build/lib/libprotobuf-lite.a 2024-04-03T16:22:26.4667606Z inflating: build/lib/libprotobuf.a 2024-04-03T16:22:26.4748513Z inflating: build/lib/libgtest.a 2024-04-03T16:22:26.4841553Z inflating: build/lib/libbenchmark.a 2024-04-03T16:22:26.4850974Z inflating: build/lib/libittnotify.a 2024-04-03T16:22:26.4932099Z inflating: build/lib/libasmjit.a 2024-04-03T16:22:26.4966720Z inflating: build/lib/libtensorpipe_uv.a 2024-04-03T16:22:26.5124957Z inflating: build/lib/libgloo.a 2024-04-03T16:22:26.5152105Z inflating: build/lib/libfmt.a 2024-04-03T16:22:26.5154383Z inflating: build/lib/libcaffe2_nvrtc.so 2024-04-03T16:22:26.5265367Z inflating: build/lib/libc10.so 2024-04-03T16:22:26.5266238Z inflating: build/lib/libfoxi_loader.a 2024-04-03T16:22:26.5268803Z inflating: build/lib/libtorch_global_deps.so 2024-04-03T16:22:26.5286116Z inflating: build/lib/libqnnpack.a 2024-04-03T16:22:26.5310457Z inflating: build/lib/libpytorch_qnnpack.a 2024-04-03T16:22:26.5937094Z inflating: build/lib/libprotoc.a 2024-04-03T16:22:26.5959748Z inflating: build/lib/libgmock.a 2024-04-03T16:22:26.5960336Z inflating: build/lib/libgtest_main.a 2024-04-03T16:22:26.5961208Z inflating: build/lib/libbenchmark_main.a 2024-04-03T16:22:26.5983965Z inflating: build/lib/libnnpack.a 2024-04-03T16:22:27.7900615Z inflating: build/lib/libdnnl.a 2024-04-03T16:22:27.8613599Z inflating: build/lib/libtensorpipe.a 2024-04-03T16:22:27.8680598Z inflating: build/lib/libc10_cuda.so 2024-04-03T16:22:28.0239007Z inflating: build/lib/libfbgemm.a 2024-04-03T16:22:28.0239632Z inflating: build/lib/libgmock_main.a 2024-04-03T16:22:28.0557788Z inflating: build/lib/libtensorpipe_cuda.a 2024-04-03T16:22:28.1193820Z inflating: build/lib/libkineto.a 2024-04-03T16:22:28.1677585Z inflating: build/lib/libgloo_cuda.a 2024-04-03T16:22:28.1725912Z inflating: build/lib/libcaffe2_protos.a 2024-04-03T16:22:28.1777608Z inflating: build/lib/libonnx_proto.a 2024-04-03T16:22:28.2006979Z inflating: build/lib/libXNNPACK.a 2024-04-03T16:22:28.2848858Z inflating: build/lib/libonnx.a 2024-04-03T16:22:31.2021883Z inflating: build/lib/libtorch_cpu.so 2024-04-03T16:22:31.2026934Z inflating: build/lib/libshm.so 2024-04-03T16:22:31.2032617Z inflating: build/lib/libunbox_lib.a 2024-04-03T16:22:33.6256735Z inflating: build/lib/libtorch_cuda.so 2024-04-03T16:22:33.6257866Z inflating: build/lib/libtorch.so 2024-04-03T16:22:33.6261131Z inflating: build/lib/libc10d_cuda_test.so 2024-04-03T16:22:34.7627607Z inflating: build/lib/libtorch_cuda_linalg.so 2024-04-03T16:22:34.7652888Z inflating: build/lib/libjitbackend_test.so 2024-04-03T16:22:34.7724234Z inflating: build/lib/libtorchbind_test.so 2024-04-03T16:22:34.7755569Z inflating: build/lib/libbackend_with_compiler.so 2024-04-03T16:22:35.0094435Z inflating: build/lib/libtorch_python.so 2024-04-03T16:22:35.0136668Z inflating: build/lib/libnnapi_backend.so 2024-04-03T16:22:35.0137270Z creating: build/bin/ 2024-04-03T16:22:35.0199816Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-04-03T16:22:35.0263838Z inflating: build/bin/c10_DeviceGuard_test 2024-04-03T16:22:35.0326644Z inflating: build/bin/c10_Device_test 2024-04-03T16:22:35.0400546Z inflating: build/bin/c10_DispatchKeySet_test 2024-04-03T16:22:35.0461371Z inflating: build/bin/c10_StreamGuard_test 2024-04-03T16:22:35.0528383Z inflating: build/bin/c10_Scalar_test 2024-04-03T16:22:35.0592149Z inflating: build/bin/c10_SymInt_test 2024-04-03T16:22:35.0659885Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-04-03T16:22:35.0729689Z inflating: build/bin/c10_InlineStreamGuard_test 2024-04-03T16:22:35.0799921Z inflating: build/bin/c10_SizesAndStrides_test 2024-04-03T16:22:35.0888632Z inflating: build/bin/c10_cow_test 2024-04-03T16:22:35.0955200Z inflating: build/bin/c10_Bitset_test 2024-04-03T16:22:35.1016140Z inflating: build/bin/c10_ConstexprCrc_test 2024-04-03T16:22:35.1078452Z inflating: build/bin/c10_DeadlockDetection_test 2024-04-03T16:22:35.1142127Z inflating: build/bin/c10_Half_test 2024-04-03T16:22:35.1212240Z inflating: build/bin/c10_LeftRight_test 2024-04-03T16:22:35.1281067Z inflating: build/bin/c10_Metaprogramming_test 2024-04-03T16:22:35.1343397Z inflating: build/bin/c10_Synchronized_test 2024-04-03T16:22:35.1413007Z inflating: build/bin/c10_ThreadLocal_test 2024-04-03T16:22:35.1477481Z inflating: build/bin/c10_TypeIndex_test 2024-04-03T16:22:35.1541125Z inflating: build/bin/c10_TypeList_test 2024-04-03T16:22:35.1602387Z inflating: build/bin/c10_TypeTraits_test 2024-04-03T16:22:35.1666956Z inflating: build/bin/c10_accumulate_test 2024-04-03T16:22:35.1736088Z inflating: build/bin/c10_bfloat16_test 2024-04-03T16:22:35.1799766Z inflating: build/bin/c10_bit_cast_test 2024-04-03T16:22:35.1870199Z inflating: build/bin/c10_complex_math_test 2024-04-03T16:22:35.1939569Z inflating: build/bin/c10_complex_test 2024-04-03T16:22:35.2005515Z inflating: build/bin/c10_exception_test 2024-04-03T16:22:35.2068415Z inflating: build/bin/c10_flags_test 2024-04-03T16:22:35.2130722Z inflating: build/bin/c10_generic_math_test 2024-04-03T16:22:35.2194346Z inflating: build/bin/c10_irange_test 2024-04-03T16:22:35.2400114Z inflating: build/bin/c10_intrusive_ptr_test 2024-04-03T16:22:35.2469654Z inflating: build/bin/c10_logging_test 2024-04-03T16:22:35.2563094Z inflating: build/bin/c10_optional_test 2024-04-03T16:22:35.2641034Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-04-03T16:22:35.2708376Z inflating: build/bin/c10_registry_test 2024-04-03T16:22:35.2897025Z inflating: build/bin/c10_small_vector_test 2024-04-03T16:22:35.2961498Z inflating: build/bin/c10_ssize_test 2024-04-03T16:22:35.3026290Z inflating: build/bin/c10_string_util_test 2024-04-03T16:22:35.3089251Z inflating: build/bin/c10_tempfile_test 2024-04-03T16:22:35.3162557Z inflating: build/bin/c10_string_view_test 2024-04-03T16:22:35.3223030Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-04-03T16:22:35.3293109Z inflating: build/bin/c10_typeid_test 2024-04-03T16:22:35.3850057Z inflating: build/bin/protoc-3.13.0.0 2024-04-03T16:22:35.4406312Z inflating: build/bin/protoc 2024-04-03T16:22:35.4472576Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-04-03T16:22:35.4538383Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-04-03T16:22:35.4604371Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-04-03T16:22:35.4669545Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-04-03T16:22:35.4735276Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-04-03T16:22:35.4801514Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-04-03T16:22:35.4867158Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-04-03T16:22:35.4928287Z inflating: build/bin/c10_cuda_CUDATest 2024-04-03T16:22:35.5339076Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-04-03T16:22:35.5784818Z inflating: build/bin/vec_test_all_types_AVX2 2024-04-03T16:22:35.6214724Z inflating: build/bin/vec_test_all_types_AVX512 2024-04-03T16:22:35.6280867Z inflating: build/bin/FileStoreTest 2024-04-03T16:22:35.6350254Z inflating: build/bin/TCPStoreTest 2024-04-03T16:22:35.6416747Z inflating: build/bin/HashStoreTest 2024-04-03T16:22:35.6433863Z inflating: build/bin/ProcessGroupMPITest 2024-04-03T16:22:35.6439100Z inflating: build/bin/torch_shm_manager 2024-04-03T16:22:35.6506022Z inflating: build/bin/test_edge_op_registration 2024-04-03T16:22:35.6509801Z inflating: build/bin/example_allreduce 2024-04-03T16:22:35.6574605Z inflating: build/bin/Dimname_test 2024-04-03T16:22:35.6666946Z inflating: build/bin/Dict_test 2024-04-03T16:22:35.6747379Z inflating: build/bin/MaybeOwned_test 2024-04-03T16:22:35.6821209Z inflating: build/bin/apply_utils_test 2024-04-03T16:22:35.6892978Z inflating: build/bin/NamedTensor_test 2024-04-03T16:22:35.6966348Z inflating: build/bin/atest 2024-04-03T16:22:35.7045105Z inflating: build/bin/basic 2024-04-03T16:22:35.7113729Z inflating: build/bin/broadcast_test 2024-04-03T16:22:35.7176936Z inflating: build/bin/cpu_allocator_test 2024-04-03T16:22:35.7249561Z inflating: build/bin/cpu_generator_test 2024-04-03T16:22:35.7316080Z inflating: build/bin/cpu_profiling_allocator_test 2024-04-03T16:22:35.7432124Z inflating: build/bin/cpu_rng_test 2024-04-03T16:22:35.7494390Z inflating: build/bin/dispatch_key_set_test 2024-04-03T16:22:35.7557330Z inflating: build/bin/dlconvertor_test 2024-04-03T16:22:35.7629529Z inflating: build/bin/extension_backend_test 2024-04-03T16:22:35.7698671Z inflating: build/bin/half_test 2024-04-03T16:22:35.7818076Z inflating: build/bin/ivalue_test 2024-04-03T16:22:35.7880298Z inflating: build/bin/lazy_tensor_test 2024-04-03T16:22:35.7948410Z inflating: build/bin/math_kernel_test 2024-04-03T16:22:35.8015775Z inflating: build/bin/memory_format_test 2024-04-03T16:22:35.8082058Z inflating: build/bin/memory_overlapping_test 2024-04-03T16:22:35.8148340Z inflating: build/bin/mobile_memory_cleanup 2024-04-03T16:22:35.8211285Z inflating: build/bin/operator_name_test 2024-04-03T16:22:35.8281110Z inflating: build/bin/native_test 2024-04-03T16:22:35.8345480Z inflating: build/bin/operators_test 2024-04-03T16:22:35.8410916Z inflating: build/bin/packedtensoraccessor_test 2024-04-03T16:22:35.8494905Z inflating: build/bin/pow_test 2024-04-03T16:22:35.8566524Z inflating: build/bin/quantized_test 2024-04-03T16:22:35.8628820Z inflating: build/bin/reduce_ops_test 2024-04-03T16:22:35.8692624Z inflating: build/bin/reportMemoryUsage_test 2024-04-03T16:22:35.8763239Z inflating: build/bin/scalar_tensor_test 2024-04-03T16:22:35.8836507Z inflating: build/bin/scalar_test 2024-04-03T16:22:35.8901015Z inflating: build/bin/StorageUtils_test 2024-04-03T16:22:35.8966707Z inflating: build/bin/stride_properties_test 2024-04-03T16:22:35.9065219Z inflating: build/bin/tensor_iterator_test 2024-04-03T16:22:35.9134897Z inflating: build/bin/type_ptr_test 2024-04-03T16:22:35.9138509Z inflating: build/bin/thread_init_test 2024-04-03T16:22:35.9206260Z inflating: build/bin/test_parallel 2024-04-03T16:22:35.9271456Z inflating: build/bin/undefined_tensor_test 2024-04-03T16:22:35.9346028Z inflating: build/bin/type_test 2024-04-03T16:22:35.9348181Z inflating: build/bin/verify_api_visibility 2024-04-03T16:22:35.9435059Z inflating: build/bin/legacy_vmap_test 2024-04-03T16:22:35.9499454Z inflating: build/bin/weakref_test 2024-04-03T16:22:35.9563840Z inflating: build/bin/wrapdim_test 2024-04-03T16:22:35.9638992Z inflating: build/bin/IListRef_test 2024-04-03T16:22:35.9703502Z inflating: build/bin/xla_tensor_test 2024-04-03T16:22:35.9836136Z inflating: build/bin/List_test 2024-04-03T16:22:35.9988763Z inflating: build/bin/kernel_function_legacy_test 2024-04-03T16:22:36.0071922Z inflating: build/bin/KernelFunction_test 2024-04-03T16:22:36.0192679Z inflating: build/bin/kernel_function_test 2024-04-03T16:22:36.0353781Z inflating: build/bin/kernel_lambda_legacy_test 2024-04-03T16:22:36.0482746Z inflating: build/bin/kernel_lambda_test 2024-04-03T16:22:36.0559027Z inflating: build/bin/kernel_stackbased_test 2024-04-03T16:22:36.0679431Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-04-03T16:22:36.0743087Z inflating: build/bin/CppSignature_test 2024-04-03T16:22:36.0804281Z inflating: build/bin/op_allowlist_test 2024-04-03T16:22:36.1180778Z inflating: build/bin/op_registration_test 2024-04-03T16:22:36.1250757Z inflating: build/bin/backend_fallback_test 2024-04-03T16:22:36.1330515Z inflating: build/bin/inline_container_test 2024-04-03T16:22:36.1396749Z inflating: build/bin/cuda_apply_test 2024-04-03T16:22:36.1459960Z inflating: build/bin/cuda_allocator_test 2024-04-03T16:22:36.1533971Z inflating: build/bin/cuda_atomic_ops_test 2024-04-03T16:22:36.1602213Z inflating: build/bin/cuda_caching_host_allocator_test 2024-04-03T16:22:36.1689964Z inflating: build/bin/cuda_complex_math_test 2024-04-03T16:22:36.1752192Z inflating: build/bin/cuda_device_test 2024-04-03T16:22:36.1825838Z inflating: build/bin/cuda_complex_test 2024-04-03T16:22:36.1898154Z inflating: build/bin/cuda_cub_test 2024-04-03T16:22:36.1961809Z inflating: build/bin/cuda_dlconvertor_test 2024-04-03T16:22:36.2025581Z inflating: build/bin/cuda_integer_divider_test 2024-04-03T16:22:36.2107437Z inflating: build/bin/cuda_distributions_test 2024-04-03T16:22:36.2169460Z inflating: build/bin/cuda_half_test 2024-04-03T16:22:36.2240279Z inflating: build/bin/cuda_generator_test 2024-04-03T16:22:36.2301762Z inflating: build/bin/cuda_optional_test 2024-04-03T16:22:36.2364163Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-04-03T16:22:36.2429916Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-04-03T16:22:36.2505143Z inflating: build/bin/cuda_stream_test 2024-04-03T16:22:36.2569545Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-04-03T16:22:36.2631381Z inflating: build/bin/cuda_cudnn_test 2024-04-03T16:22:36.2696311Z inflating: build/bin/cuda_vectorized_test 2024-04-03T16:22:36.2715261Z inflating: build/bin/tutorial_tensorexpr 2024-04-03T16:22:36.2797228Z inflating: build/bin/ProcessGroupGlooTest 2024-04-03T16:22:36.2869026Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-04-03T16:22:36.2947229Z inflating: build/bin/ProcessGroupNCCLTest 2024-04-03T16:22:36.3013021Z inflating: build/bin/ProcessGroupUCCTest 2024-04-03T16:22:36.3090058Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-04-03T16:22:36.3158567Z inflating: build/bin/test_dist_autograd 2024-04-03T16:22:36.3161757Z inflating: build/bin/parallel_benchmark 2024-04-03T16:22:36.3248342Z inflating: build/bin/test_cpp_rpc 2024-04-03T16:22:36.3332109Z inflating: build/bin/test_mobile_nnc 2024-04-03T16:22:36.3343676Z inflating: build/bin/aot_model_compiler_test 2024-04-03T16:22:36.4381692Z inflating: build/bin/test_tensorexpr 2024-04-03T16:22:36.4813776Z inflating: build/bin/test_lazy 2024-04-03T16:22:36.5519589Z inflating: build/bin/test_jit 2024-04-03T16:22:36.6995339Z inflating: build/bin/test_api 2024-04-03T16:22:36.6995892Z creating: .additional_ci_files/ 2024-04-03T16:22:36.7040302Z inflating: .additional_ci_files/test-times.json 2024-04-03T16:22:36.7220075Z inflating: .additional_ci_files/test-class-times.json 2024-04-03T16:22:36.7335469Z ##[group]Run rm artifacts.zip 2024-04-03T16:22:36.7335970Z rm artifacts.zip 2024-04-03T16:22:36.7344630Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:22:36.7345259Z env: 2024-04-03T16:22:36.7345850Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:36.7346407Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:36.7347012Z ##[endgroup] 2024-04-03T16:22:37.0608725Z ##[group]Run df -H 2024-04-03T16:22:37.0609138Z df -H 2024-04-03T16:22:37.0617671Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:22:37.0618314Z env: 2024-04-03T16:22:37.0618660Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:37.0619213Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:37.0619823Z ##[endgroup] 2024-04-03T16:22:37.0655439Z Filesystem Size Used Avail Use% Mounted on 2024-04-03T16:22:37.0656436Z devtmpfs 129G 0 129G 0% /dev 2024-04-03T16:22:37.0657252Z tmpfs 129G 0 129G 0% /dev/shm 2024-04-03T16:22:37.0657894Z tmpfs 129G 590k 129G 1% /run 2024-04-03T16:22:37.0658673Z tmpfs 129G 0 129G 0% /sys/fs/cgroup 2024-04-03T16:22:37.0659317Z /dev/xvda1 162G 37G 125G 23% / 2024-04-03T16:22:37.0660055Z tmpfs 26G 0 26G 0% /run/user/0 2024-04-03T16:22:37.0709856Z Prepare all required actions 2024-04-03T16:22:37.0710438Z Getting action download info 2024-04-03T16:22:37.1914564Z ##[group]Run ./.github/actions/download-td-artifacts 2024-04-03T16:22:37.1915144Z with: 2024-04-03T16:22:37.1915467Z env: 2024-04-03T16:22:37.1915811Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:37.1916362Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:37.1916979Z ##[endgroup] 2024-04-03T16:22:37.1963161Z ##[group]Run seemethere/download-artifact-s3@v4 2024-04-03T16:22:37.1963733Z with: 2024-04-03T16:22:37.1964048Z name: td_results 2024-04-03T16:22:37.1964446Z s3-bucket: gha-artifacts 2024-04-03T16:22:37.1964889Z region: us-east-1 2024-04-03T16:22:37.1965243Z env: 2024-04-03T16:22:37.1965586Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:37.1966157Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:37.1966750Z ##[endgroup] 2024-04-03T16:22:37.8114966Z (node:55332) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-04-03T16:22:37.8115798Z 2024-04-03T16:22:37.8116203Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-04-03T16:22:37.8117077Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-04-03T16:22:37.8118177Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-04-03T16:22:37.9070754Z Found 1 objects with prefix pytorch/pytorch/8536950919/td_results/ 2024-04-03T16:22:37.9071830Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-04-03T16:22:38.0048307Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-04-03T16:22:38.0058577Z Artifact download has finished successfully 2024-04-03T16:22:38.0253467Z ##[group]Run mkdir -p .additional_ci_files 2024-04-03T16:22:38.0254073Z mkdir -p .additional_ci_files 2024-04-03T16:22:38.0254817Z mv td_results.json .additional_ci_files/td_results.json 2024-04-03T16:22:38.0263605Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:22:38.0264232Z env: 2024-04-03T16:22:38.0264569Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:38.0265107Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:38.0265969Z ##[endgroup] 2024-04-03T16:22:38.0352647Z ##[group]Run .github/scripts/parse_ref.py 2024-04-03T16:22:38.0353231Z .github/scripts/parse_ref.py 2024-04-03T16:22:38.0361228Z shell: /usr/bin/bash -e {0} 2024-04-03T16:22:38.0361761Z env: 2024-04-03T16:22:38.0362103Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:38.0362670Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:38.0363261Z ##[endgroup] 2024-04-03T16:22:38.0674093Z Prepare all required actions 2024-04-03T16:22:38.0725161Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-04-03T16:22:38.0725731Z with: 2024-04-03T16:22:38.0726690Z github-token: *** 2024-04-03T16:22:38.0727100Z env: 2024-04-03T16:22:38.0727429Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:38.0727990Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:38.0728573Z ##[endgroup] 2024-04-03T16:22:38.0750909Z ##[group]Run set -eux 2024-04-03T16:22:38.0751321Z set -eux 2024-04-03T16:22:38.0752055Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-04-03T16:22:38.0760175Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:22:38.0760782Z env: 2024-04-03T16:22:38.0761117Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:38.0761675Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:38.0762493Z GITHUB_TOKEN: *** 2024-04-03T16:22:38.0762859Z ##[endgroup] 2024-04-03T16:22:38.0788070Z + python3 .github/scripts/get_workflow_job_id.py 8536950919 i-01dbe7bd2ee9fbb9d 2024-04-03T16:22:47.5141073Z setting job-id=23402706518 2024-04-03T16:22:47.5142209Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:22:47.5419274Z Prepare all required actions 2024-04-03T16:22:47.5419973Z Getting action download info 2024-04-03T16:22:47.6887012Z ##[group]Run ./.github/actions/filter-test-configs 2024-04-03T16:22:47.6887602Z with: 2024-04-03T16:22:47.6888241Z github-token: *** 2024-04-03T16:22:47.6890307Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-04-03T16:22:47.6892702Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:22:47.6893647Z env: 2024-04-03T16:22:47.6893987Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:47.6894577Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:47.6895208Z ##[endgroup] 2024-04-03T16:22:47.6964064Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-04-03T16:22:47.6964787Z with: 2024-04-03T16:22:47.6965136Z shell: bash 2024-04-03T16:22:47.6966948Z timeout_minutes: 10 2024-04-03T16:22:47.6967360Z max_attempts: 5 2024-04-03T16:22:47.6967767Z retry_wait_seconds: 30 2024-04-03T16:22:47.6968772Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-04-03T16:22:47.6969850Z polling_interval_seconds: 1 2024-04-03T16:22:47.6970314Z warning_on_retry: true 2024-04-03T16:22:47.6970764Z continue_on_error: false 2024-04-03T16:22:47.6971178Z env: 2024-04-03T16:22:47.6971532Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:47.6972120Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:47.6973005Z GITHUB_TOKEN: *** 2024-04-03T16:22:47.6973398Z ##[endgroup] 2024-04-03T16:22:47.7748009Z + python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-04-03T16:22:48.0685704Z Defaulting to user installation because normal site-packages is not writeable 2024-04-03T16:22:48.2155749Z Collecting requests==2.26.0 2024-04-03T16:22:48.2365288Z Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB) 2024-04-03T16:22:48.4155129Z Collecting pyyaml==6.0.1 2024-04-03T16:22:48.4207232Z Downloading PyYAML-6.0.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (670 kB) 2024-04-03T16:22:48.5189506Z Collecting certifi>=2017.4.17 2024-04-03T16:22:48.5243961Z Downloading certifi-2024.2.2-py3-none-any.whl (163 kB) 2024-04-03T16:22:48.6603471Z Collecting urllib3<1.27,>=1.21.1 2024-04-03T16:22:48.6656323Z Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 2024-04-03T16:22:48.7410945Z Collecting idna<4,>=2.5; python_version >= "3" 2024-04-03T16:22:48.7465179Z Downloading idna-3.6-py3-none-any.whl (61 kB) 2024-04-03T16:22:49.1055600Z Collecting charset-normalizer~=2.0.0; python_version >= "3" 2024-04-03T16:22:49.1110126Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-04-03T16:22:49.2035072Z Installing collected packages: certifi, urllib3, idna, charset-normalizer, requests, pyyaml 2024-04-03T16:22:49.4109930Z WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2024-04-03T16:22:49.4111381Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-04-03T16:22:49.5455271Z Successfully installed certifi-2024.2.2 charset-normalizer-2.0.12 idna-3.6 pyyaml-6.0.1 requests-2.26.0 urllib3-1.26.18 2024-04-03T16:22:49.7745244Z Command completed after 1 attempt(s). 2024-04-03T16:22:49.7807424Z ##[group]Run set -x 2024-04-03T16:22:49.7807835Z set -x 2024-04-03T16:22:49.7808228Z  2024-04-03T16:22:49.7808940Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-04-03T16:22:49.7809800Z # in runner workspace 2024-04-03T16:22:49.7810492Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-04-03T16:22:49.7820232Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:22:49.7820931Z env: 2024-04-03T16:22:49.7821268Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:49.7821857Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:49.7822486Z ##[endgroup] 2024-04-03T16:22:49.7853464Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-04-03T16:22:49.8167056Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-04-03T16:22:49.8167741Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-04-03T16:22:49.8168313Z echo "Job name: ${JOB_NAME}" 2024-04-03T16:22:49.8168822Z  2024-04-03T16:22:49.8169517Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-04-03T16:22:49.8170361Z # in runner workspace 2024-04-03T16:22:49.8171105Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-04-03T16:22:49.8171953Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-04-03T16:22:49.8172538Z  --job-name "${JOB_NAME}" \ 2024-04-03T16:22:49.8174533Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" \ 2024-04-03T16:22:49.8176768Z  --pr-number "${PR_NUMBER}" \ 2024-04-03T16:22:49.8177302Z  --tag "${TAG}" \ 2024-04-03T16:22:49.8177798Z  --event-name "${EVENT_NAME}" \ 2024-04-03T16:22:49.8178337Z  --schedule "${SCHEDULE}" \ 2024-04-03T16:22:49.8178871Z  --branch "${HEAD_BRANCH}" 2024-04-03T16:22:49.8187366Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:22:49.8187986Z env: 2024-04-03T16:22:49.8188334Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:49.8188930Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:49.8189807Z GITHUB_TOKEN: *** 2024-04-03T16:22:49.8190617Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:22:49.8191544Z PR_NUMBER: 123154 2024-04-03T16:22:49.8191936Z TAG: 2024-04-03T16:22:49.8192266Z EVENT_NAME: pull_request 2024-04-03T16:22:49.8192703Z SCHEDULE: 2024-04-03T16:22:49.8193065Z HEAD_BRANCH: 2024-04-03T16:22:49.8193416Z ##[endgroup] 2024-04-03T16:22:49.8219041Z Workflow: pull 2024-04-03T16:22:49.8219972Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:22:50.0751614Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-04-03T16:22:50.1928905Z fatal: unknown commit main 2024-04-03T16:22:50.1955572Z /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/filter_test_configs.py:458: UserWarning: failed to get commit messages: Command '['git', 'cherry', '-v', 'main']' returned non-zero exit status 128. 2024-04-03T16:22:50.1957624Z warnings.warn(f"failed to get commit messages: {e}") 2024-04-03T16:22:50.2095278Z ##[group]Run echo "Filtered matrix:" 2024-04-03T16:22:50.2095836Z echo "Filtered matrix:" 2024-04-03T16:22:50.2097759Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" 2024-04-03T16:22:50.2099646Z  2024-04-03T16:22:50.2099986Z echo 2024-04-03T16:22:50.2100454Z echo "Is the current job unstable? False" 2024-04-03T16:22:50.2101021Z  2024-04-03T16:22:50.2101336Z echo 2024-04-03T16:22:50.2101776Z echo "Is keep-going label set? False" 2024-04-03T16:22:50.2102548Z  2024-04-03T16:22:50.2103016Z echo 2024-04-03T16:22:50.2103430Z echo "Renabled issues? " 2024-04-03T16:22:50.2111752Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:22:50.2112383Z env: 2024-04-03T16:22:50.2112723Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:50.2113272Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:50.2113878Z ##[endgroup] 2024-04-03T16:22:50.2138301Z Filtered matrix: 2024-04-03T16:22:50.2140369Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}]} 2024-04-03T16:22:50.2141978Z 2024-04-03T16:22:50.2142174Z Is the current job unstable? False 2024-04-03T16:22:50.2142584Z 2024-04-03T16:22:50.2142852Z Is keep-going label set? False 2024-04-03T16:22:50.2143176Z 2024-04-03T16:22:50.2143363Z Renabled issues? 2024-04-03T16:22:50.2197893Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-04-03T16:22:50.2198823Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-04-03T16:22:50.2206709Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:22:50.2207339Z env: 2024-04-03T16:22:50.2207659Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:50.2208223Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:50.2208832Z JOB_TIMEOUT: 360 2024-04-03T16:22:50.2209189Z ##[endgroup] 2024-04-03T16:22:50.2315685Z ##[group]Run set -x 2024-04-03T16:22:50.2316227Z set -x 2024-04-03T16:22:50.2316593Z  2024-04-03T16:22:50.2317033Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-04-03T16:22:50.2317712Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-04-03T16:22:50.2318423Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-04-03T16:22:50.2319067Z  TEST_COMMAND=.ci/onnx/test.sh 2024-04-03T16:22:50.2319582Z else 2024-04-03T16:22:50.2320008Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-04-03T16:22:50.2320534Z fi 2024-04-03T16:22:50.2320853Z  2024-04-03T16:22:50.2321440Z # detached container should get cleaned up by teardown_ec2_linux 2024-04-03T16:22:50.2322409Z # TODO: Stop building test binaries as part of the build phase 2024-04-03T16:22:50.2323248Z # Used for GPU_FLAG since that doesn't play nice 2024-04-03T16:22:50.2323996Z # shellcheck disable=SC2086,SC2090 2024-04-03T16:22:50.2324565Z container_name=$(docker run \ 2024-04-03T16:22:50.2325091Z  ${GPU_FLAG:-} \ 2024-04-03T16:22:50.2325549Z  -e BUILD_ENVIRONMENT \ 2024-04-03T16:22:50.2326023Z  -e PR_NUMBER \ 2024-04-03T16:22:50.2326471Z  -e GITHUB_ACTIONS \ 2024-04-03T16:22:50.2326955Z  -e GITHUB_REPOSITORY \ 2024-04-03T16:22:50.2327442Z  -e GITHUB_WORKFLOW \ 2024-04-03T16:22:50.2327930Z  -e GITHUB_JOB \ 2024-04-03T16:22:50.2328374Z  -e GITHUB_RUN_ID \ 2024-04-03T16:22:50.2328852Z  -e GITHUB_RUN_NUMBER \ 2024-04-03T16:22:50.2329335Z  -e GITHUB_RUN_ATTEMPT \ 2024-04-03T16:22:50.2329824Z  -e JOB_ID \ 2024-04-03T16:22:50.2330243Z  -e JOB_NAME \ 2024-04-03T16:22:50.2330655Z  -e BASE_SHA \ 2024-04-03T16:22:50.2331081Z  -e BRANCH \ 2024-04-03T16:22:50.2331494Z  -e SHA1 \ 2024-04-03T16:22:50.2331902Z  -e AWS_DEFAULT_REGION \ 2024-04-03T16:22:50.2332405Z  -e IN_WHEEL_TEST \ 2024-04-03T16:22:50.2332872Z  -e SHARD_NUMBER \ 2024-04-03T16:22:50.2333310Z  -e TEST_CONFIG \ 2024-04-03T16:22:50.2333766Z  -e NUM_TEST_SHARDS \ 2024-04-03T16:22:50.2334253Z  -e REENABLED_ISSUES \ 2024-04-03T16:22:50.2334744Z  -e CONTINUE_THROUGH_ERROR \ 2024-04-03T16:22:50.2335267Z  -e VERBOSE_TEST_LOGS \ 2024-04-03T16:22:50.2335970Z  -e NO_TEST_TIMEOUT \ 2024-04-03T16:22:50.2336418Z  -e NO_TD \ 2024-04-03T16:22:50.2336833Z  -e PR_LABELS \ 2024-04-03T16:22:50.2337336Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-04-03T16:22:50.2337890Z  -e SCCACHE_BUCKET \ 2024-04-03T16:22:50.2338385Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-04-03T16:22:50.2338893Z  -e XLA_CUDA \ 2024-04-03T16:22:50.2339369Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-04-03T16:22:50.2340013Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-04-03T16:22:50.2340667Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-04-03T16:22:50.2341288Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-04-03T16:22:50.2341871Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-04-03T16:22:50.2342447Z  -e DASHBOARD_TAG \ 2024-04-03T16:22:50.2343031Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-04-03T16:22:50.2343716Z  --ulimit stack=10485760:83886080 \ 2024-04-03T16:22:50.2344321Z  --security-opt seccomp=unconfined \ 2024-04-03T16:22:50.2344902Z  --cap-add=SYS_PTRACE \ 2024-04-03T16:22:50.2345370Z  --ipc=host \ 2024-04-03T16:22:50.2346220Z  --shm-size="${SHM_SIZE}" \ 2024-04-03T16:22:50.2346717Z  --tty \ 2024-04-03T16:22:50.2347084Z  --detach \ 2024-04-03T16:22:50.2347522Z  --name="${container_name}" \ 2024-04-03T16:22:50.2348044Z  --user jenkins \ 2024-04-03T16:22:50.2348773Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-04-03T16:22:50.2349504Z  -w /var/lib/jenkins/workspace \ 2024-04-03T16:22:50.2350044Z  "${DOCKER_IMAGE}" 2024-04-03T16:22:50.2350457Z ) 2024-04-03T16:22:50.2350959Z # Propagate download.pytorch.org IP to container 2024-04-03T16:22:50.2352134Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-04-03T16:22:50.2353358Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-04-03T16:22:50.2354524Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-04-03T16:22:50.2362640Z shell: /usr/bin/bash -e {0} 2024-04-03T16:22:50.2363083Z env: 2024-04-03T16:22:50.2363399Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:50.2363967Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:50.2364700Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-04-03T16:22:50.2365291Z PR_NUMBER: 123154 2024-04-03T16:22:50.2365714Z GITHUB_REPOSITORY: pytorch/pytorch 2024-04-03T16:22:50.2366224Z GITHUB_WORKFLOW: pull 2024-04-03T16:22:50.2366619Z GITHUB_JOB: test 2024-04-03T16:22:50.2367002Z GITHUB_RUN_ID: 8536950919 2024-04-03T16:22:50.2367444Z GITHUB_RUN_NUMBER: 192590 2024-04-03T16:22:50.2367866Z GITHUB_RUN_ATTEMPT: 2 2024-04-03T16:22:50.2368274Z JOB_ID: 23402706518 2024-04-03T16:22:50.2369102Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:22:50.2370210Z BRANCH: pull/123154 2024-04-03T16:22:50.2370670Z SHA1: d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:22:50.2371322Z BASE_SHA: 1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:22:50.2371905Z TEST_CONFIG: distributed 2024-04-03T16:22:50.2372317Z SHARD_NUMBER: 3 2024-04-03T16:22:50.2372695Z NUM_TEST_SHARDS: 3 2024-04-03T16:22:50.2373091Z REENABLED_ISSUES: 2024-04-03T16:22:50.2373495Z CONTINUE_THROUGH_ERROR: False 2024-04-03T16:22:50.2373973Z VERBOSE_TEST_LOGS: False 2024-04-03T16:22:50.2374405Z NO_TEST_TIMEOUT: False 2024-04-03T16:22:50.2374799Z NO_TD: False 2024-04-03T16:22:50.2375263Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-04-03T16:22:50.2375874Z SCCACHE_S3_KEY_PREFIX: pull 2024-04-03T16:22:50.2376300Z SHM_SIZE: 2g 2024-04-03T16:22:50.2377545Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:22:50.2379002Z XLA_CUDA: 2024-04-03T16:22:50.2379577Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-04-03T16:22:50.2380340Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-04-03T16:22:50.2380875Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-04-03T16:22:50.2381364Z DASHBOARD_TAG: 2024-04-03T16:22:50.2381748Z HUGGING_FACE_HUB_TOKEN: 2024-04-03T16:22:50.2382171Z ##[endgroup] 2024-04-03T16:22:50.2407342Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-04-03T16:22:50.2408087Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-04-03T16:22:50.2408705Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-04-03T16:22:50.2417502Z +++ nproc --ignore=2 2024-04-03T16:22:50.2450819Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_8536950919 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:23:01.2778571Z + container_name=e451a0adc73818f3a047cb83ff91d039adc3cd55a8f5a0cb827c973fea5bd896 2024-04-03T16:23:01.2780793Z + grep download.pytorch.org /etc/hosts 2024-04-03T16:23:01.2784721Z + docker exec -i e451a0adc73818f3a047cb83ff91d039adc3cd55a8f5a0cb827c973fea5bd896 sudo bash -c '/bin/cat >> /etc/hosts' 2024-04-03T16:23:01.3660949Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-04-03T16:23:01.3713533Z + echo DOCKER_CONTAINER_ID=e451a0adc73818f3a047cb83ff91d039adc3cd55a8f5a0cb827c973fea5bd896 2024-04-03T16:23:01.3718251Z ++ echo dist/torch-2.4.0a0+gitd253bd7-cp310-cp310-linux_x86_64.whl 2024-04-03T16:23:01.3721138Z + docker exec -t e451a0adc73818f3a047cb83ff91d039adc3cd55a8f5a0cb827c973fea5bd896 sh -c 'pip install dist/torch-2.4.0a0+gitd253bd7-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-04-03T16:23:01.9250785Z Processing ./dist/torch-2.4.0a0+gitd253bd7-cp310-cp310-linux_x86_64.whl 2024-04-03T16:23:03.0361674Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (3.13.1) 2024-04-03T16:23:03.0371239Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (4.10.0) 2024-04-03T16:23:03.0375547Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (1.12) 2024-04-03T16:23:03.0380711Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (2.8.8) 2024-04-03T16:23:03.0385344Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (3.1.3) 2024-04-03T16:23:03.0390739Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (2024.2.0) 2024-04-03T16:23:03.0405567Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (3.3.0) 2024-04-03T16:23:03.0491258Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.4.0a0+gitd253bd7) (1.21.2) 2024-04-03T16:23:03.1120090Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.4.0a0+gitd253bd7) (2.1.5) 2024-04-03T16:23:03.1354171Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.4.0a0+gitd253bd7) (1.3.0) 2024-04-03T16:23:04.1800149Z Installing collected packages: torch 2024-04-03T16:23:16.4914411Z Successfully installed torch-2.4.0a0+gitd253bd7 2024-04-03T16:23:16.5594512Z + echo 'Environment variables:' 2024-04-03T16:23:16.5594946Z Environment variables: 2024-04-03T16:23:16.5595352Z + env 2024-04-03T16:23:16.5602391Z INSTALLED_DB=yes 2024-04-03T16:23:16.5602857Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-04-03T16:23:16.5603305Z NVIDIA_VISIBLE_DEVICES=all 2024-04-03T16:23:16.5603831Z NV_NVML_DEV_VERSION=11.8.86-1 2024-04-03T16:23:16.5604522Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:23:16.5606889Z CONTINUE_THROUGH_ERROR=False 2024-04-03T16:23:16.5607620Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-04-03T16:23:16.5608255Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-04-03T16:23:16.5609078Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-04-03T16:23:16.5610011Z HOSTNAME=e451a0adc738 2024-04-03T16:23:16.5611417Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_650b9172-447a-433b-9fb8-f30cfe917174 2024-04-03T16:23:16.5612469Z GITHUB_ACTION=__self 2024-04-03T16:23:16.5612903Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-04-03T16:23:16.5615638Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-04-03T16:23:16.5618655Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-04-03T16:23:16.5619308Z NV_NVTX_VERSION=11.8.86-1 2024-04-03T16:23:16.5619706Z GITHUB_RUN_NUMBER=192590 2024-04-03T16:23:16.5620178Z TEST_CONFIG=distributed 2024-04-03T16:23:16.5620574Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-04-03T16:23:16.5621186Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-04-03T16:23:16.5621769Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-04-03T16:23:16.5622264Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-04-03T16:23:16.5622807Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-04-03T16:23:16.5623287Z GITHUB_TRIGGERING_ACTOR=soulitzer 2024-04-03T16:23:16.5623840Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-04-03T16:23:16.5624432Z GITHUB_REF_TYPE=branch 2024-04-03T16:23:16.5624903Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-04-03T16:23:16.5625356Z NCCL_VERSION=2.16.2-1 2024-04-03T16:23:16.5626306Z BASE_SHA=1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:23:16.5626821Z XLA_CUDA= 2024-04-03T16:23:16.5627149Z HUGGING_FACE_HUB_TOKEN= 2024-04-03T16:23:16.5629689Z *** 2024-04-03T16:23:16.5630070Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-04-03T16:23:16.5630605Z GITHUB_REPOSITORY_ID=65600975 2024-04-03T16:23:16.5631040Z GITHUB_ACTIONS=true 2024-04-03T16:23:16.5631411Z NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:16.5632086Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-04-03T16:23:16.5632754Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-04-03T16:23:16.5633350Z SHA1=d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:23:16.5633956Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-04-03T16:23:16.5634558Z GITHUB_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:23:16.5635472Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/123154/merge 2024-04-03T16:23:16.5636288Z UCC_HOME=/usr 2024-04-03T16:23:16.5636928Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-04-03T16:23:16.5637436Z VERBOSE_TEST_LOGS=False 2024-04-03T16:23:16.5637863Z NVIDIA_PRODUCT_NAME=CUDA 2024-04-03T16:23:16.5638416Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-04-03T16:23:16.5639013Z GITHUB_REF=refs/pull/123154/merge 2024-04-03T16:23:16.5639539Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-04-03T16:23:16.5639987Z SHARD_NUMBER=3 2024-04-03T16:23:16.5640380Z GITHUB_REF_PROTECTED=false 2024-04-03T16:23:16.5640810Z HOME=/var/lib/jenkins 2024-04-03T16:23:16.5641252Z GITHUB_API_URL=https://api.github.com 2024-04-03T16:23:16.5641806Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-04-03T16:23:16.5642400Z UCX_COMMIT=00bcc6bb18fc282eb160623b4c0d300147f579af 2024-04-03T16:23:16.5642986Z SCCACHE_S3_KEY_PREFIX=pull 2024-04-03T16:23:16.5643421Z CUDA_VERSION=11.8.0 2024-04-03T16:23:16.5644010Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-04-03T16:23:16.5644537Z NUM_TEST_SHARDS=3 2024-04-03T16:23:16.5644908Z UCX_HOME=/usr 2024-04-03T16:23:16.5645547Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-04-03T16:23:16.5646912Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_650b9172-447a-433b-9fb8-f30cfe917174 2024-04-03T16:23:16.5648468Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:23:16.5650027Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_650b9172-447a-433b-9fb8-f30cfe917174 2024-04-03T16:23:16.5651588Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-04-03T16:23:16.5652546Z GITHUB_EVENT_NAME=pull_request 2024-04-03T16:23:16.5653001Z DASHBOARD_TAG= 2024-04-03T16:23:16.5653372Z GITHUB_RUN_ID=8536950919 2024-04-03T16:23:16.5653929Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-04-03T16:23:16.5654608Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-04-03T16:23:16.5655898Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_650b9172-447a-433b-9fb8-f30cfe917174 2024-04-03T16:23:16.5656996Z GITHUB_ACTOR=caic99 2024-04-03T16:23:16.5657439Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-04-03T16:23:16.5657914Z PR_NUMBER=123154 2024-04-03T16:23:16.5658275Z GITHUB_RUN_ATTEMPT=2 2024-04-03T16:23:16.5658690Z ANACONDA_PYTHON_VERSION=3.10 2024-04-03T16:23:16.5659246Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-04-03T16:23:16.5659810Z TERM=xterm 2024-04-03T16:23:16.5660215Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-04-03T16:23:16.5660730Z INSTALLED_VISION=yes 2024-04-03T16:23:16.5661119Z BRANCH=pull/123154 2024-04-03T16:23:16.5661523Z OPENSSL_ROOT_DIR=/opt/openssl 2024-04-03T16:23:16.5662024Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-04-03T16:23:16.5662533Z CUDA_PATH=/usr/local/cuda 2024-04-03T16:23:16.5663487Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-04-03T16:23:16.5664447Z GITHUB_SERVER_URL=https://github.com 2024-04-03T16:23:16.5665042Z UCC_COMMIT=7cb07a76ccedad7e56ceb136b865eb9319c258ea 2024-04-03T16:23:16.5665892Z REENABLED_ISSUES= 2024-04-03T16:23:16.5666285Z SHLVL=1 2024-04-03T16:23:16.5666586Z MAX_JOBS=30 2024-04-03T16:23:16.5666979Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-04-03T16:23:16.5667418Z NVARCH=x86_64 2024-04-03T16:23:16.5667762Z GITHUB_ACTOR_ID=78061359 2024-04-03T16:23:16.5668344Z GITHUB_WORKFLOW_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:23:16.5669002Z GITHUB_REF_NAME=123154/merge 2024-04-03T16:23:16.5669521Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-04-03T16:23:16.5670362Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-04-03T16:23:16.5671097Z GITHUB_JOB=test 2024-04-03T16:23:16.5671559Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-04-03T16:23:16.5672275Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-04-03T16:23:16.5672910Z NO_TEST_TIMEOUT=False 2024-04-03T16:23:16.5673363Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-04-03T16:23:16.5673911Z GITHUB_REPOSITORY=pytorch/pytorch 2024-04-03T16:23:16.5674605Z NV_NVPROF_VERSION=11.8.87-1 2024-04-03T16:23:16.5675035Z GITHUB_RETENTION_DAYS=90 2024-04-03T16:23:16.5675467Z OPENSSL_DIR=/opt/openssl 2024-04-03T16:23:16.5675903Z GITHUB_ACTION_REPOSITORY= 2024-04-03T16:23:16.5677211Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:23:16.5678595Z GITHUB_BASE_REF=main 2024-04-03T16:23:16.5679021Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-04-03T16:23:16.5679485Z CI=true 2024-04-03T16:23:16.5679857Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-04-03T16:23:16.5680376Z GITHUB_REPOSITORY_OWNER=pytorch 2024-04-03T16:23:16.5680821Z JOB_ID=23402706518 2024-04-03T16:23:16.5681205Z INSTALLED_PROTOBUF=yes 2024-04-03T16:23:16.5681648Z GITHUB_HEAD_REF=patch-1 2024-04-03T16:23:16.5682043Z GITHUB_ACTION_REF= 2024-04-03T16:23:16.5682584Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-04-03T16:23:16.5683174Z GITHUB_WORKFLOW=pull 2024-04-03T16:23:16.5683585Z DEBIAN_FRONTEND=noninteractive 2024-04-03T16:23:16.5684744Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_650b9172-447a-433b-9fb8-f30cfe917174 2024-04-03T16:23:16.5685782Z NO_TD=False 2024-04-03T16:23:16.5686151Z SKIP_SCCACHE_INITIALIZATION=1 2024-04-03T16:23:16.5686585Z _=/usr/bin/env 2024-04-03T16:23:16.5687181Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-04-03T16:23:16.5824606Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-04-03T16:23:16.5826075Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-04-03T16:23:16.5828028Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-04-03T16:23:16.5829178Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-04-03T16:23:16.5829930Z + BUILD_DIR=build 2024-04-03T16:23:16.5830334Z + BUILD_RENAMED_DIR=build_renamed 2024-04-03T16:23:16.5830819Z + BUILD_BIN_DIR=build/bin 2024-04-03T16:23:16.5831386Z + SHARD_NUMBER=3 2024-04-03T16:23:16.5832013Z + NUM_TEST_SHARDS=3 2024-04-03T16:23:16.5832404Z + export VALGRIND=ON 2024-04-03T16:23:16.5832777Z + VALGRIND=ON 2024-04-03T16:23:16.5833338Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-04-03T16:23:16.5833931Z + [[ 0 == \1 ]] 2024-04-03T16:23:16.5834277Z + [[ False == \1 ]] 2024-04-03T16:23:16.5834819Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-04-03T16:23:16.5835445Z ++ realpath build/custom_test_artifacts 2024-04-03T16:23:16.5840909Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-04-03T16:23:16.5841795Z + [[ -n '' ]] 2024-04-03T16:23:16.5844447Z ++ dirname .ci/pytorch/test.sh 2024-04-03T16:23:16.5851835Z + source .ci/pytorch/common.sh 2024-04-03T16:23:16.5856007Z +++ dirname .ci/pytorch/common.sh 2024-04-03T16:23:16.5863670Z ++ source .ci/pytorch/common_utils.sh 2024-04-03T16:23:16.5866407Z +++ declare -f -t trap_add 2024-04-03T16:23:16.5873979Z ++ set -ex 2024-04-03T16:23:16.5880524Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-04-03T16:23:16.5881400Z ++ BUILD_TEST_LIBTORCH=0 2024-04-03T16:23:16.5882265Z + echo 'Environment variables' 2024-04-03T16:23:16.5883132Z Environment variables 2024-04-03T16:23:16.5883790Z + env 2024-04-03T16:23:16.5884328Z INSTALLED_DB=yes 2024-04-03T16:23:16.5884929Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-04-03T16:23:16.5885418Z NVIDIA_VISIBLE_DEVICES=all 2024-04-03T16:23:16.5886160Z NV_NVML_DEV_VERSION=11.8.86-1 2024-04-03T16:23:16.5886996Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:23:16.5887729Z CONTINUE_THROUGH_ERROR=False 2024-04-03T16:23:16.5888330Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-04-03T16:23:16.5889067Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-04-03T16:23:16.5890315Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-04-03T16:23:16.5891262Z HOSTNAME=e451a0adc738 2024-04-03T16:23:16.5892688Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_650b9172-447a-433b-9fb8-f30cfe917174 2024-04-03T16:23:16.5894355Z GITHUB_ACTION=__self 2024-04-03T16:23:16.5895117Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-04-03T16:23:16.5897834Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-04-03T16:23:16.5900737Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-04-03T16:23:16.5901379Z NV_NVTX_VERSION=11.8.86-1 2024-04-03T16:23:16.5901812Z GITHUB_RUN_NUMBER=192590 2024-04-03T16:23:16.5902227Z TEST_CONFIG=distributed 2024-04-03T16:23:16.5902665Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-04-03T16:23:16.5903255Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-04-03T16:23:16.5903821Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-04-03T16:23:16.5904370Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-04-03T16:23:16.5904892Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-04-03T16:23:16.5905352Z GITHUB_TRIGGERING_ACTOR=soulitzer 2024-04-03T16:23:16.5906378Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-04-03T16:23:16.5906971Z GITHUB_REF_TYPE=branch 2024-04-03T16:23:16.5907381Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-04-03T16:23:16.5907855Z NCCL_VERSION=2.16.2-1 2024-04-03T16:23:16.5908333Z BASE_SHA=1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:23:16.5909024Z XLA_CUDA= 2024-04-03T16:23:16.5909395Z HUGGING_FACE_HUB_TOKEN= 2024-04-03T16:23:16.5909889Z *** 2024-04-03T16:23:16.5910221Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-04-03T16:23:16.5910722Z GITHUB_REPOSITORY_ID=65600975 2024-04-03T16:23:16.5911174Z GITHUB_ACTIONS=true 2024-04-03T16:23:16.5911641Z NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:16.5912235Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-04-03T16:23:16.5912920Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-04-03T16:23:16.5913520Z SHA1=d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:23:16.5914138Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-04-03T16:23:16.5914760Z GITHUB_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:23:16.5915668Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/123154/merge 2024-04-03T16:23:16.5916459Z UCC_HOME=/usr 2024-04-03T16:23:16.5916881Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-04-03T16:23:16.5917391Z VERBOSE_TEST_LOGS=False 2024-04-03T16:23:16.5917806Z NVIDIA_PRODUCT_NAME=CUDA 2024-04-03T16:23:16.5918379Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-04-03T16:23:16.5918969Z GITHUB_REF=refs/pull/123154/merge 2024-04-03T16:23:16.5919471Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-04-03T16:23:16.5919942Z SHARD_NUMBER=3 2024-04-03T16:23:16.5920321Z GITHUB_REF_PROTECTED=false 2024-04-03T16:23:16.5920734Z HOME=/var/lib/jenkins 2024-04-03T16:23:16.5921185Z GITHUB_API_URL=https://api.github.com 2024-04-03T16:23:16.5921743Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-04-03T16:23:16.5922317Z UCX_COMMIT=00bcc6bb18fc282eb160623b4c0d300147f579af 2024-04-03T16:23:16.5922918Z SCCACHE_S3_KEY_PREFIX=pull 2024-04-03T16:23:16.5923349Z CUDA_VERSION=11.8.0 2024-04-03T16:23:16.5923861Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-04-03T16:23:16.5924422Z NUM_TEST_SHARDS=3 2024-04-03T16:23:16.5924793Z UCX_HOME=/usr 2024-04-03T16:23:16.5925411Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-04-03T16:23:16.5926788Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_650b9172-447a-433b-9fb8-f30cfe917174 2024-04-03T16:23:16.5928347Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:23:16.5929877Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_650b9172-447a-433b-9fb8-f30cfe917174 2024-04-03T16:23:16.5931305Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-04-03T16:23:16.5932304Z GITHUB_EVENT_NAME=pull_request 2024-04-03T16:23:16.5932756Z DASHBOARD_TAG= 2024-04-03T16:23:16.5933110Z GITHUB_RUN_ID=8536950919 2024-04-03T16:23:16.5933682Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-04-03T16:23:16.5934362Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-04-03T16:23:16.5935613Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_650b9172-447a-433b-9fb8-f30cfe917174 2024-04-03T16:23:16.5936717Z GITHUB_ACTOR=caic99 2024-04-03T16:23:16.5937166Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-04-03T16:23:16.5937631Z PR_NUMBER=123154 2024-04-03T16:23:16.5938009Z GITHUB_RUN_ATTEMPT=2 2024-04-03T16:23:16.5938400Z VALGRIND=ON 2024-04-03T16:23:16.5938750Z ANACONDA_PYTHON_VERSION=3.10 2024-04-03T16:23:16.5939306Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-04-03T16:23:16.5939884Z TERM=xterm 2024-04-03T16:23:16.5940281Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-04-03T16:23:16.5940803Z INSTALLED_VISION=yes 2024-04-03T16:23:16.5941204Z BRANCH=pull/123154 2024-04-03T16:23:16.5941589Z OPENSSL_ROOT_DIR=/opt/openssl 2024-04-03T16:23:16.5942086Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-04-03T16:23:16.5942616Z CUDA_PATH=/usr/local/cuda 2024-04-03T16:23:16.5943559Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-04-03T16:23:16.5944563Z GITHUB_SERVER_URL=https://github.com 2024-04-03T16:23:16.5945177Z UCC_COMMIT=7cb07a76ccedad7e56ceb136b865eb9319c258ea 2024-04-03T16:23:16.5945917Z REENABLED_ISSUES= 2024-04-03T16:23:16.5946427Z SHLVL=1 2024-04-03T16:23:16.5946765Z MAX_JOBS=30 2024-04-03T16:23:16.5947147Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-04-03T16:23:16.5947585Z NVARCH=x86_64 2024-04-03T16:23:16.5947951Z GITHUB_ACTOR_ID=78061359 2024-04-03T16:23:16.5948517Z GITHUB_WORKFLOW_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:23:16.5949174Z GITHUB_REF_NAME=123154/merge 2024-04-03T16:23:16.5949714Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-04-03T16:23:16.5950539Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-04-03T16:23:16.5951263Z GITHUB_JOB=test 2024-04-03T16:23:16.5951760Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-04-03T16:23:16.5952443Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-04-03T16:23:16.5953088Z NO_TEST_TIMEOUT=False 2024-04-03T16:23:16.5953554Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-04-03T16:23:16.5954089Z GITHUB_REPOSITORY=pytorch/pytorch 2024-04-03T16:23:16.5954611Z NV_NVPROF_VERSION=11.8.87-1 2024-04-03T16:23:16.5955071Z GITHUB_RETENTION_DAYS=90 2024-04-03T16:23:16.5955491Z OPENSSL_DIR=/opt/openssl 2024-04-03T16:23:16.5955928Z GITHUB_ACTION_REPOSITORY= 2024-04-03T16:23:16.5957259Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:23:16.5958622Z GITHUB_BASE_REF=main 2024-04-03T16:23:16.5959050Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-04-03T16:23:16.5959521Z CI=true 2024-04-03T16:23:16.5959895Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-04-03T16:23:16.5960414Z GITHUB_REPOSITORY_OWNER=pytorch 2024-04-03T16:23:16.5960877Z JOB_ID=23402706518 2024-04-03T16:23:16.5961246Z INSTALLED_PROTOBUF=yes 2024-04-03T16:23:16.5961686Z GITHUB_HEAD_REF=patch-1 2024-04-03T16:23:16.5962100Z GITHUB_ACTION_REF= 2024-04-03T16:23:16.5962619Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-04-03T16:23:16.5963205Z GITHUB_WORKFLOW=pull 2024-04-03T16:23:16.5963624Z DEBIAN_FRONTEND=noninteractive 2024-04-03T16:23:16.5964769Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_650b9172-447a-433b-9fb8-f30cfe917174 2024-04-03T16:23:16.5965809Z NO_TD=False 2024-04-03T16:23:16.5966183Z SKIP_SCCACHE_INITIALIZATION=1 2024-04-03T16:23:16.5966618Z _=/usr/bin/env 2024-04-03T16:23:16.5967036Z + echo 'Testing pytorch' 2024-04-03T16:23:16.5967461Z Testing pytorch 2024-04-03T16:23:16.5967842Z + export LANG=C.UTF-8 2024-04-03T16:23:16.5968401Z + LANG=C.UTF-8 2024-04-03T16:23:16.5968770Z + PR_NUMBER=123154 2024-04-03T16:23:16.5969172Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-04-03T16:23:16.5969733Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-04-03T16:23:16.5970436Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-04-03T16:23:16.5971025Z + [[ distributed == \s\l\o\w ]] 2024-04-03T16:23:16.5971720Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-04-03T16:23:16.5972522Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-04-03T16:23:16.5973170Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-04-03T16:23:16.5973767Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-04-03T16:23:16.5974300Z + [[ distributed == *crossref* ]] 2024-04-03T16:23:16.5974919Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-04-03T16:23:16.5975645Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-04-03T16:23:16.5976389Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-04-03T16:23:16.5977041Z + pip_install --user ninja==1.10.2 2024-04-03T16:23:16.5977717Z + pip install --progress-bar off --user ninja==1.10.2 2024-04-03T16:23:17.1169414Z Collecting ninja==1.10.2 2024-04-03T16:23:17.1386254Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-04-03T16:23:17.1575125Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-04-03T16:23:18.1849067Z Installing collected packages: ninja 2024-04-03T16:23:18.1947728Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-04-03T16:23:18.1949415Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-04-03T16:23:18.1999569Z Successfully installed ninja-1.10.2 2024-04-03T16:23:18.2693823Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:23:18.2696609Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:23:18.2698185Z + install_tlparse 2024-04-03T16:23:18.2698707Z + pip_install --user tlparse==0.3.7 2024-04-03T16:23:18.2699384Z + pip install --progress-bar off --user tlparse==0.3.7 2024-04-03T16:23:18.7682802Z Collecting tlparse==0.3.7 2024-04-03T16:23:18.7908016Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-04-03T16:23:18.8062101Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-04-03T16:23:19.8719370Z Installing collected packages: tlparse 2024-04-03T16:23:19.9228341Z Successfully installed tlparse-0.3.7 2024-04-03T16:23:19.9921946Z ++ python -m site --user-base 2024-04-03T16:23:20.0150206Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:23:20.0152724Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-04-03T16:23:20.0153509Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-04-03T16:23:20.0154226Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-04-03T16:23:20.0155359Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-04-03T16:23:20.0157169Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-04-03T16:23:20.0158074Z + cd test 2024-04-03T16:23:20.0158742Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-04-03T16:23:21.8609636Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-04-03T16:23:21.8610425Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-04-03T16:23:21.8613355Z + DYNAMO_BENCHMARK_FLAGS=() 2024-04-03T16:23:21.8614556Z + [[ distributed == *dynamo_eager* ]] 2024-04-03T16:23:21.8615111Z + [[ distributed == *aot_eager* ]] 2024-04-03T16:23:21.8615593Z + [[ distributed == *aot_inductor* ]] 2024-04-03T16:23:21.8616111Z + [[ distributed == *inductor* ]] 2024-04-03T16:23:21.8616592Z + [[ distributed == *dynamic* ]] 2024-04-03T16:23:21.8617093Z + [[ distributed == *cpu_inductor* ]] 2024-04-03T16:23:21.8617799Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-04-03T16:23:21.8629930Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *tbb* ]] 2024-04-03T16:23:21.8647637Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-04-03T16:23:21.8648411Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-04-03T16:23:21.8667057Z + cd test 2024-04-03T16:23:21.8667916Z + python -c 'import torch; print(torch.__config__.show())' 2024-04-03T16:23:23.5258553Z PyTorch built with: 2024-04-03T16:23:23.5259231Z - GCC 9.4 2024-04-03T16:23:23.5259626Z - C++ Version: 201703 2024-04-03T16:23:23.5260680Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-04-03T16:23:23.5262094Z - Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361) 2024-04-03T16:23:23.5262937Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-04-03T16:23:23.5263583Z - LAPACK is enabled (usually provided by MKL) 2024-04-03T16:23:23.5264188Z - NNPACK is enabled 2024-04-03T16:23:23.5264645Z - CPU capability usage: AVX2 2024-04-03T16:23:23.5265156Z - CUDA Runtime 11.8 2024-04-03T16:23:23.5266496Z - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52 2024-04-03T16:23:23.5267244Z - CuDNN 8.7 2024-04-03T16:23:23.5267639Z - Magma 2.6.1 2024-04-03T16:23:23.5276929Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-04-03T16:23:23.5285744Z 2024-04-03T16:23:23.7723369Z + cd test 2024-04-03T16:23:23.7724201Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-04-03T16:23:25.3456556Z ATen/Parallel: 2024-04-03T16:23:25.3457535Z at::get_num_threads() : 16 2024-04-03T16:23:25.3458113Z at::get_num_interop_threads() : 32 2024-04-03T16:23:25.3458643Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-04-03T16:23:25.3459145Z omp_get_max_threads() : 16 2024-04-03T16:23:25.3460339Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-04-03T16:23:25.3461395Z mkl_get_max_threads() : 16 2024-04-03T16:23:25.3462187Z Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361) 2024-04-03T16:23:25.3462966Z std::thread::hardware_concurrency() : 32 2024-04-03T16:23:25.3463503Z Environment variables: 2024-04-03T16:23:25.3463930Z OMP_NUM_THREADS : [not set] 2024-04-03T16:23:25.3464373Z MKL_NUM_THREADS : [not set] 2024-04-03T16:23:25.3464841Z ATen parallel backend: OpenMP 2024-04-03T16:23:25.3465834Z 2024-04-03T16:23:25.5833676Z + [[ distributed == *backward* ]] 2024-04-03T16:23:25.5834895Z + [[ distributed == *xla* ]] 2024-04-03T16:23:25.5835441Z + [[ distributed == *executorch* ]] 2024-04-03T16:23:25.5836086Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-04-03T16:23:25.5836870Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-04-03T16:23:25.5837503Z + [[ distributed == distributed ]] 2024-04-03T16:23:25.5837988Z + test_distributed 2024-04-03T16:23:25.5838505Z + echo 'Testing distributed python tests' 2024-04-03T16:23:25.5839053Z Testing distributed python tests 2024-04-03T16:23:25.5839831Z + python test/run_test.py --distributed-tests --shard 3 3 --verbose 2024-04-03T16:23:25.6958950Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-04-03T16:23:25.6960411Z import pkg_resources 2024-04-03T16:23:28.7661774Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-04-03T16:23:28.8260717Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-04-03T16:23:28.8823823Z Ignoring disabled issues: [''] 2024-04-03T16:23:28.8910038Z Found test times from artifacts 2024-04-03T16:23:28.9240932Z Found test times from artifacts 2024-04-03T16:23:28.9250638Z Running all tests 2024-04-03T16:23:28.9387444Z Name: tests to run (est. time: 72.86min) 2024-04-03T16:23:28.9388942Z Serial tests (62): 2024-04-03T16:23:28.9390031Z distributed/_shard/sharded_tensor/test_logger 1/1 2024-04-03T16:23:28.9391036Z distributed/_tensor/test_init 1/1 2024-04-03T16:23:28.9391609Z distributed/_composable/test_checkpoint 1/1 2024-04-03T16:23:28.9392930Z distributed/_tensor/test_dtensor_compile 1/1 2024-04-03T16:23:28.9394284Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 2024-04-03T16:23:28.9395226Z distributed/checkpoint/test_save_load_api 1/1 2024-04-03T16:23:28.9395868Z distributed/_spmd/test_data_parallel 1/1 2024-04-03T16:23:28.9396467Z distributed/test_distributed_spawn 1/7 2024-04-03T16:23:28.9397049Z distributed/test_distributed_spawn 4/7 2024-04-03T16:23:28.9397613Z distributed/test_distributed_spawn 7/7 2024-04-03T16:23:28.9398217Z distributed/optim/test_named_optimizer 1/1 2024-04-03T16:23:28.9398831Z distributed/_composable/test_compose 1/1 2024-04-03T16:23:28.9399436Z distributed/_tensor/debug/test_comm_mode 1/1 2024-04-03T16:23:28.9400143Z distributed/tensor/parallel/test_tp_examples 1/1 2024-04-03T16:23:28.9400807Z distributed/test_inductor_collectives 1/1 2024-04-03T16:23:28.9401445Z distributed/fsdp/test_fsdp_backward_prefetch 1/1 2024-04-03T16:23:28.9402110Z distributed/_composable/test_replicate 1/1 2024-04-03T16:23:28.9402803Z distributed/_shard/sharding_plan/test_sharding_plan 1/1 2024-04-03T16:23:28.9403612Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 2024-04-03T16:23:28.9404499Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 2024-04-03T16:23:28.9405227Z distributed/_tensor/test_optimizers 1/1 2024-04-03T16:23:28.9405918Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 2024-04-03T16:23:28.9406619Z distributed/fsdp/test_fsdp_flatten_params 1/1 2024-04-03T16:23:28.9407226Z distributed/test_functional_api 1/1 2024-04-03T16:23:28.9407828Z distributed/fsdp/test_fsdp_ignored_modules 1/1 2024-04-03T16:23:28.9408473Z distributed/checkpoint/test_checkpoint 1/1 2024-04-03T16:23:28.9409073Z distributed/rpc/test_share_memory 1/1 2024-04-03T16:23:28.9409690Z distributed/fsdp/test_fsdp_freezing_weights 1/1 2024-04-03T16:23:28.9410318Z distributed/fsdp/test_fsdp_traversal 1/1 2024-04-03T16:23:28.9410902Z distributed/fsdp/test_fsdp_input 1/1 2024-04-03T16:23:28.9411459Z distributed/fsdp/test_fsdp_uneven 1/1 2024-04-03T16:23:28.9412288Z distributed/fsdp/test_fsdp_multiple_wrapping 1/1 2024-04-03T16:23:28.9413081Z distributed/_composable/fully_shard/test_fully_shard_runtime 1/1 2024-04-03T16:23:28.9413843Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 2024-04-03T16:23:28.9414451Z distributed/fsdp/test_fsdp_exec_order 1/1 2024-04-03T16:23:28.9415052Z distributed/pipeline/sync/test_pipe 1/1 2024-04-03T16:23:28.9415653Z distributed/fsdp/test_fsdp_fine_tune 1/1 2024-04-03T16:23:28.9416212Z distributed/fsdp/test_fsdp_core 1/2 2024-04-03T16:23:28.9416902Z distributed/algorithms/quantization/test_quantization 1/1 2024-04-03T16:23:28.9417629Z distributed/fsdp/test_fsdp_checkpoint 1/1 2024-04-03T16:23:28.9418225Z distributed/pipeline/sync/test_worker 1/1 2024-04-03T16:23:28.9418865Z distributed/pipeline/sync/test_transparency 1/1 2024-04-03T16:23:28.9419696Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 1/1 2024-04-03T16:23:28.9420502Z distributed/fsdp/test_fsdp_unshard_params 1/1 2024-04-03T16:23:28.9421155Z distributed/fsdp/test_fsdp_hybrid_shard 1/1 2024-04-03T16:23:28.9421782Z distributed/pipeline/sync/test_inplace 1/1 2024-04-03T16:23:28.9422405Z distributed/rpc/cuda/test_tensorpipe_agent 1/2 2024-04-03T16:23:28.9423033Z distributed/rpc/test_faulty_agent 1/1 2024-04-03T16:23:28.9423587Z distributed/test_pg_wrapper 1/1 2024-04-03T16:23:28.9424212Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 2024-04-03T16:23:28.9424882Z distributed/test_c10d_spawn_ucc 1/1 2024-04-03T16:23:28.9425823Z distributed/pipeline/sync/test_balance 1/1 2024-04-03T16:23:28.9426631Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 2024-04-03T16:23:28.9427388Z distributed/pipeline/sync/test_dependency 1/1 2024-04-03T16:23:28.9428037Z distributed/pipeline/sync/skip/test_api 1/1 2024-04-03T16:23:28.9428729Z distributed/pipeline/sync/test_deferred_batch_norm 1/1 2024-04-03T16:23:28.9429399Z distributed/checkpoint/test_state_dict 1/1 2024-04-03T16:23:28.9430157Z distributed/pipeline/sync/skip/test_verify_skippables 1/1 2024-04-03T16:23:28.9430815Z distributed/test_data_parallel 1/1 2024-04-03T16:23:28.9431322Z distributed/test_store 1/1 2024-04-03T16:23:28.9431921Z distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 2024-04-03T16:23:28.9432568Z distributed/test_c10d_nccl 2/2 2024-04-03T16:23:28.9433041Z Parallel tests (0): 2024-04-03T16:23:28.9433469Z Name: excluded (est. time: 0.0min) 2024-04-03T16:23:28.9433956Z Serial tests (0): 2024-04-03T16:23:28.9434340Z Parallel tests (0): 2024-04-03T16:23:28.9435110Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-04-03T16:23:28.9452738Z Running distributed/_shard/sharded_tensor/test_logger 1/1 ... [2024-04-03 16:23:28.944910] 2024-04-03T16:23:28.9457861Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:23:28.945335] 2024-04-03T16:23:32.5175520Z 2024-04-03T16:23:32.5177785Z distributed/_shard/sharded_tensor/test_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_bdc5285b23114a1a_.log 2024-04-03T16:23:32.5180109Z Running 1 items in this shard: test/distributed/_shard/sharded_tensor/test_logger.py::ShardingSpecLoggerTest::test_get_or_create_logger 2024-04-03T16:23:32.5181134Z 2024-04-03T16:23:32.5182548Z Running distributed/_tensor/test_init 1/1 ... [2024-04-03 16:23:32.517972] 2024-04-03T16:23:32.5188825Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:23:32.518404] 2024-04-03T16:24:00.1432768Z 2024-04-03T16:24:00.1434708Z distributed/_tensor/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_init_1.1_5320e5fe4e0137ce_.log 2024-04-03T16:24:00.1440014Z Running 7 items in this shard: test/distributed/_tensor/test_init.py::DTensorInitOpsTest::test_init_ops, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_empty, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_full, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_ones, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros_full_mesh, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros_submesh 2024-04-03T16:24:00.1444258Z 2024-04-03T16:24:00.1444837Z Running distributed/_composable/test_checkpoint 1/1 ... [2024-04-03 16:24:00.143594] 2024-04-03T16:24:00.1447350Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:24:00.144000] 2024-04-03T16:24:04.0238805Z 2024-04-03T16:24:04.0240821Z distributed/_composable/test_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_checkpoint_1.1_f592fa157770fb92_.log 2024-04-03T16:24:04.0245368Z Running 5 items in this shard: test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_clears_state_on_error_in_forward, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_multi_args, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_random_cpu, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_tensor_only_cpu, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_tensor_only_gpu 2024-04-03T16:24:04.0248524Z 2024-04-03T16:24:04.0249739Z Running distributed/_tensor/test_dtensor_compile 1/1 ... [2024-04-03 16:24:04.024119] 2024-04-03T16:24:04.0254498Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:24:04.024518] 2024-04-03T16:24:43.5231475Z 2024-04-03T16:24:43.5233368Z distributed/_tensor/test_dtensor_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_compile_1.1_04f66d18ea139d04_.log 2024-04-03T16:24:43.5268883Z Running 32 items in this shard: test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_different_gradient_placement, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamo_device_mesh_attrs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_noncontiguous_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_graph_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs_forward_hook, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True_use_native_funcol_True 2024-04-03T16:24:43.5319712Z 2024-04-03T16:24:43.5320920Z Running distributed/checkpoint/e2e/test_fsdp_ep 1/1 ... [2024-04-03 16:24:43.523413] 2024-04-03T16:24:43.5325894Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_fsdp_ep.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:24:43.524788] 2024-04-03T16:24:50.4027292Z 2024-04-03T16:24:50.4029349Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_7b2189c283a19011_.log 2024-04-03T16:24:50.4031450Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_fsdp_ep.py::TestFSDPWithEP::test_e2e 2024-04-03T16:24:50.4032399Z 2024-04-03T16:24:50.4033650Z Running distributed/checkpoint/test_save_load_api 1/1 ... [2024-04-03 16:24:50.403040] 2024-04-03T16:24:50.4039282Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_save_load_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:24:50.403478] 2024-04-03T16:24:57.3316954Z 2024-04-03T16:24:57.3319065Z distributed/checkpoint/test_save_load_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_save_load_api_1.1_ae554707cd6c3fa6_.log 2024-04-03T16:24:57.3321907Z Running 1 items in this shard: test/distributed/checkpoint/test_save_load_api.py::TestSaveAndLoadAPI::test_auto_detect 2024-04-03T16:24:57.3322847Z 2024-04-03T16:24:57.3323382Z Running distributed/_spmd/test_data_parallel 1/1 ... [2024-04-03 16:24:57.331922] 2024-04-03T16:24:57.3327770Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_spmd/test_data_parallel.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:24:57.332324] 2024-04-03T16:25:00.4037847Z 2024-04-03T16:25:00.4039588Z distributed/_spmd/test_data_parallel 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._spmd.test_data_parallel_1.1_38bc5c6b5868eb5b_.log 2024-04-03T16:25:00.4040965Z 2024-04-03T16:25:00.4043412Z Running distributed/test_distributed_spawn 1/7 ... [2024-04-03 16:25:00.403991] 2024-04-03T16:25:00.4052528Z /usr/bin/mpiexec 2024-04-03T16:25:00.4053488Z MPI not available -- MPI backend tests will be skipped 2024-04-03T16:25:00.4054869Z Running distributed tests for the test backend with env init_method 2024-04-03T16:25:00.4063274Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:25:00.405876] 2024-04-03T16:25:03.8983934Z 2024-04-03T16:25:03.8986288Z distributed/test_distributed_spawn 1/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.7_0601af37cdbb41cd_.log 2024-04-03T16:25:03.8987811Z Running 0 items in this shard: 2024-04-03T16:25:03.8988207Z 2024-04-03T16:25:03.8995536Z Running distributed tests for the test backend with file init_method 2024-04-03T16:25:03.9002452Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:25:03.899791] 2024-04-03T16:25:07.3627385Z 2024-04-03T16:25:07.3629517Z distributed/test_distributed_spawn 1/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.7_edd3d9f7ae01ba50_.log 2024-04-03T16:25:07.3631033Z Running 0 items in this shard: 2024-04-03T16:25:07.3631366Z 2024-04-03T16:25:07.3638172Z Running distributed tests for the nccl backend with env init_method 2024-04-03T16:25:07.3645153Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:25:07.364099] 2024-04-03T16:29:52.8330216Z 2024-04-03T16:29:52.8331992Z distributed/test_distributed_spawn 1/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.7_53301db6d035a5c0_.log 2024-04-03T16:29:52.8362370Z Running 38 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-04-03T16:29:52.8389509Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2024-04-03T16:29:52.8391452Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2024-04-03T16:29:52.8393349Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2024-04-03T16:29:52.8395039Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2024-04-03T16:29:52.8396937Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min 2024-04-03T16:29:52.8398704Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2024-04-03T16:29:52.8400384Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2024-04-03T16:29:52.8402084Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2024-04-03T16:29:52.8403822Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2024-04-03T16:29:52.8405469Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2024-04-03T16:29:52.8407200Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2024-04-03T16:29:52.8408975Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2024-04-03T16:29:52.8410694Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2024-04-03T16:29:52.8412751Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2024-04-03T16:29:52.8414672Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2024-04-03T16:29:52.8416459Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2024-04-03T16:29:52.8418252Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2024-04-03T16:29:52.8419999Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2024-04-03T16:29:52.8421732Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2024-04-03T16:29:52.8423652Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2024-04-03T16:29:52.8426297Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2024-04-03T16:29:52.8428450Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2024-04-03T16:29:52.8430449Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2024-04-03T16:29:52.8432229Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2024-04-03T16:29:52.8434076Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn 2024-04-03T16:29:52.8436038Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2024-04-03T16:29:52.8437903Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2024-04-03T16:29:52.8439696Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2024-04-03T16:29:52.8441657Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order 2024-04-03T16:29:52.8443486Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2024-04-03T16:29:52.8445368Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2024-04-03T16:29:52.8447176Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2024-04-03T16:29:52.8448779Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2024-04-03T16:29:52.8450350Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda 2024-04-03T16:29:52.8451999Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2024-04-03T16:29:52.8453725Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2024-04-03T16:29:52.8455380Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2024-04-03T16:29:52.8457109Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-04-03T16:29:52.8458301Z 2024-04-03T16:29:52.8458689Z Running distributed tests for the nccl backend with file init_method 2024-04-03T16:29:52.8461163Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:29:52.835556] 2024-04-03T16:34:38.3671517Z 2024-04-03T16:34:38.3675402Z distributed/test_distributed_spawn 1/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.7_dc23d305cc089c79_.log 2024-04-03T16:34:38.3702334Z Running 38 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-04-03T16:34:38.3729386Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2024-04-03T16:34:38.3731370Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2024-04-03T16:34:38.3733259Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2024-04-03T16:34:38.3734926Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2024-04-03T16:34:38.3736647Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min 2024-04-03T16:34:38.3738413Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2024-04-03T16:34:38.3740109Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2024-04-03T16:34:38.3741790Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2024-04-03T16:34:38.3743502Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2024-04-03T16:34:38.3745150Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2024-04-03T16:34:38.3747342Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2024-04-03T16:34:38.3749314Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2024-04-03T16:34:38.3751010Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2024-04-03T16:34:38.3752945Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2024-04-03T16:34:38.3754870Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2024-04-03T16:34:38.3756644Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2024-04-03T16:34:38.3758432Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2024-04-03T16:34:38.3760175Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2024-04-03T16:34:38.3761939Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2024-04-03T16:34:38.3763835Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2024-04-03T16:34:38.3766094Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2024-04-03T16:34:38.3768329Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2024-04-03T16:34:38.3770327Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2024-04-03T16:34:38.3772093Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2024-04-03T16:34:38.3773949Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn 2024-04-03T16:34:38.3775894Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2024-04-03T16:34:38.3777781Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2024-04-03T16:34:38.3779555Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2024-04-03T16:34:38.3781415Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order 2024-04-03T16:34:38.3783259Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2024-04-03T16:34:38.3785160Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2024-04-03T16:34:38.3787522Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2024-04-03T16:34:38.3789127Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2024-04-03T16:34:38.3790698Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda 2024-04-03T16:34:38.3792363Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2024-04-03T16:34:38.3794065Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2024-04-03T16:34:38.3795894Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2024-04-03T16:34:38.3797685Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-04-03T16:34:38.3798722Z 2024-04-03T16:34:38.3799103Z Running distributed tests for the gloo backend with env init_method 2024-04-03T16:34:38.3801534Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:34:38.369508] 2024-04-03T16:39:01.9197389Z 2024-04-03T16:39:01.9199377Z distributed/test_distributed_spawn 1/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.7_00533d651908b9e9_.log 2024-04-03T16:39:01.9228505Z Running 38 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-04-03T16:39:01.9255067Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2024-04-03T16:39:01.9257062Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2024-04-03T16:39:01.9258995Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2024-04-03T16:39:01.9260704Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2024-04-03T16:39:01.9262404Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min 2024-04-03T16:39:01.9264150Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2024-04-03T16:39:01.9266269Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2024-04-03T16:39:01.9267977Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2024-04-03T16:39:01.9269741Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2024-04-03T16:39:01.9271374Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2024-04-03T16:39:01.9273115Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2024-04-03T16:39:01.9274881Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2024-04-03T16:39:01.9276606Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2024-04-03T16:39:01.9278516Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2024-04-03T16:39:01.9280421Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2024-04-03T16:39:01.9282222Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2024-04-03T16:39:01.9283980Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2024-04-03T16:39:01.9285716Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2024-04-03T16:39:01.9287460Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2024-04-03T16:39:01.9289543Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2024-04-03T16:39:01.9291645Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2024-04-03T16:39:01.9293850Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2024-04-03T16:39:01.9295896Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2024-04-03T16:39:01.9297667Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2024-04-03T16:39:01.9299504Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn 2024-04-03T16:39:01.9301444Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2024-04-03T16:39:01.9303318Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2024-04-03T16:39:01.9305220Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2024-04-03T16:39:01.9307432Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order 2024-04-03T16:39:01.9309257Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2024-04-03T16:39:01.9311166Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2024-04-03T16:39:01.9312945Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2024-04-03T16:39:01.9314537Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2024-04-03T16:39:01.9316106Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda 2024-04-03T16:39:01.9317768Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2024-04-03T16:39:01.9319459Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2024-04-03T16:39:01.9321100Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2024-04-03T16:39:01.9322844Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-04-03T16:39:01.9323877Z 2024-04-03T16:39:01.9324262Z Running distributed tests for the gloo backend with file init_method 2024-04-03T16:39:01.9326705Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:39:01.922056] 2024-04-03T16:43:25.8247491Z 2024-04-03T16:43:25.8251101Z distributed/test_distributed_spawn 1/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.7_059406b002abc96e_.log 2024-04-03T16:43:25.8281215Z Running 38 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-04-03T16:43:25.8308343Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2024-04-03T16:43:25.8310534Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2024-04-03T16:43:25.8312428Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2024-04-03T16:43:25.8314155Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2024-04-03T16:43:25.8315868Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min 2024-04-03T16:43:25.8317631Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2024-04-03T16:43:25.8319329Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2024-04-03T16:43:25.8321014Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2024-04-03T16:43:25.8322740Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2024-04-03T16:43:25.8324382Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2024-04-03T16:43:25.8326225Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2024-04-03T16:43:25.8328019Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2024-04-03T16:43:25.8329738Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2024-04-03T16:43:25.8331666Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2024-04-03T16:43:25.8333577Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2024-04-03T16:43:25.8335360Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2024-04-03T16:43:25.8337150Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2024-04-03T16:43:25.8338901Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2024-04-03T16:43:25.8340636Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2024-04-03T16:43:25.8342551Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2024-04-03T16:43:25.8344685Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2024-04-03T16:43:25.8347429Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2024-04-03T16:43:25.8349427Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2024-04-03T16:43:25.8351206Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2024-04-03T16:43:25.8353055Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn 2024-04-03T16:43:25.8355176Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2024-04-03T16:43:25.8357039Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2024-04-03T16:43:25.8358830Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2024-04-03T16:43:25.8360646Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order 2024-04-03T16:43:25.8362477Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2024-04-03T16:43:25.8364358Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2024-04-03T16:43:25.8366160Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2024-04-03T16:43:25.8367758Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2024-04-03T16:43:25.8369327Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda 2024-04-03T16:43:25.8371104Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2024-04-03T16:43:25.8372846Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2024-04-03T16:43:25.8374495Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2024-04-03T16:43:25.8376238Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-04-03T16:43:25.8377289Z 2024-04-03T16:43:25.8377646Z Running distributed tests for the ucc backend with env init_method 2024-04-03T16:43:25.8380114Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:43:25.827311] 2024-04-03T16:47:34.7352717Z 2024-04-03T16:47:34.7356582Z distributed/test_distributed_spawn 1/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.7_117a7f4f5f876735_.log 2024-04-03T16:47:34.7383622Z Running 38 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-04-03T16:47:34.7410767Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2024-04-03T16:47:34.7412717Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2024-04-03T16:47:34.7414623Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2024-04-03T16:47:34.7416315Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2024-04-03T16:47:34.7418043Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min 2024-04-03T16:47:34.7419807Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2024-04-03T16:47:34.7421511Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2024-04-03T16:47:34.7423183Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2024-04-03T16:47:34.7425148Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2024-04-03T16:47:34.7427268Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2024-04-03T16:47:34.7428997Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2024-04-03T16:47:34.7430779Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2024-04-03T16:47:34.7432502Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2024-04-03T16:47:34.7434434Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2024-04-03T16:47:34.7436344Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2024-04-03T16:47:34.7438127Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2024-04-03T16:47:34.7439905Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2024-04-03T16:47:34.7441798Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2024-04-03T16:47:34.7443566Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2024-04-03T16:47:34.7445482Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2024-04-03T16:47:34.7447604Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2024-04-03T16:47:34.7449829Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2024-04-03T16:47:34.7451817Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2024-04-03T16:47:34.7453596Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2024-04-03T16:47:34.7455446Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn 2024-04-03T16:47:34.7457395Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2024-04-03T16:47:34.7459265Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2024-04-03T16:47:34.7461057Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2024-04-03T16:47:34.7462859Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order 2024-04-03T16:47:34.7464703Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2024-04-03T16:47:34.7467114Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2024-04-03T16:47:34.7468924Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2024-04-03T16:47:34.7470681Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2024-04-03T16:47:34.7472255Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda 2024-04-03T16:47:34.7473902Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2024-04-03T16:47:34.7475625Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2024-04-03T16:47:34.7477270Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2024-04-03T16:47:34.7479005Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-04-03T16:47:34.7480043Z 2024-04-03T16:47:34.7480403Z Running distributed tests for the ucc backend with file init_method 2024-04-03T16:47:34.7482871Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:47:34.738134] 2024-04-03T16:51:45.0475928Z 2024-04-03T16:51:45.0480738Z distributed/test_distributed_spawn 1/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.7_7f9804eb8b719486_.log 2024-04-03T16:51:45.0512014Z Running 38 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-04-03T16:51:45.0540561Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2024-04-03T16:51:45.0542486Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2024-04-03T16:51:45.0544415Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2024-04-03T16:51:45.0546545Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2024-04-03T16:51:45.0548258Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min 2024-04-03T16:51:45.0549997Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2024-04-03T16:51:45.0551694Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2024-04-03T16:51:45.0553378Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2024-04-03T16:51:45.0555102Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2024-04-03T16:51:45.0556735Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2024-04-03T16:51:45.0558472Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2024-04-03T16:51:45.0560235Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2024-04-03T16:51:45.0561957Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2024-04-03T16:51:45.0563871Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2024-04-03T16:51:45.0565782Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2024-04-03T16:51:45.0567746Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2024-04-03T16:51:45.0569526Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2024-04-03T16:51:45.0571241Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2024-04-03T16:51:45.0572994Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2024-04-03T16:51:45.0574905Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2024-04-03T16:51:45.0577058Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2024-04-03T16:51:45.0579260Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2024-04-03T16:51:45.0581257Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2024-04-03T16:51:45.0583017Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2024-04-03T16:51:45.0584969Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn 2024-04-03T16:51:45.0587335Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2024-04-03T16:51:45.0589220Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2024-04-03T16:51:45.0591017Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2024-04-03T16:51:45.0592816Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order 2024-04-03T16:51:45.0594624Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2024-04-03T16:51:45.0596544Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2024-04-03T16:51:45.0598339Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2024-04-03T16:51:45.0599936Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2024-04-03T16:51:45.0601484Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda 2024-04-03T16:51:45.0603151Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2024-04-03T16:51:45.0604863Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2024-04-03T16:51:45.0606493Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2024-04-03T16:51:45.0608234Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-04-03T16:51:45.0609290Z 2024-04-03T16:51:45.0609832Z Running distributed/test_distributed_spawn 4/7 ... [2024-04-03 16:51:45.049853] 2024-04-03T16:51:45.0610585Z /usr/bin/mpiexec 2024-04-03T16:51:45.0611137Z MPI not available -- MPI backend tests will be skipped 2024-04-03T16:51:45.0612115Z Running distributed tests for the test backend with env init_method 2024-04-03T16:51:45.0614537Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:51:45.051699] 2024-04-03T16:51:48.6140175Z 2024-04-03T16:51:48.6142035Z distributed/test_distributed_spawn 4/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.7_b0f9aba2f77ecf94_.log 2024-04-03T16:51:48.6143537Z Running 0 items in this shard: 2024-04-03T16:51:48.6143885Z 2024-04-03T16:51:48.6151040Z Running distributed tests for the test backend with file init_method 2024-04-03T16:51:48.6157940Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:51:48.615369] 2024-04-03T16:51:52.1123588Z 2024-04-03T16:51:52.1125363Z distributed/test_distributed_spawn 4/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.7_99b891ef90ace298_.log 2024-04-03T16:51:52.1126878Z Running 0 items in this shard: 2024-04-03T16:51:52.1127203Z 2024-04-03T16:51:52.1134438Z Running distributed tests for the nccl backend with env init_method 2024-04-03T16:51:52.1141325Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:51:52.113704] 2024-04-03T16:55:56.1612489Z 2024-04-03T16:55:56.1614596Z distributed/test_distributed_spawn 4/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.7_8446e0b7e6c8743d_.log 2024-04-03T16:55:56.1645250Z Running 41 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_default_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2024-04-03T16:55:56.1702085Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2024-04-03T16:55:56.1706478Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2024-04-03T16:55:56.1710704Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2024-04-03T16:55:56.1714658Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2024-04-03T16:55:56.1718405Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2024-04-03T16:55:56.1722135Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2024-04-03T16:55:56.1725928Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2024-04-03T16:55:56.1729642Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2024-04-03T16:55:56.1733149Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2024-04-03T16:55:56.1736660Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2024-04-03T16:55:56.1740014Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2024-04-03T16:55:56.1743771Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2024-04-03T16:55:56.1747638Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group 2024-04-03T16:55:56.1751300Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split 2024-04-03T16:55:56.1755340Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda 2024-04-03T16:55:56.1759340Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2024-04-03T16:55:56.1763384Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2024-04-03T16:55:56.1767381Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda 2024-04-03T16:55:56.1771033Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group 2024-04-03T16:55:56.1774310Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2024-04-03T16:55:56.1778106Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2024-04-03T16:55:56.1781740Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group 2024-04-03T16:55:56.1785335Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2024-04-03T16:55:56.1789276Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2024-04-03T16:55:56.1792755Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2024-04-03T16:55:56.1796135Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2024-04-03T16:55:56.1799831Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2024-04-03T16:55:56.1803677Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2024-04-03T16:55:56.1807606Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_default_group 2024-04-03T16:55:56.1811301Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2024-04-03T16:55:56.1814421Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2024-04-03T16:55:56.1817612Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future 2024-04-03T16:55:56.1820957Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2024-04-03T16:55:56.1824470Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler 2024-04-03T16:55:56.1828616Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2024-04-03T16:55:56.1832486Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2024-04-03T16:55:56.1836213Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort 2024-04-03T16:55:56.1839557Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2024-04-03T16:55:56.1842775Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2024-04-03T16:55:56.1846019Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2024-04-03T16:55:56.1849610Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2024-04-03T16:55:56.1851773Z 2024-04-03T16:55:56.1852612Z Running distributed tests for the nccl backend with file init_method 2024-04-03T16:55:56.1857637Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:55:56.163698] 2024-04-03T17:00:01.1028140Z 2024-04-03T17:00:01.1029936Z distributed/test_distributed_spawn 4/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.7_20c512541393f1d1_.log 2024-04-03T17:00:01.1060909Z Running 41 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_default_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2024-04-03T17:00:01.1089424Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2024-04-03T17:00:01.1091391Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2024-04-03T17:00:01.1093457Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2024-04-03T17:00:01.1095411Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2024-04-03T17:00:01.1097225Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2024-04-03T17:00:01.1099030Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2024-04-03T17:00:01.1100887Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2024-04-03T17:00:01.1102640Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2024-04-03T17:00:01.1104338Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2024-04-03T17:00:01.1106302Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2024-04-03T17:00:01.1107925Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2024-04-03T17:00:01.1109521Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2024-04-03T17:00:01.1111208Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group 2024-04-03T17:00:01.1112953Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split 2024-04-03T17:00:01.1114826Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda 2024-04-03T17:00:01.1116754Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2024-04-03T17:00:01.1118860Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2024-04-03T17:00:01.1120841Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda 2024-04-03T17:00:01.1122604Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group 2024-04-03T17:00:01.1124205Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2024-04-03T17:00:01.1125874Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2024-04-03T17:00:01.1127600Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group 2024-04-03T17:00:01.1129306Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2024-04-03T17:00:01.1131033Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2024-04-03T17:00:01.1132687Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2024-04-03T17:00:01.1134430Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2024-04-03T17:00:01.1136207Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2024-04-03T17:00:01.1138069Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2024-04-03T17:00:01.1139968Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_default_group 2024-04-03T17:00:01.1141759Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2024-04-03T17:00:01.1143320Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2024-04-03T17:00:01.1144859Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future 2024-04-03T17:00:01.1146649Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2024-04-03T17:00:01.1148362Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler 2024-04-03T17:00:01.1150219Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2024-04-03T17:00:01.1152112Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2024-04-03T17:00:01.1153790Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort 2024-04-03T17:00:01.1155391Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2024-04-03T17:00:01.1156965Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2024-04-03T17:00:01.1158537Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2024-04-03T17:00:01.1160270Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2024-04-03T17:00:01.1161445Z 2024-04-03T17:00:01.1161811Z Running distributed tests for the gloo backend with env init_method 2024-04-03T17:00:01.1164293Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:00:01.105814] 2024-04-03T17:04:13.8063089Z 2024-04-03T17:04:13.8065254Z distributed/test_distributed_spawn 4/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.7_14afd39b8a1c0b1b_.log 2024-04-03T17:04:13.8097853Z Running 41 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_default_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2024-04-03T17:04:13.8126142Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2024-04-03T17:04:13.8128075Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2024-04-03T17:04:13.8130132Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2024-04-03T17:04:13.8132090Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2024-04-03T17:04:13.8134051Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2024-04-03T17:04:13.8135872Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2024-04-03T17:04:13.8137722Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2024-04-03T17:04:13.8139505Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2024-04-03T17:04:13.8141198Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2024-04-03T17:04:13.8142861Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2024-04-03T17:04:13.8144475Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2024-04-03T17:04:13.8146549Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2024-04-03T17:04:13.8148242Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group 2024-04-03T17:04:13.8149970Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split 2024-04-03T17:04:13.8151872Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda 2024-04-03T17:04:13.8153798Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2024-04-03T17:04:13.8155741Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2024-04-03T17:04:13.8157708Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda 2024-04-03T17:04:13.8159479Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group 2024-04-03T17:04:13.8161076Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2024-04-03T17:04:13.8162904Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2024-04-03T17:04:13.8164607Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group 2024-04-03T17:04:13.8166312Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2024-04-03T17:04:13.8168052Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2024-04-03T17:04:13.8169685Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2024-04-03T17:04:13.8171316Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2024-04-03T17:04:13.8173098Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2024-04-03T17:04:13.8174951Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2024-04-03T17:04:13.8176810Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_default_group 2024-04-03T17:04:13.8178702Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2024-04-03T17:04:13.8180363Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2024-04-03T17:04:13.8181897Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future 2024-04-03T17:04:13.8183498Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2024-04-03T17:04:13.8185216Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler 2024-04-03T17:04:13.8187480Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2024-04-03T17:04:13.8189362Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2024-04-03T17:04:13.8191037Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort 2024-04-03T17:04:13.8192668Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2024-04-03T17:04:13.8194230Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2024-04-03T17:04:13.8195804Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2024-04-03T17:04:13.8197515Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2024-04-03T17:04:13.8198556Z 2024-04-03T17:04:13.8198926Z Running distributed tests for the gloo backend with file init_method 2024-04-03T17:04:13.8201416Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:04:13.808994] 2024-04-03T17:08:26.6828645Z 2024-04-03T17:08:26.6831058Z distributed/test_distributed_spawn 4/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.7_dc8c6a2d91480ebb_.log 2024-04-03T17:08:26.6863554Z Running 41 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_default_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2024-04-03T17:08:26.6891713Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2024-04-03T17:08:26.6893682Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2024-04-03T17:08:26.6895757Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2024-04-03T17:08:26.6897750Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2024-04-03T17:08:26.6899564Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2024-04-03T17:08:26.6901360Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2024-04-03T17:08:26.6903191Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2024-04-03T17:08:26.6904960Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2024-04-03T17:08:26.6907294Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2024-04-03T17:08:26.6908999Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2024-04-03T17:08:26.6910588Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2024-04-03T17:08:26.6912187Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2024-04-03T17:08:26.6913874Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group 2024-04-03T17:08:26.6915597Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split 2024-04-03T17:08:26.6917492Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda 2024-04-03T17:08:26.6919422Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2024-04-03T17:08:26.6921360Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2024-04-03T17:08:26.6923325Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda 2024-04-03T17:08:26.6925105Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group 2024-04-03T17:08:26.6926694Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2024-04-03T17:08:26.6928352Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2024-04-03T17:08:26.6930053Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group 2024-04-03T17:08:26.6931759Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2024-04-03T17:08:26.6933493Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2024-04-03T17:08:26.6935135Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2024-04-03T17:08:26.6936887Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2024-04-03T17:08:26.6938661Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2024-04-03T17:08:26.6940524Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2024-04-03T17:08:26.6942413Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_default_group 2024-04-03T17:08:26.6944175Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2024-04-03T17:08:26.6946230Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2024-04-03T17:08:26.6947797Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future 2024-04-03T17:08:26.6949413Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2024-04-03T17:08:26.6951103Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler 2024-04-03T17:08:26.6953101Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2024-04-03T17:08:26.6955003Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2024-04-03T17:08:26.6956666Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort 2024-04-03T17:08:26.6958291Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2024-04-03T17:08:26.6959851Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2024-04-03T17:08:26.6961413Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2024-04-03T17:08:26.6963119Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2024-04-03T17:08:26.6964157Z 2024-04-03T17:08:26.6964512Z Running distributed tests for the ucc backend with env init_method 2024-04-03T17:08:26.6966968Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:08:26.685368] 2024-04-03T17:12:08.8579880Z 2024-04-03T17:12:08.8585013Z distributed/test_distributed_spawn 4/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.7_f9bb301952eeb94f_.log 2024-04-03T17:12:08.8620255Z Running 41 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_default_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2024-04-03T17:12:08.8648398Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2024-04-03T17:12:08.8650355Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2024-04-03T17:12:08.8652422Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2024-04-03T17:12:08.8654356Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2024-04-03T17:12:08.8656424Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2024-04-03T17:12:08.8658226Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2024-04-03T17:12:08.8660082Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2024-04-03T17:12:08.8661840Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2024-04-03T17:12:08.8663769Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2024-04-03T17:12:08.8665462Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2024-04-03T17:12:08.8667406Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2024-04-03T17:12:08.8669005Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2024-04-03T17:12:08.8670686Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group 2024-04-03T17:12:08.8672577Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split 2024-04-03T17:12:08.8674507Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda 2024-04-03T17:12:08.8676417Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2024-04-03T17:12:08.8678368Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2024-04-03T17:12:08.8680345Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda 2024-04-03T17:12:08.8682128Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group 2024-04-03T17:12:08.8683706Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2024-04-03T17:12:08.8685373Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2024-04-03T17:12:08.8687097Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group 2024-04-03T17:12:08.8688806Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2024-04-03T17:12:08.8690528Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2024-04-03T17:12:08.8692172Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2024-04-03T17:12:08.8693798Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2024-04-03T17:12:08.8695561Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2024-04-03T17:12:08.8697420Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2024-04-03T17:12:08.8699305Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_default_group 2024-04-03T17:12:08.8701238Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2024-04-03T17:12:08.8702804Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2024-04-03T17:12:08.8704336Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future 2024-04-03T17:12:08.8706481Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2024-04-03T17:12:08.8708226Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler 2024-04-03T17:12:08.8710089Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2024-04-03T17:12:08.8711967Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2024-04-03T17:12:08.8713657Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort 2024-04-03T17:12:08.8715275Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2024-04-03T17:12:08.8716829Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2024-04-03T17:12:08.8718528Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2024-04-03T17:12:08.8720291Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2024-04-03T17:12:08.8721315Z 2024-04-03T17:12:08.8721700Z Running distributed tests for the ucc backend with file init_method 2024-04-03T17:12:08.8724150Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:12:08.860722] 2024-04-03T17:15:51.0014894Z 2024-04-03T17:15:51.0019915Z distributed/test_distributed_spawn 4/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.7_30ec712cbfee1a74_.log 2024-04-03T17:15:51.0051893Z Running 41 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_default_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2024-04-03T17:15:51.0080709Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2024-04-03T17:15:51.0082644Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2024-04-03T17:15:51.0084704Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2024-04-03T17:15:51.0086662Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2024-04-03T17:15:51.0088465Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2024-04-03T17:15:51.0090273Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2024-04-03T17:15:51.0092097Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2024-04-03T17:15:51.0093899Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2024-04-03T17:15:51.0095777Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2024-04-03T17:15:51.0097457Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2024-04-03T17:15:51.0099066Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2024-04-03T17:15:51.0100657Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2024-04-03T17:15:51.0102338Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group 2024-04-03T17:15:51.0104088Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split 2024-04-03T17:15:51.0106344Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda 2024-04-03T17:15:51.0108298Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2024-04-03T17:15:51.0110234Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2024-04-03T17:15:51.0112352Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda 2024-04-03T17:15:51.0114130Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group 2024-04-03T17:15:51.0115725Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2024-04-03T17:15:51.0117388Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2024-04-03T17:15:51.0119248Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group 2024-04-03T17:15:51.0120937Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2024-04-03T17:15:51.0122923Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2024-04-03T17:15:51.0124609Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2024-04-03T17:15:51.0126234Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2024-04-03T17:15:51.0127983Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2024-04-03T17:15:51.0129846Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2024-04-03T17:15:51.0131849Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_default_group 2024-04-03T17:15:51.0133626Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2024-04-03T17:15:51.0135194Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2024-04-03T17:15:51.0136726Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future 2024-04-03T17:15:51.0138333Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2024-04-03T17:15:51.0140025Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler 2024-04-03T17:15:51.0142055Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2024-04-03T17:15:51.0143929Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2024-04-03T17:15:51.0145906Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort 2024-04-03T17:15:51.0147627Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2024-04-03T17:15:51.0149193Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2024-04-03T17:15:51.0150769Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2024-04-03T17:15:51.0152508Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2024-04-03T17:15:51.0153528Z 2024-04-03T17:15:51.0154063Z Running distributed/test_distributed_spawn 7/7 ... [2024-04-03 17:15:51.004243] 2024-04-03T17:15:51.0154814Z /usr/bin/mpiexec 2024-04-03T17:15:51.0155379Z MPI not available -- MPI backend tests will be skipped 2024-04-03T17:15:51.0156186Z Running distributed tests for the test backend with env init_method 2024-04-03T17:15:51.0158748Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:15:51.006249] 2024-04-03T17:15:54.5915550Z 2024-04-03T17:15:54.5919692Z distributed/test_distributed_spawn 7/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.7_b3347970d7198181_.log 2024-04-03T17:15:54.5922755Z Running 0 items in this shard: 2024-04-03T17:15:54.5930323Z 2024-04-03T17:15:54.5931042Z Running distributed tests for the test backend with file init_method 2024-04-03T17:15:54.5940699Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:15:54.593525] 2024-04-03T17:15:58.1749457Z 2024-04-03T17:15:58.1751170Z distributed/test_distributed_spawn 7/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.7_2f45b07ac9dec14e_.log 2024-04-03T17:15:58.1752681Z Running 0 items in this shard: 2024-04-03T17:15:58.1753011Z 2024-04-03T17:15:58.1759846Z Running distributed tests for the nccl backend with env init_method 2024-04-03T17:15:58.1767000Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:15:58.176291] 2024-04-03T17:19:50.1465873Z 2024-04-03T17:19:50.1472094Z distributed/test_distributed_spawn 7/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.7_b401ac7049deb526_.log 2024-04-03T17:19:50.1499032Z Running 34 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-04-03T17:19:50.1523883Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2024-04-03T17:19:50.1525844Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2024-04-03T17:19:50.1527700Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync 2024-04-03T17:19:50.1529589Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2024-04-03T17:19:50.1531585Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2024-04-03T17:19:50.1533610Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2024-04-03T17:19:50.1535318Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2024-04-03T17:19:50.1537067Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2024-04-03T17:19:50.1538914Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2024-04-03T17:19:50.1540729Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2024-04-03T17:19:50.1542491Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2024-04-03T17:19:50.1544194Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group 2024-04-03T17:19:50.1546622Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2024-04-03T17:19:50.1548337Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2024-04-03T17:19:50.1550413Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2024-04-03T17:19:50.1552364Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2024-04-03T17:19:50.1554117Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2024-04-03T17:19:50.1555977Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2024-04-03T17:19:50.1557735Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2024-04-03T17:19:50.1559438Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2024-04-03T17:19:50.1561121Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2024-04-03T17:19:50.1562787Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params 2024-04-03T17:19:50.1564486Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2024-04-03T17:19:50.1566255Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2024-04-03T17:19:50.1568134Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2024-04-03T17:19:50.1569947Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2024-04-03T17:19:50.1571722Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2024-04-03T17:19:50.1573629Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2024-04-03T17:19:50.1575575Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2024-04-03T17:19:50.1577524Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2024-04-03T17:19:50.1579576Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product 2024-04-03T17:19:50.1581299Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2024-04-03T17:19:50.1582995Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2024-04-03T17:19:50.1584795Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-04-03T17:19:50.1586290Z 2024-04-03T17:19:50.1586665Z Running distributed tests for the nccl backend with file init_method 2024-04-03T17:19:50.1589126Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:19:50.149155] 2024-04-03T17:23:42.5522578Z 2024-04-03T17:23:42.5526195Z distributed/test_distributed_spawn 7/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.7_0afed1f42cf4ccc1_.log 2024-04-03T17:23:42.5551062Z Running 34 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-04-03T17:23:42.5601896Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2024-04-03T17:23:42.5605983Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2024-04-03T17:23:42.5609838Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync 2024-04-03T17:23:42.5614013Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2024-04-03T17:23:42.5618153Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2024-04-03T17:23:42.5621945Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2024-04-03T17:23:42.5625424Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2024-04-03T17:23:42.5629469Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2024-04-03T17:23:42.5633266Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2024-04-03T17:23:42.5637048Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2024-04-03T17:23:42.5640757Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2024-04-03T17:23:42.5644337Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group 2024-04-03T17:23:42.5647806Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2024-04-03T17:23:42.5651315Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2024-04-03T17:23:42.5655463Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2024-04-03T17:23:42.5659418Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2024-04-03T17:23:42.5663073Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2024-04-03T17:23:42.5667347Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2024-04-03T17:23:42.5670974Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2024-04-03T17:23:42.5674859Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2024-04-03T17:23:42.5678325Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2024-04-03T17:23:42.5681651Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params 2024-04-03T17:23:42.5685186Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2024-04-03T17:23:42.5688885Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2024-04-03T17:23:42.5692748Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2024-04-03T17:23:42.5696645Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2024-04-03T17:23:42.5700408Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2024-04-03T17:23:42.5704405Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2024-04-03T17:23:42.5709120Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2024-04-03T17:23:42.5713267Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2024-04-03T17:23:42.5717235Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product 2024-04-03T17:23:42.5720790Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2024-04-03T17:23:42.5724276Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2024-04-03T17:23:42.5727929Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-04-03T17:23:42.5730183Z 2024-04-03T17:23:42.5730936Z Running distributed tests for the gloo backend with env init_method 2024-04-03T17:23:42.5735830Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:23:42.554902] 2024-04-03T17:27:52.1200988Z 2024-04-03T17:27:52.1204520Z distributed/test_distributed_spawn 7/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.7_099c77c62c01a5cc_.log 2024-04-03T17:27:52.1233004Z Running 34 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-04-03T17:27:52.1257411Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2024-04-03T17:27:52.1259456Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2024-04-03T17:27:52.1261282Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync 2024-04-03T17:27:52.1263180Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2024-04-03T17:27:52.1265156Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2024-04-03T17:27:52.1267579Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2024-04-03T17:27:52.1269282Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2024-04-03T17:27:52.1271038Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2024-04-03T17:27:52.1273061Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2024-04-03T17:27:52.1274878Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2024-04-03T17:27:52.1276636Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2024-04-03T17:27:52.1278353Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group 2024-04-03T17:27:52.1280016Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2024-04-03T17:27:52.1281718Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2024-04-03T17:27:52.1283653Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2024-04-03T17:27:52.1285580Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2024-04-03T17:27:52.1287325Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2024-04-03T17:27:52.1289300Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2024-04-03T17:27:52.1291068Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2024-04-03T17:27:52.1292773Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2024-04-03T17:27:52.1294473Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2024-04-03T17:27:52.1309338Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params 2024-04-03T17:27:52.1311085Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2024-04-03T17:27:52.1312880Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2024-04-03T17:27:52.1314755Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2024-04-03T17:27:52.1316596Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2024-04-03T17:27:52.1318363Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2024-04-03T17:27:52.1320280Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2024-04-03T17:27:52.1322222Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2024-04-03T17:27:52.1324195Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2024-04-03T17:27:52.1326095Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product 2024-04-03T17:27:52.1327815Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2024-04-03T17:27:52.1329517Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2024-04-03T17:27:52.1331552Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-04-03T17:27:52.1332618Z 2024-04-03T17:27:52.1332983Z Running distributed tests for the gloo backend with file init_method 2024-04-03T17:27:52.1335460Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:27:52.122682] 2024-04-03T17:32:00.8114387Z 2024-04-03T17:32:00.8116274Z distributed/test_distributed_spawn 7/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.7_369b067f6ab52d86_.log 2024-04-03T17:32:00.8142675Z Running 34 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-04-03T17:32:00.8167597Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2024-04-03T17:32:00.8169566Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2024-04-03T17:32:00.8171452Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync 2024-04-03T17:32:00.8173348Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2024-04-03T17:32:00.8175339Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2024-04-03T17:32:00.8177349Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2024-04-03T17:32:00.8179063Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2024-04-03T17:32:00.8180824Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2024-04-03T17:32:00.8182668Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2024-04-03T17:32:00.8184488Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2024-04-03T17:32:00.8186844Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2024-04-03T17:32:00.8188566Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group 2024-04-03T17:32:00.8190237Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2024-04-03T17:32:00.8191941Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2024-04-03T17:32:00.8193871Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2024-04-03T17:32:00.8195809Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2024-04-03T17:32:00.8197564Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2024-04-03T17:32:00.8199421Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2024-04-03T17:32:00.8201175Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2024-04-03T17:32:00.8202876Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2024-04-03T17:32:00.8204558Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2024-04-03T17:32:00.8206217Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params 2024-04-03T17:32:00.8208091Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2024-04-03T17:32:00.8209861Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2024-04-03T17:32:00.8211732Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2024-04-03T17:32:00.8213569Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2024-04-03T17:32:00.8215330Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2024-04-03T17:32:00.8217231Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2024-04-03T17:32:00.8219182Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2024-04-03T17:32:00.8221142Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2024-04-03T17:32:00.8223151Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product 2024-04-03T17:32:00.8224895Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2024-04-03T17:32:00.8227152Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2024-04-03T17:32:00.8228951Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-04-03T17:32:00.8230041Z 2024-04-03T17:32:00.8230395Z Running distributed tests for the ucc backend with env init_method 2024-04-03T17:32:00.8232853Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:32:00.813818] 2024-04-03T17:35:35.5059582Z 2024-04-03T17:35:35.5061392Z distributed/test_distributed_spawn 7/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.7_b4a0fdb855a0c2d4_.log 2024-04-03T17:35:35.5092177Z Running 34 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-04-03T17:35:35.5121147Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2024-04-03T17:35:35.5123381Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2024-04-03T17:35:35.5125479Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync 2024-04-03T17:35:35.5127365Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2024-04-03T17:35:35.5129587Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2024-04-03T17:35:35.5131448Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2024-04-03T17:35:35.5133136Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2024-04-03T17:35:35.5134900Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2024-04-03T17:35:35.5136982Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2024-04-03T17:35:35.5138804Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2024-04-03T17:35:35.5140754Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2024-04-03T17:35:35.5142467Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group 2024-04-03T17:35:35.5144132Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2024-04-03T17:35:35.5146244Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2024-04-03T17:35:35.5148195Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2024-04-03T17:35:35.5150384Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2024-04-03T17:35:35.5152149Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2024-04-03T17:35:35.5154016Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2024-04-03T17:35:35.5155758Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2024-04-03T17:35:35.5157603Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2024-04-03T17:35:35.5159318Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2024-04-03T17:35:35.5160976Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params 2024-04-03T17:35:35.5162671Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2024-04-03T17:35:35.5164451Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2024-04-03T17:35:35.5166323Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2024-04-03T17:35:35.5168158Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2024-04-03T17:35:35.5169926Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2024-04-03T17:35:35.5171839Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2024-04-03T17:35:35.5173779Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2024-04-03T17:35:35.5176000Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2024-04-03T17:35:35.5177909Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product 2024-04-03T17:35:35.5179639Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2024-04-03T17:35:35.5181343Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2024-04-03T17:35:35.5183183Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-04-03T17:35:35.5184266Z 2024-04-03T17:35:35.5184623Z Running distributed tests for the ucc backend with file init_method 2024-04-03T17:35:35.5187586Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:35:35.508476] 2024-04-03T17:39:09.7978482Z 2024-04-03T17:39:09.7980324Z distributed/test_distributed_spawn 7/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.7_8f3b0333eaf97b10_.log 2024-04-03T17:39:09.8007823Z Running 34 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-04-03T17:39:09.8032690Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2024-04-03T17:39:09.8034658Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2024-04-03T17:39:09.8036542Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync 2024-04-03T17:39:09.8038430Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2024-04-03T17:39:09.8040372Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2024-04-03T17:39:09.8042243Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2024-04-03T17:39:09.8043958Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2024-04-03T17:39:09.8045718Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2024-04-03T17:39:09.8047692Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2024-04-03T17:39:09.8049532Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2024-04-03T17:39:09.8051312Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2024-04-03T17:39:09.8053032Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group 2024-04-03T17:39:09.8054679Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2024-04-03T17:39:09.8056386Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2024-04-03T17:39:09.8058344Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2024-04-03T17:39:09.8060277Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2024-04-03T17:39:09.8062018Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2024-04-03T17:39:09.8063898Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2024-04-03T17:39:09.8066035Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2024-04-03T17:39:09.8067742Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2024-04-03T17:39:09.8069434Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2024-04-03T17:39:09.8071106Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params 2024-04-03T17:39:09.8072830Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2024-04-03T17:39:09.8074582Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2024-04-03T17:39:09.8076603Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2024-04-03T17:39:09.8078442Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2024-04-03T17:39:09.8080228Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2024-04-03T17:39:09.8082130Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2024-04-03T17:39:09.8084074Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2024-04-03T17:39:09.8086045Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2024-04-03T17:39:09.8087977Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product 2024-04-03T17:39:09.8089692Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2024-04-03T17:39:09.8091397Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2024-04-03T17:39:09.8093320Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-04-03T17:39:09.8094397Z 2024-04-03T17:39:09.8095052Z Running distributed/optim/test_named_optimizer 1/1 ... [2024-04-03 17:39:09.800191] 2024-04-03T17:39:09.8097489Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_named_optimizer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:39:09.800592] 2024-04-03T17:39:13.0298375Z 2024-04-03T17:39:13.0300398Z distributed/optim/test_named_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_named_optimizer_1.1_1774d2ff12840844_.log 2024-04-03T17:39:13.0301818Z 2024-04-03T17:39:13.0303987Z Running distributed/_composable/test_compose 1/1 ... [2024-04-03 17:39:13.030065] 2024-04-03T17:39:13.0309420Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_compose.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:39:13.030480] 2024-04-03T17:39:54.7800514Z 2024-04-03T17:39:54.7803868Z distributed/_composable/test_compose 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_compose_1.1_72ec906ad1796bec_.log 2024-04-03T17:39:54.7811338Z Running 9 items in this shard: test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_checkpoint_fsdp_submodules_non_reentrant, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_checkpoint_fsdp_submodules_with_param, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_checkpoint_fsdp_submodules_with_param_no_shard, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_checkpoint_fully_shard_cast_forward_inputs, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_composable_fsdp_replicate, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_fully_shard_replicate_composability, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_fully_shard_replicate_correct_replicate_params, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_state_dict_fsdp_submodules, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_wrap_same_submodule 2024-04-03T17:39:54.7818080Z 2024-04-03T17:39:54.7818749Z Running distributed/_tensor/debug/test_comm_mode 1/1 ... [2024-04-03 17:39:54.780361] 2024-04-03T17:39:54.7821147Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/debug/test_comm_mode.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:39:54.780772] 2024-04-03T17:39:58.5024367Z 2024-04-03T17:39:58.5026560Z distributed/_tensor/debug/test_comm_mode 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_comm_mode_1.1_4b085a7970ce12c0_.log 2024-04-03T17:39:58.5029237Z Running 2 items in this shard: test/distributed/_tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode, test/distributed/_tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode_with_dtensor 2024-04-03T17:39:58.5030688Z 2024-04-03T17:39:58.5031302Z Running distributed/tensor/parallel/test_tp_examples 1/1 ... [2024-04-03 17:39:58.502733] 2024-04-03T17:39:58.5036412Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_examples.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:39:58.503167] 2024-04-03T17:40:33.3412048Z 2024-04-03T17:40:33.3416214Z distributed/tensor/parallel/test_tp_examples 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_examples_1.1_f98e22abf68fffc5_.log 2024-04-03T17:40:33.3423716Z Running 7 items in this shard: test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_loss_parallel, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_inference, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_False_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_True_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_True, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_weight_tying 2024-04-03T17:40:33.3430377Z 2024-04-03T17:40:33.3430941Z Running distributed/test_inductor_collectives 1/1 ... [2024-04-03 17:40:33.341497] 2024-04-03T17:40:33.3433381Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_inductor_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:40:33.341903] 2024-04-03T17:40:44.0279067Z 2024-04-03T17:40:44.0281164Z distributed/test_inductor_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_inductor_collectives_1.1_954504a6275e4c63_.log 2024-04-03T17:40:44.0344145Z Running 75 items in this shard: test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_input_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_output_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_contiguous_input, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_into_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_output_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_input_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_broadcast_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_c10d_functional_tagged_pt2_compliant, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_inductor_allreduce_eager_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_permute_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_reduce_scatter_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_backwards_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_backwards_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_use_native_funcol_False_source_GroupMember_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_use_native_funcol_False_source__get_default_group, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_use_native_funcol_False_source_group_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_use_native_funcol_True_source_GroupMember_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_use_native_funcol_True_source__get_default_group, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_use_native_funcol_True_source_group_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_graphbreaks_unsupported_async_op_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_graphbreaks_unsupported_async_op_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_pg_var_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_pg_var_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_args_match_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_args_match_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_list_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_list_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_to_all_single_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_to_all_single_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_False_reduce_op0, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_False_reduce_op1, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_False_reduce_op2, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_False_reduce_op3, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_False_reduce_op4, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_True_reduce_op0, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_True_reduce_op1, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_True_reduce_op2, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_True_reduce_op3, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_True_reduce_op4, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_False_pg_mode_kwargs, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_False_pg_mode_kwargs_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_False_pg_mode_positional, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_False_pg_mode_positional_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_False_pg_mode_unspecified, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_True_pg_mode_kwargs, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_True_pg_mode_kwargs_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_True_pg_mode_positional, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_True_pg_mode_positional_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_True_pg_mode_unspecified, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_reduce_scatter_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_reduce_scatter_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_support_collective_op_with_async_op_False_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_support_collective_op_with_async_op_False_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_pg_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_pg_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allgather_coalesced_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allgather_coalesced_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allreduce_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allreduce_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_reduce_scatter_tensor_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_reduce_scatter_tensor_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_all_gather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_doesnt_mutate_shared, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_reduce_scatter_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_single_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_steal_buffer, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_meta 2024-04-03T17:40:44.0406011Z 2024-04-03T17:40:44.0406633Z Running distributed/fsdp/test_fsdp_backward_prefetch 1/1 ... [2024-04-03 17:40:44.028267] 2024-04-03T17:40:44.0409186Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_backward_prefetch.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:40:44.028680] 2024-04-03T17:40:53.6624414Z 2024-04-03T17:40:53.6626819Z distributed/fsdp/test_fsdp_backward_prefetch 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_backward_prefetch_1.1_c01256328cc8beb8_.log 2024-04-03T17:40:53.6629164Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_backward_prefetch.py::TestBackwardPrefetch::test_backward_prefetch 2024-04-03T17:40:53.6630148Z 2024-04-03T17:40:53.6630693Z Running distributed/_composable/test_replicate 1/1 ... [2024-04-03 17:40:53.662731] 2024-04-03T17:40:53.6635693Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_replicate.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:40:53.663149] 2024-04-03T17:41:27.6480363Z 2024-04-03T17:41:27.6484177Z distributed/_composable/test_replicate 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_replicate_1.1_4edbda9424142652_.log 2024-04-03T17:41:27.6491228Z Running 9 items in this shard: test/distributed/_composable/test_replicate.py::ReplicateStateDictTest::test_replicate_non_root_multiple_save_load, test/distributed/_composable/test_replicate.py::ReplicateStateDictTest::test_replicate_single_module_save_load, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_device_id, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_ignore_module, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_move_args_kwargs_to_device, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_multi_module, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_single_module, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_with_kwargs, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_wrong_device_id_type 2024-04-03T17:41:27.6497059Z 2024-04-03T17:41:27.6497701Z Running distributed/_shard/sharding_plan/test_sharding_plan 1/1 ... [2024-04-03 17:41:27.648312] 2024-04-03T17:41:27.6500329Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:41:27.648715] 2024-04-03T17:41:41.2925858Z 2024-04-03T17:41:41.2928164Z distributed/_shard/sharding_plan/test_sharding_plan 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharding_plan.test_sharding_plan_1.1_5c3f2919ac34196c_.log 2024-04-03T17:41:41.2931946Z Running 3 items in this shard: test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_custom_sharding_planner, test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_shard_module_sub_process_group, test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_sharding_plan_errors 2024-04-03T17:41:41.2934376Z 2024-04-03T17:41:41.2935090Z Running distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 ... [2024-04-03 17:41:41.292842] 2024-04-03T17:41:41.2937819Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:41:41.293308] 2024-04-03T17:41:55.5850860Z 2024-04-03T17:41:55.5853302Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_clip_grad_norm__1.1_42c874edff863625_.log 2024-04-03T17:41:55.5857449Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py::TestClipGradNormWorldSize2::test_clip_grad_norm_1d, test/distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py::TestClipGradNormWorldSize4::test_clip_grad_norm_2d 2024-04-03T17:41:55.5859382Z 2024-04-03T17:41:55.5860095Z Running distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 ... [2024-04-03 17:41:55.585431] 2024-04-03T17:41:55.5863294Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_tp_checkpoint_conversion.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:41:55.585913] 2024-04-03T17:42:03.7666835Z 2024-04-03T17:42:03.7669433Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion_1.1_e1832f98755cfebd_.log 2024-04-03T17:42:03.7672349Z Running 1 items in this shard: test/distributed/checkpoint/test_fsdp_tp_checkpoint_conversion.py::TestFsdpTpCheckpointConversion::test_fsdp_to_tp 2024-04-03T17:42:03.7673601Z 2024-04-03T17:42:03.7674144Z Running distributed/_tensor/test_optimizers 1/1 ... [2024-04-03 17:42:03.766966] 2024-04-03T17:42:03.7678455Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_optimizers.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:42:03.767373] 2024-04-03T17:42:55.6895922Z 2024-04-03T17:42:55.6899970Z distributed/_tensor/test_optimizers 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_optimizers_1.1_7b8a4ae776f35b9c_.log 2024-04-03T17:42:55.6907423Z Running 10 items in this shard: test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_RMSprop_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adadelta_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adagrad_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adam_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adamax_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adamw_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_asgd_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_nadam_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_radam_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_sgd_1d_sharding 2024-04-03T17:42:55.6913802Z 2024-04-03T17:42:55.6914486Z Running distributed/_composable/fsdp/test_fully_shard_frozen 1/1 ... [2024-04-03 17:42:55.689812] 2024-04-03T17:42:55.6917141Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_frozen.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:42:55.690226] 2024-04-03T17:43:35.6813094Z 2024-04-03T17:43:35.6817289Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_edf706a973aa33a5_.log 2024-04-03T17:43:35.6821625Z Running 3 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_multi_forward_mixed_requires_grad, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_across_groups, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_per_group 2024-04-03T17:43:35.6824575Z 2024-04-03T17:43:35.6825217Z Running distributed/fsdp/test_fsdp_flatten_params 1/1 ... [2024-04-03 17:43:35.681578] 2024-04-03T17:43:35.6828022Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_flatten_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:43:35.682020] 2024-04-03T17:44:22.5438170Z 2024-04-03T17:44:22.5441915Z distributed/fsdp/test_fsdp_flatten_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_flatten_params_1.1_08891095b96f30b5_.log 2024-04-03T17:44:22.5451583Z Running 11 items in this shard: test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_empty_module, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_aligned_full_precision, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_aligned_mixed_precision, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_unaligned, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flatten_nothing, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_numel_with_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_numel_without_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_output_with_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_output_without_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_partial_flattening, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_pnorm_after_step_with_shared_params 2024-04-03T17:44:22.5459446Z 2024-04-03T17:44:22.5459962Z Running distributed/test_functional_api 1/1 ... [2024-04-03 17:44:22.544117] 2024-04-03T17:44:22.5462350Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_functional_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:44:22.544522] 2024-04-03T17:46:22.2127214Z 2024-04-03T17:46:22.2130899Z distributed/test_functional_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_functional_api_1.1_ca0301fccc2e1f63_.log 2024-04-03T17:46:22.2239814Z Running 77 items in this shard: test/distributed/test_functional_api.py::TestExpand::test_expand_1d_rank_list, test/distributed/test_functional_api.py::TestExpand::test_expand_2d_rank_list, test/distributed/test_functional_api.py::TestExpand::test_expand_device_mesh, test/distributed/test_functional_api.py::TestExpand::test_expand_device_mesh_tuple, test/distributed/test_functional_api.py::TestExpand::test_expand_process_group, test/distributed/test_functional_api.py::TestPgTag::test_find_or_create_pg, test/distributed/test_functional_api.py::TestPgTag::test_find_root_pg, test/distributed/test_functional_api.py::TestPgTag::test_pg_creation_with_tag, test/distributed/test_functional_api.py::TestPgTag::test_pg_lookup_roundtrip, test/distributed/test_functional_api.py::TestPgTag::test_pg_lookup_with_tag, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_into_tensor_coalesced_device_cpu_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_into_tensor_coalesced_device_cpu_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_into_tensor_coalesced_device_cuda_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_into_tensor_coalesced_device_cuda_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_tensor_device_cpu_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_tensor_device_cpu_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_tensor_device_cuda_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_tensor_device_cuda_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_coalesced_eager_device_cpu_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_coalesced_eager_device_cpu_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_coalesced_eager_device_cuda_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_coalesced_eager_device_cuda_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_eager_device_cpu_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_eager_device_cpu_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_eager_device_cuda_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_eager_device_cuda_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_broadcast_device_cpu_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_broadcast_device_cpu_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_broadcast_device_cuda_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_broadcast_device_cuda_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_into_tensor_coalesced_device_cpu_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_into_tensor_coalesced_device_cpu_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_into_tensor_coalesced_device_cuda_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_into_tensor_coalesced_device_cuda_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_tensor_device_cpu_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_tensor_device_cpu_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_tensor_device_cuda_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_tensor_device_cuda_use_native_funcol_True, test/distributed/test_functional_api.py::TestMetaCollectives::test_all_reduce_use_native_funcol_False, test/distributed/test_functional_api.py::TestMetaCollectives::test_all_reduce_use_native_funcol_True, test/distributed/test_functional_api.py::TestGradCollectives::test_all_reduce_use_native_funcol_False, test/distributed/test_functional_api.py::TestGradCollectives::test_all_reduce_use_native_funcol_True, test/distributed/test_functional_api.py::TestMakeFx::test_all_reduce_tracing_use_native_funcol_False, test/distributed/test_functional_api.py::TestMakeFx::test_all_reduce_tracing_use_native_funcol_True, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_gather_into_tensor_coalesced_use_native_funcol_False, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_gather_into_tensor_coalesced_use_native_funcol_True, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_to_all_single_1d_input_use_native_funcol_False, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_to_all_single_1d_input_use_native_funcol_True, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_to_all_single_input_split_sizes_none, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_to_all_single_output_split_sizes_none, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_to_all_single_split_sizes_none, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_to_all_single_use_native_funcol_False, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_to_all_single_use_native_funcol_True, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_tracing_use_native_funcol_False, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_tracing_use_native_funcol_True, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_tracing_with_fakepg_use_native_funcol_False, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_tracing_with_fakepg_use_native_funcol_True, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_gather_into_tensor_coalesced_use_native_funcol_False, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_gather_into_tensor_coalesced_use_native_funcol_True, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_to_all_single_1d_input_use_native_funcol_False, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_to_all_single_1d_input_use_native_funcol_True, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_to_all_single_input_split_sizes_none, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_to_all_single_output_split_sizes_none, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_to_all_single_split_sizes_none, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_to_all_single_use_native_funcol_False, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_to_all_single_use_native_funcol_True, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_permute_tensor_with_sub_group_use_native_funcol_False, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_permute_tensor_with_sub_group_use_native_funcol_True, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_tracing_use_native_funcol_False, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_tracing_use_native_funcol_True, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_tracing_with_fakepg_use_native_funcol_False, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_tracing_with_fakepg_use_native_funcol_True, test/distributed/test_functional_api.py::TestOpWaitiness::test_add_triggers_wait, test/distributed/test_functional_api.py::TestOpWaitiness::test_dead_wrapper_plus_view, test/distributed/test_functional_api.py::TestOpWaitiness::test_dead_wrapper_triggers_wait, test/distributed/test_functional_api.py::TestOpWaitiness::test_view_does_not_trigger_wait, test/distributed/test_functional_api.py::TestOpWaitiness::test_wait_reduce_outstanding_work_count 2024-04-03T17:46:22.2353595Z 2024-04-03T17:46:22.2354842Z Running distributed/fsdp/test_fsdp_ignored_modules 1/1 ... [2024-04-03 17:46:22.213186] 2024-04-03T17:46:22.2360104Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_ignored_modules.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:46:22.213629] 2024-04-03T17:47:22.2534556Z 2024-04-03T17:47:22.2537013Z distributed/fsdp/test_fsdp_ignored_modules 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_ignored_modules_1.1_cc8d154f55e6b3db_.log 2024-04-03T17:47:22.2549861Z Running 13 items in this shard: test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_diff_ignored_modules_across_ranks, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_invalid_composable_False, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_invalid_composable_True, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_nested, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_nested_composable, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_False_composable_False, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_False_composable_True, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_True_composable_False, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_True_composable_True, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_transformer, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_transformer_composable, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_states_auto_wrap, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_states_check 2024-04-03T17:47:22.2560862Z 2024-04-03T17:47:22.2561447Z Running distributed/checkpoint/test_checkpoint 1/1 ... [2024-04-03 17:47:22.253821] 2024-04-03T17:47:22.2563945Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:47:22.254258] 2024-04-03T17:47:54.5881219Z 2024-04-03T17:47:54.5885296Z distributed/checkpoint/test_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_checkpoint_1.1_763d4cf5bbe10aa5_.log 2024-04-03T17:47:54.5892659Z Running 8 items in this shard: test/distributed/checkpoint/test_checkpoint.py::TestDistributedCheckpointing::test_default_metadata, test/distributed/checkpoint/test_checkpoint.py::TestDistributedCheckpointing::test_tensor_metadata_with_missing_rank_spec, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_dummy_reader_works, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_dummy_writer_works, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_load_error_handling, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_load_error_handling_no_dist, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_save_error_handling, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_save_error_handling_no_dist 2024-04-03T17:47:54.5898357Z 2024-04-03T17:47:54.5898906Z Running distributed/rpc/test_share_memory 1/1 ... [2024-04-03 17:47:54.588334] 2024-04-03T17:47:54.5901389Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/rpc/test_share_memory.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:47:54.588754] 2024-04-03T17:48:03.1331107Z 2024-04-03T17:48:03.1332888Z distributed/rpc/test_share_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.test_share_memory_1.1_0bbc2379a34af0dd_.log 2024-04-03T17:48:03.1335193Z Running 1 items in this shard: test/distributed/rpc/test_share_memory.py::TestRPCPickler::test_case 2024-04-03T17:48:03.1336525Z Running 1 items in this shard: test/distributed/rpc/test_share_memory.py::TestRPCPickler::test_case 2024-04-03T17:48:03.1337311Z 2024-04-03T17:48:03.1337897Z Running distributed/fsdp/test_fsdp_freezing_weights 1/1 ... [2024-04-03 17:48:03.133279] 2024-04-03T17:48:03.1340940Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_freezing_weights.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:48:03.133681] 2024-04-03T17:50:29.9690409Z 2024-04-03T17:50:29.9692572Z distributed/fsdp/test_fsdp_freezing_weights 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_freezing_weights_1.1_94c0efac8066a46a_.log 2024-04-03T17:50:29.9740447Z Running 32 items in this shard: test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True 2024-04-03T17:50:29.9787116Z 2024-04-03T17:50:29.9787697Z Running distributed/fsdp/test_fsdp_traversal 1/1 ... [2024-04-03 17:50:29.969354] 2024-04-03T17:50:29.9790144Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:50:29.969819] 2024-04-03T17:50:37.8006252Z 2024-04-03T17:50:37.8007986Z distributed/fsdp/test_fsdp_traversal 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_traversal_1.1_907a08fff0ad5454_.log 2024-04-03T17:50:37.8010123Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_traversal.py::TestTraversal::test_fsdp_modules 2024-04-03T17:50:37.8010998Z 2024-04-03T17:50:37.8011521Z Running distributed/fsdp/test_fsdp_input 1/1 ... [2024-04-03 17:50:37.800840] 2024-04-03T17:50:37.8016793Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:50:37.801264] 2024-04-03T17:50:49.5382714Z 2024-04-03T17:50:49.5384704Z distributed/fsdp/test_fsdp_input 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_input_1.1_dd481dbae4d04fa8_.log 2024-04-03T17:50:49.5387909Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_input.py::TestInput::test_input_type_dict, test/distributed/fsdp/test_fsdp_input.py::TestInput::test_input_type_list 2024-04-03T17:50:49.5389235Z 2024-04-03T17:50:49.5389866Z Running distributed/fsdp/test_fsdp_uneven 1/1 ... [2024-04-03 17:50:49.538525] 2024-04-03T17:50:49.5394259Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:50:49.538986] 2024-04-03T17:50:57.3692839Z 2024-04-03T17:50:57.3694578Z distributed/fsdp/test_fsdp_uneven 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_uneven_1.1_91f5baa042f208ec_.log 2024-04-03T17:50:57.3696658Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShard::test_one_iteration 2024-04-03T17:50:57.3697801Z 2024-04-03T17:50:57.3699705Z Running distributed/fsdp/test_fsdp_multiple_wrapping 1/1 ... [2024-04-03 17:50:57.369640] 2024-04-03T17:50:57.3704834Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:50:57.370063] 2024-04-03T17:51:04.8500649Z 2024-04-03T17:51:04.8502561Z distributed/fsdp/test_fsdp_multiple_wrapping 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_multiple_wrapping_1.1_5c4c46e97cb340a3_.log 2024-04-03T17:51:04.8504876Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_multiple_wrapping.py::TestMultipleWrapping::test_multiple_wrapping 2024-04-03T17:51:04.8506598Z 2024-04-03T17:51:04.8507801Z Running distributed/_composable/fully_shard/test_fully_shard_runtime 1/1 ... [2024-04-03 17:51:04.850374] 2024-04-03T17:51:04.8513799Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_runtime.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:51:04.850852] 2024-04-03T17:51:19.2445451Z 2024-04-03T17:51:19.2447584Z distributed/_composable/fully_shard/test_fully_shard_runtime 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_runtime_1.1_531af965937050b9_.log 2024-04-03T17:51:19.2451052Z Running 2 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_runtime.py::TestRuntime::test_training, test/distributed/_composable/fully_shard/test_fully_shard_runtime.py::TestRuntime::test_unshard_reshard_order 2024-04-03T17:51:19.2452701Z 2024-04-03T17:51:19.2453286Z Running distributed/fsdp/test_fsdp_clip_grad_norm 1/1 ... [2024-04-03 17:51:19.244784] 2024-04-03T17:51:19.2456426Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:51:19.245207] 2024-04-03T17:51:55.5283903Z 2024-04-03T17:51:55.5286022Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_clip_grad_norm_1.1_8524e74f6a7070eb_.log 2024-04-03T17:51:55.5291718Z Running 4 items in this shard: test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNorm::test_ddp_parity, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNorm::test_low_precision_grads, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNorm::test_no_gradients, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNorm::test_non_root 2024-04-03T17:51:55.5294296Z 2024-04-03T17:51:55.5294947Z Running distributed/fsdp/test_fsdp_exec_order 1/1 ... [2024-04-03 17:51:55.529067] 2024-04-03T17:51:55.5298891Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_exec_order.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:51:55.529475] 2024-04-03T17:52:30.6672025Z 2024-04-03T17:52:30.6673877Z distributed/fsdp/test_fsdp_exec_order 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_exec_order_1.1_20e98eb01b817e9c_.log 2024-04-03T17:52:30.6684705Z Running 8 items in this shard: test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_invalid_first_iter_order_sharding_strategy0, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_invalid_first_iter_order_sharding_strategy1, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_invalid_later_iter_order_sharding_strategy0_iters_before_path_change_1, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_invalid_later_iter_order_sharding_strategy0_iters_before_path_change_3, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_invalid_later_iter_order_sharding_strategy1_iters_before_path_change_1, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_invalid_later_iter_order_sharding_strategy1_iters_before_path_change_3, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_train_eval_sharding_strategy0, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_train_eval_sharding_strategy1 2024-04-03T17:52:30.6691633Z 2024-04-03T17:52:30.6692230Z Running distributed/pipeline/sync/test_pipe 1/1 ... [2024-04-03 17:52:30.667624] 2024-04-03T17:52:30.6695020Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_pipe.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:52:30.668112] 2024-04-03T17:52:36.5945219Z 2024-04-03T17:52:36.5947376Z distributed/pipeline/sync/test_pipe 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_pipe_1.1_b65732e662869d46_.log 2024-04-03T17:52:36.5976375Z Running 56 items in this shard: test/distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc, test/distributed/pipeline/sync/test_pipe.py::test_parameters, test/distributed/pipeline/sync/test_pipe.py::test_public_attrs, test/distributed/pipeline/sync/test_pipe.py::test_sequential_like, test/distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_small, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input, test/distributed/pipeline/sync/test_pipe.py::test_no_grad, test/distributed/pipeline/sync/test_pipe.py::test_exception, test/distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap, test/distributed/pipeline/sync/test_pipe.py::test_nested_input, test/distributed/pipeline/sync/test_pipe.py::test_input_pair, test/distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input, test/distributed/pipeline/sync/test_pipe.py::test_input_singleton, test/distributed/pipeline/sync/test_pipe.py::test_input_varargs, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence, test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[never], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[always], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always], test/distributed/pipeline/sync/test_pipe.py::test_devices, test/distributed/pipeline/sync/test_pipe.py::test_partitions, test/distributed/pipeline/sync/test_pipe.py::test_merged_partitions, test/distributed/pipeline/sync/test_pipe.py::test_deny_moving, test/distributed/pipeline/sync/test_pipe.py::test_empty_module, test/distributed/pipeline/sync/test_pipe.py::test_named_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_forward_lockstep, test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last], test/distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device, test/distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper 2024-04-03T17:52:36.6004631Z 2024-04-03T17:52:36.6005163Z Running distributed/fsdp/test_fsdp_fine_tune 1/1 ... [2024-04-03 17:52:36.594825] 2024-04-03T17:52:36.6007604Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fine_tune.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:52:36.595249] 2024-04-03T17:53:07.9223585Z 2024-04-03T17:53:07.9225455Z distributed/fsdp/test_fsdp_fine_tune 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_fine_tune_1.1_21ef5b58070f5c39_.log 2024-04-03T17:53:07.9232320Z Running 4 items in this shard: test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTune::test_backward_reshard_hooks, test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTune::test_hooks_multi_traversal, test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTune::test_parity_with_ddp, test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTune::test_parity_with_non_frozen_fsdp 2024-04-03T17:53:07.9235684Z 2024-04-03T17:53:07.9236430Z Running distributed/fsdp/test_fsdp_core 1/2 ... [2024-04-03 17:53:07.922754] 2024-04-03T17:53:07.9239304Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '--shard-id=1', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:53:07.923179] 2024-04-03T18:00:10.7308523Z 2024-04-03T18:00:10.7312533Z distributed/fsdp/test_fsdp_core 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_core_1.2_1251799597747aa0_.log 2024-04-03T18:00:10.7329855Z Running 22 items in this shard: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParamInit::test_param_change_after_init_mixed_precision_False, test/distributed/fsdp/test_fsdp_core.py::TestParamInit::test_param_change_after_init_mixed_precision_True, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_pre_backward_hook_registration_cuda_first_False, test/distributed/fsdp/test_fsdp_core.py::TestNoGrad::test_transformer_no_grad_mixed_precision_False, test/distributed/fsdp/test_fsdp_core.py::TestNoGrad::test_transformer_no_grad_mixed_precision_True, test/distributed/fsdp/test_fsdp_core.py::TestAutograd::test_unshard_params_as_tensors 2024-04-03T18:00:10.7346985Z 2024-04-03T18:00:10.7347709Z Running distributed/algorithms/quantization/test_quantization 1/1 ... [2024-04-03 18:00:10.732561] 2024-04-03T18:00:10.7348594Z /usr/bin/mpiexec 2024-04-03T18:00:10.7349162Z MPI not available -- MPI backend tests will be skipped 2024-04-03T18:00:10.7350017Z Running distributed tests for the test backend with env init_method 2024-04-03T18:00:10.7352572Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:00:10.734428] 2024-04-03T18:00:13.9682784Z 2024-04-03T18:00:13.9684981Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_ebe94a5e0c5a3c86_.log 2024-04-03T18:00:13.9687715Z Running 0 items in this shard: 2024-04-03T18:00:13.9688462Z 2024-04-03T18:00:13.9696925Z Running distributed tests for the test backend with file init_method 2024-04-03T18:00:13.9706886Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:00:13.970067] 2024-04-03T18:00:17.0790695Z 2024-04-03T18:00:17.0793115Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_ccb00e2d19b0a174_.log 2024-04-03T18:00:17.0795336Z Running 0 items in this shard: 2024-04-03T18:00:17.0795682Z 2024-04-03T18:00:17.0804001Z Running distributed tests for the nccl backend with env init_method 2024-04-03T18:00:17.0811306Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:00:17.080675] 2024-04-03T18:00:57.5343226Z 2024-04-03T18:00:57.5345335Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_077cf16a76dd90ad_.log 2024-04-03T18:00:57.5352977Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-04-03T18:00:57.5359260Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16 2024-04-03T18:00:57.5361445Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16 2024-04-03T18:00:57.5363290Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16 2024-04-03T18:00:57.5365271Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16 2024-04-03T18:00:57.5367178Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16 2024-04-03T18:00:57.5369237Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-04-03T18:00:57.5370359Z 2024-04-03T18:00:57.5370739Z Running distributed tests for the nccl backend with file init_method 2024-04-03T18:00:57.5373444Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:00:57.535828] 2024-04-03T18:01:38.0901579Z 2024-04-03T18:01:38.0905925Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_37f8c5c30eb53ad9_.log 2024-04-03T18:01:38.0912218Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-04-03T18:01:38.0917823Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16 2024-04-03T18:01:38.0920095Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16 2024-04-03T18:01:38.0922014Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16 2024-04-03T18:01:38.0923938Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16 2024-04-03T18:01:38.0925886Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16 2024-04-03T18:01:38.0927892Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-04-03T18:01:38.0928996Z 2024-04-03T18:01:38.0929399Z Running distributed tests for the gloo backend with env init_method 2024-04-03T18:01:38.0932020Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:01:38.091849] 2024-04-03T18:02:07.4688571Z 2024-04-03T18:02:07.4693010Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_a824c422a6708f46_.log 2024-04-03T18:02:07.4699292Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-04-03T18:02:07.4704921Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16 2024-04-03T18:02:07.4707451Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16 2024-04-03T18:02:07.4709385Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16 2024-04-03T18:02:07.4711289Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16 2024-04-03T18:02:07.4713268Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16 2024-04-03T18:02:07.4715279Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-04-03T18:02:07.4716382Z 2024-04-03T18:02:07.4716775Z Running distributed tests for the gloo backend with file init_method 2024-04-03T18:02:07.4719399Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:02:07.470708] 2024-04-03T18:02:36.9782169Z 2024-04-03T18:02:36.9786565Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_af965abe4bb9486e_.log 2024-04-03T18:02:36.9793273Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-04-03T18:02:36.9798861Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16 2024-04-03T18:02:36.9800760Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16 2024-04-03T18:02:36.9802721Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16 2024-04-03T18:02:36.9804589Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16 2024-04-03T18:02:36.9806709Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16 2024-04-03T18:02:36.9808747Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-04-03T18:02:36.9809881Z 2024-04-03T18:02:36.9810230Z Running distributed tests for the ucc backend with env init_method 2024-04-03T18:02:36.9812821Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:02:36.979715] 2024-04-03T18:02:40.1195549Z 2024-04-03T18:02:40.1199471Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_bec76d76296ef90b_.log 2024-04-03T18:02:40.1203190Z Running 0 items in this shard: 2024-04-03T18:02:40.1203912Z 2024-04-03T18:02:40.1211134Z Running distributed tests for the ucc backend with file init_method 2024-04-03T18:02:40.1221263Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:02:40.121550] 2024-04-03T18:02:43.3034061Z 2024-04-03T18:02:43.3037854Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_f26cc8364d311ba1_.log 2024-04-03T18:02:43.3041593Z Running 0 items in this shard: 2024-04-03T18:02:43.3042245Z 2024-04-03T18:02:43.3054339Z Running distributed/fsdp/test_fsdp_checkpoint 1/1 ... [2024-04-03 18:02:43.305037] 2024-04-03T18:02:43.3060916Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:02:43.305502] 2024-04-03T18:04:01.0324787Z 2024-04-03T18:04:01.0329021Z distributed/fsdp/test_fsdp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_checkpoint_1.1_ed2fe2f341499fe6_.log 2024-04-03T18:04:01.0347808Z Running 17 items in this shard: test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpointSubmodule::test_checkpoint_submodule_use_reentrant_False 2024-04-03T18:04:01.0364607Z 2024-04-03T18:04:01.0365187Z Running distributed/pipeline/sync/test_worker 1/1 ... [2024-04-03 18:04:01.032737] 2024-04-03T18:04:01.0367683Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_worker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:04:01.033143] 2024-04-03T18:04:03.4532868Z 2024-04-03T18:04:03.4534765Z distributed/pipeline/sync/test_worker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_worker_1.1_d48c8e607e82038a_.log 2024-04-03T18:04:03.4541017Z Running 6 items in this shard: test/distributed/pipeline/sync/test_worker.py::test_compute_multithreading, test/distributed/pipeline/sync/test_worker.py::test_compute_success, test/distributed/pipeline/sync/test_worker.py::test_compute_exception, test/distributed/pipeline/sync/test_worker.py::test_grad_mode[True], test/distributed/pipeline/sync/test_worker.py::test_grad_mode[False], test/distributed/pipeline/sync/test_worker.py::test_worker_per_device 2024-04-03T18:04:03.4544852Z 2024-04-03T18:04:03.4546099Z Running distributed/pipeline/sync/test_transparency 1/1 ... [2024-04-03 18:04:03.453595] 2024-04-03T18:04:03.4548687Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_transparency.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:04:03.454039] 2024-04-03T18:04:06.2251761Z 2024-04-03T18:04:06.2253748Z distributed/pipeline/sync/test_transparency 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_transparency_1.1_7a7763c818d30bd8_.log 2024-04-03T18:04:06.2255899Z Running 1 items in this shard: test/distributed/pipeline/sync/test_transparency.py::test_simple_linears 2024-04-03T18:04:06.2256736Z 2024-04-03T18:04:06.2260188Z Running distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 1/1 ... [2024-04-03 18:04:06.225635] 2024-04-03T18:04:06.2266155Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:04:06.226136] 2024-04-03T18:04:18.7670615Z 2024-04-03T18:04:18.7673286Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_optim_checkpoint_1.1_2793910f1cf7594e_.log 2024-04-03T18:04:18.7677566Z Running 2 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint.py::TestOptimStateCheckpointing::test_optim_state_dict_save_load, test/distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint.py::TestOptimStateCheckpointing::test_optim_state_dict_submodule_fully_shard 2024-04-03T18:04:18.7680256Z 2024-04-03T18:04:18.7680970Z Running distributed/fsdp/test_fsdp_unshard_params 1/1 ... [2024-04-03 18:04:18.767344] 2024-04-03T18:04:18.7683467Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_unshard_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:04:18.767804] 2024-04-03T18:05:22.8115168Z 2024-04-03T18:05:22.8119070Z distributed/fsdp/test_fsdp_unshard_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_unshard_params_1.1_e901e7c4adde1487_.log 2024-04-03T18:05:22.8130776Z Running 14 items in this shard: test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_named_parameters_and_buffers, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_param_data, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_recurse, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_respects_reshard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_writeback, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_singleton_param_writeback, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_with_grads_core, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_with_grads_none_grads, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsNoShard::test_unshard_params_param_data_no_shard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsNoShard::test_unshard_params_writeback_no_shard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_offload_to_cpu_no_shard_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_rank0_only_with_writeback_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_unshard_params_from_backward_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_unshard_params_from_forward_raises 2024-04-03T18:05:22.8141248Z 2024-04-03T18:05:22.8141831Z Running distributed/fsdp/test_fsdp_hybrid_shard 1/1 ... [2024-04-03 18:05:22.811911] 2024-04-03T18:05:22.8144308Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_hybrid_shard.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:05:22.812320] 2024-04-03T18:06:11.0288627Z 2024-04-03T18:06:11.0292300Z distributed/fsdp/test_fsdp_hybrid_shard 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_hybrid_shard_1.1_fab3fb41a1eb25a1_.log 2024-04-03T18:06:11.0298190Z Running 6 items in this shard: test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_fsdp_hybrid_shard_basic_setup, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_fsdp_hybrid_shard_parity, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_hsdp_save_load_state_dict, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_hsdp_sync_module_state, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_invalid_pg_specification_raises, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_raises_manual_wrap_hybrid_shard_when_none_policy 2024-04-03T18:06:11.0302596Z 2024-04-03T18:06:11.0303172Z Running distributed/pipeline/sync/test_inplace 1/1 ... [2024-04-03 18:06:11.029198] 2024-04-03T18:06:11.0306243Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_inplace.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:06:11.029634] 2024-04-03T18:06:13.4995574Z 2024-04-03T18:06:13.4997633Z distributed/pipeline/sync/test_inplace 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_inplace_1.1_0233b43f411056a4_.log 2024-04-03T18:06:13.5000845Z Running 3 items in this shard: test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad 2024-04-03T18:06:13.5002867Z 2024-04-03T18:06:13.5003450Z Running distributed/rpc/cuda/test_tensorpipe_agent 1/2 ... [2024-04-03 18:06:13.500000] 2024-04-03T18:06:13.5008773Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '--shard-id=1', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:06:13.500449] 2024-04-03T18:14:15.8805797Z 2024-04-03T18:14:15.8807729Z distributed/rpc/cuda/test_tensorpipe_agent 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.cuda.test_tensorpipe_agent_1.2_5efa29acd448dda8_.log 2024-04-03T18:14:15.8855120Z Running 51 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRpcTest::test_profiler_remote_cuda, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_simple, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_to_cpu_continuation, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_to_cpu_continuation_gpu_root, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_invalid_devices, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_valid_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDdpComparisonTest::test_ddp_dist_autograd_local_vs_remote_gpu, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_never_find_unused, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_except_last, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_never, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_never_find_unused, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_async_execution_with_cuda_future, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_cuda_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_list_with_cuda_sparse_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_list_with_cuda_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_not_cuda, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_nested, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_default_to_non_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_5, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_6, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_5, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_6, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_8, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_non_default_to_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_to_cpu_non_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_in_options, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_min_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_remote, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_response_loop, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_one_to_many, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_return_to_gpu, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_return_to_gpu_self, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_wrong_worker_name, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_mismatch, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_devices_option_mismatch_reverse, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_dist_autograd_sync_streams, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_gradients_synchronizations 2024-04-03T18:14:15.8898936Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRpcTest::test_profiler_remote_cuda 2024-04-03T18:14:15.8900763Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_simple 2024-04-03T18:14:15.8902663Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_to_cpu_continuation 2024-04-03T18:14:15.8904717Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_to_cpu_continuation_gpu_root 2024-04-03T18:14:15.8907242Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_invalid_devices 2024-04-03T18:14:15.8909084Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_valid_device 2024-04-03T18:14:15.8911086Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDdpComparisonTest::test_ddp_dist_autograd_local_vs_remote_gpu 2024-04-03T18:14:15.8913231Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_never_find_unused 2024-04-03T18:14:15.8915212Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_except_last 2024-04-03T18:14:15.8917117Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_never 2024-04-03T18:14:15.8919051Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_never_find_unused 2024-04-03T18:14:15.8921137Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_async_execution_with_cuda_future 2024-04-03T18:14:15.8923339Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_cuda_tensor 2024-04-03T18:14:15.8925638Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_list_with_cuda_sparse_tensor 2024-04-03T18:14:15.8928017Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_list_with_cuda_tensor 2024-04-03T18:14:15.8930409Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_not_cuda 2024-04-03T18:14:15.8932488Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_nested 2024-04-03T18:14:15.8934479Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu 2024-04-03T18:14:15.8936492Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_default 2024-04-03T18:14:15.8938642Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_default_to_non_default 2024-04-03T18:14:15.8940797Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_1 2024-04-03T18:14:15.8942837Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_2 2024-04-03T18:14:15.8944894Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_3 2024-04-03T18:14:15.8947559Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_4 2024-04-03T18:14:15.8949650Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_5 2024-04-03T18:14:15.8951691Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_6 2024-04-03T18:14:15.8953783Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_2 2024-04-03T18:14:15.8955899Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_3 2024-04-03T18:14:15.8958011Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_4 2024-04-03T18:14:15.8960110Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_5 2024-04-03T18:14:15.8962221Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_6 2024-04-03T18:14:15.8964321Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_8 2024-04-03T18:14:15.8966494Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_non_default_to_default 2024-04-03T18:14:15.8968677Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_to_cpu_non_default 2024-04-03T18:14:15.8970794Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_in_options 2024-04-03T18:14:15.8972888Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_min_device 2024-04-03T18:14:15.8975093Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_remote 2024-04-03T18:14:15.8977464Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_response_loop 2024-04-03T18:14:15.8979642Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_one_to_many 2024-04-03T18:14:15.8981721Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_return_to_gpu 2024-04-03T18:14:15.8983851Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_return_to_gpu_self 2024-04-03T18:14:15.8986515Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_wrong_worker_name 2024-04-03T18:14:15.8988602Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_mismatch 2024-04-03T18:14:15.8990692Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_devices_option_mismatch_reverse 2024-04-03T18:14:15.8992890Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization1 2024-04-03T18:14:15.8995213Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization3 2024-04-03T18:14:15.8997413Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization1 2024-04-03T18:14:15.8999558Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization2 2024-04-03T18:14:15.9001707Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization3 2024-04-03T18:14:15.9003847Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_dist_autograd_sync_streams 2024-04-03T18:14:15.9006023Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_gradients_synchronizations 2024-04-03T18:14:15.9007223Z 2024-04-03T18:14:15.9007780Z Running distributed/rpc/test_faulty_agent 1/1 ... [2024-04-03 18:14:15.883144] 2024-04-03T18:14:15.9010253Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/rpc/test_faulty_agent.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:14:15.883566] 2024-04-03T18:14:19.6354243Z 2024-04-03T18:14:19.6356049Z distributed/rpc/test_faulty_agent 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.test_faulty_agent_1.1_9c6de7659bbb5721_.log 2024-04-03T18:14:19.6357562Z Running 0 items in this shard: 2024-04-03T18:14:19.6357914Z 2024-04-03T18:14:19.6359618Z Running distributed/test_pg_wrapper 1/1 ... [2024-04-03 18:14:19.635672] 2024-04-03T18:14:19.6364812Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_pg_wrapper.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:14:19.636084] 2024-04-03T18:16:17.2615954Z 2024-04-03T18:16:17.2618028Z distributed/test_pg_wrapper 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_pg_wrapper_1.1_f19af57a9c7dd20f_.log 2024-04-03T18:16:17.2631039Z Running 15 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_coalescing_manager_debug_mode_detail, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_hang, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_detail, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_off, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_hang, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode_off, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_debug_mode 2024-04-03T18:16:17.2643086Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_coalescing_manager_debug_mode_detail 2024-04-03T18:16:17.2644856Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_hang 2024-04-03T18:16:17.2646637Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_detail 2024-04-03T18:16:17.2648567Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_off 2024-04-03T18:16:17.2650396Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch 2024-04-03T18:16:17.2652170Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch_debug_mode 2024-04-03T18:16:17.2653896Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_hang 2024-04-03T18:16:17.2655575Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda 2024-04-03T18:16:17.2657435Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda_debug_mode 2024-04-03T18:16:17.2659339Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode 2024-04-03T18:16:17.2661254Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode_off 2024-04-03T18:16:17.2663007Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch 2024-04-03T18:16:17.2664787Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda 2024-04-03T18:16:17.2667133Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda_debug_mode 2024-04-03T18:16:17.2668991Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_debug_mode 2024-04-03T18:16:17.2670027Z 2024-04-03T18:16:17.2670904Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 ... [2024-04-03 18:16:17.262180] 2024-04-03T18:16:17.2673542Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:16:17.262828] 2024-04-03T18:16:37.5231130Z 2024-04-03T18:16:37.5235124Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_tensor_ops_1.1_90351e1710e031ff_.log 2024-04-03T18:16:37.5239842Z Running 5 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_clone, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_deep_copy, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_detach, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_inplace_copy, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_set_requires_grad 2024-04-03T18:16:37.5243193Z 2024-04-03T18:16:37.5243691Z Running distributed/test_c10d_spawn_ucc 1/1 ... [2024-04-03 18:16:37.523407] 2024-04-03T18:16:37.5246445Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_spawn_ucc.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:16:37.523825] 2024-04-03T18:17:23.0480248Z 2024-04-03T18:17:23.0481991Z distributed/test_c10d_spawn_ucc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_ucc_1.1_2d3b576963f0935b_.log 2024-04-03T18:17:23.0489156Z Running 6 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_gather, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all_single, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_allreduce, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_broadcast, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_reduce 2024-04-03T18:17:23.0493811Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_gather 2024-04-03T18:17:23.0495419Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all 2024-04-03T18:17:23.0497051Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all_single 2024-04-03T18:17:23.0498674Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_allreduce 2024-04-03T18:17:23.0500266Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_broadcast 2024-04-03T18:17:23.0501818Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_reduce 2024-04-03T18:17:23.0502701Z 2024-04-03T18:17:23.0503325Z Running distributed/pipeline/sync/test_balance 1/1 ... [2024-04-03 18:17:23.048405] 2024-04-03T18:17:23.0506278Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_balance.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:17:23.048821] 2024-04-03T18:17:30.8279864Z 2024-04-03T18:17:30.8281734Z distributed/pipeline/sync/test_balance 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_balance_1.1_aff550061fccb10c_.log 2024-04-03T18:17:30.8293556Z Running 18 items in this shard: test/distributed/pipeline/sync/test_balance.py::test_blockpartition, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale, test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu], test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda], test/distributed/pipeline/sync/test_balance.py::test_not_training, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple, test/distributed/pipeline/sync/test_balance.py::test_already_has_grad 2024-04-03T18:17:30.8303439Z 2024-04-03T18:17:30.8304347Z Running distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 ... [2024-04-03 18:17:30.828274] 2024-04-03T18:17:30.8307413Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:17:30.828697] 2024-04-03T18:17:41.2650974Z 2024-04-03T18:17:41.2653271Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_reshard_1.1_3b5d71927f2416c8_.log 2024-04-03T18:17:41.2656600Z Running 2 items in this shard: test/distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py::TestReshard::test_sharded_tensor_reshard, test/distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py::TestReshard::test_sharded_tensor_reshard_errors 2024-04-03T18:17:41.2658449Z 2024-04-03T18:17:41.2659047Z Running distributed/pipeline/sync/test_dependency 1/1 ... [2024-04-03 18:17:41.265334] 2024-04-03T18:17:41.2661846Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_dependency.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:17:41.265756] 2024-04-03T18:17:43.7853618Z 2024-04-03T18:17:43.7855745Z distributed/pipeline/sync/test_dependency 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_dependency_1.1_d04a3ab63e20334a_.log 2024-04-03T18:17:43.7860427Z Running 6 items in this shard: test/distributed/pipeline/sync/test_dependency.py::test_fork_join, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_leak, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad 2024-04-03T18:17:43.7863843Z 2024-04-03T18:17:43.7864420Z Running distributed/pipeline/sync/skip/test_api 1/1 ... [2024-04-03 18:17:43.785663] 2024-04-03T18:17:43.7867364Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:17:43.786088] 2024-04-03T18:17:46.1051951Z 2024-04-03T18:17:46.1054124Z distributed/pipeline/sync/skip/test_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_api_1.1_29082bd71ec9e6c8_.log 2024-04-03T18:17:46.1057744Z Running 3 items in this shard: test/distributed/pipeline/sync/skip/test_api.py::test_namespace_difference, test/distributed/pipeline/sync/skip/test_api.py::test_namespace_copy, test/distributed/pipeline/sync/skip/test_api.py::test_skippable_repr 2024-04-03T18:17:46.1059583Z 2024-04-03T18:17:46.1060312Z Running distributed/pipeline/sync/test_deferred_batch_norm 1/1 ... [2024-04-03 18:17:46.105499] 2024-04-03T18:17:46.1063573Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_deferred_batch_norm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:17:46.105942] 2024-04-03T18:17:49.9282589Z 2024-04-03T18:17:49.9285519Z distributed/pipeline/sync/test_deferred_batch_norm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_deferred_batch_norm_1.1_3cfb9c566e84850b_.log 2024-04-03T18:17:49.9295737Z Running 11 items in this shard: test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad 2024-04-03T18:17:49.9302234Z 2024-04-03T18:17:49.9302694Z Running distributed/checkpoint/test_state_dict 1/1 ... [2024-04-03 18:17:49.928587] 2024-04-03T18:17:49.9305165Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:17:49.929028] 2024-04-03T18:19:18.4390174Z 2024-04-03T18:19:18.4393215Z distributed/checkpoint/test_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_state_dict_1.1_e315175c13bd7934_.log 2024-04-03T18:19:18.4406613Z Running 11 items in this shard: test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_activation_ckpt_fqns, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_compiled_fsdp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_cpu_offload_full_state_dict, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_ddp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_frozen_parameters, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_fsdp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_fsdp2, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_fsdp_ddp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_partial, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_single_gpu, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_strict 2024-04-03T18:19:18.4414322Z 2024-04-03T18:19:18.4415052Z Running distributed/pipeline/sync/skip/test_verify_skippables 1/1 ... [2024-04-03 18:19:18.439512] 2024-04-03T18:19:18.4417812Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_verify_skippables.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:19:18.439945] 2024-04-03T18:19:20.8592198Z 2024-04-03T18:19:20.8594538Z distributed/pipeline/sync/skip/test_verify_skippables 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_verify_skippables_1.1_c971a164f0c8f629_.log 2024-04-03T18:19:20.8602035Z Running 9 items in this shard: test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated 2024-04-03T18:19:20.8607781Z 2024-04-03T18:19:20.8608305Z Running distributed/test_data_parallel 1/1 ... [2024-04-03 18:19:20.859453] 2024-04-03T18:19:20.8610637Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_data_parallel.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:19:20.859867] 2024-04-03T18:19:26.7863861Z 2024-04-03T18:19:26.7865742Z distributed/test_data_parallel 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_data_parallel_1.1_259678cc80509e74_.log 2024-04-03T18:19:26.7895894Z Running 46 items in this shard: test/distributed/test_data_parallel.py::TestDataParallel::test_autocast, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_buffers_requiring_grad, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_complex, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_device_args, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_function_deletion, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_lazy_linear, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_model_device, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_model_no_refcycles, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_module_zero_inputs, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_multiple_input, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_nested_input, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_nested_output, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_no_grad, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_rnn, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_small_back, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_sparse, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_cpu, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_different_len_dicts, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_gpu, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply_autocast, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply_passes_exception, test/distributed/test_data_parallel.py::TestDataParallel::test_parameter_list_dict_replica, test/distributed/test_data_parallel.py::TestDataParallel::test_replicate, test/distributed/test_data_parallel.py::TestDataParallel::test_replicate_buffers, test/distributed/test_data_parallel.py::TestDataParallel::test_save_replica_module, test/distributed/test_data_parallel.py::TestDataParallel::test_scatter_cpu, test/distributed/test_data_parallel.py::TestDataParallel::test_scatter_gpu, test/distributed/test_data_parallel.py::TestDataParallel::test_strided_grad_layout, test/distributed/test_data_parallel.py::TestDataParallel::test_zero_grad, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float64 2024-04-03T18:19:26.7925267Z 2024-04-03T18:19:26.7925754Z Running distributed/test_store 1/1 ... [2024-04-03 18:19:26.786760] 2024-04-03T18:19:26.7928086Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_store.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:19:26.787224] 2024-04-03T18:26:44.2975798Z 2024-04-03T18:26:44.2978808Z distributed/test_store 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_store_1.1_ba1e3246f41b86e1_.log 2024-04-03T18:26:44.3027446Z Running 85 items in this shard: test/distributed/test_store.py::FileStoreTest::test_append, test/distributed/test_store.py::FileStoreTest::test_compare_set, test/distributed/test_store.py::FileStoreTest::test_init_pg_and_rpc_with_same_file, test/distributed/test_store.py::FileStoreTest::test_multi_get, test/distributed/test_store.py::FileStoreTest::test_multi_set, test/distributed/test_store.py::FileStoreTest::test_refcount, test/distributed/test_store.py::FileStoreTest::test_set_get_check, test/distributed/test_store.py::FileStoreTest::test_simple_wait, test/distributed/test_store.py::HashStoreTest::test_append, test/distributed/test_store.py::HashStoreTest::test_compare_set, test/distributed/test_store.py::HashStoreTest::test_multi_get, test/distributed/test_store.py::HashStoreTest::test_multi_set, test/distributed/test_store.py::HashStoreTest::test_set_get_check, test/distributed/test_store.py::HashStoreTest::test_simple_wait, test/distributed/test_store.py::PrefixStoreTest::test_get_underlying_store, test/distributed/test_store.py::PrefixFileStoreTest::test_append, test/distributed/test_store.py::PrefixFileStoreTest::test_compare_set, test/distributed/test_store.py::PrefixFileStoreTest::test_multi_get, test/distributed/test_store.py::PrefixFileStoreTest::test_multi_set, test/distributed/test_store.py::PrefixFileStoreTest::test_set_get_check, test/distributed/test_store.py::PrefixFileStoreTest::test_simple_wait, test/distributed/test_store.py::TCPStoreTest::test_address_already_in_use, test/distributed/test_store.py::TCPStoreTest::test_append, test/distributed/test_store.py::TCPStoreTest::test_compare_set, test/distributed/test_store.py::TCPStoreTest::test_init_pg_and_rpc_with_same_socket, test/distributed/test_store.py::TCPStoreTest::test_multi_get, test/distributed/test_store.py::TCPStoreTest::test_multi_set, test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_fixed_world_size, test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_nonfixed_world_size, test/distributed/test_store.py::TCPStoreTest::test_multitenancy, test/distributed/test_store.py::TCPStoreTest::test_numkeys_delkeys, test/distributed/test_store.py::TCPStoreTest::test_set_get_check, test/distributed/test_store.py::TCPStoreTest::test_simple_wait, test/distributed/test_store.py::TCPStoreTest::test_store_timeout_on_missing_clients, test/distributed/test_store.py::TCPStoreTest::test_take_over_listen_socket, test/distributed/test_store.py::LibUvTCPStoreTest::test_address_already_in_use, test/distributed/test_store.py::LibUvTCPStoreTest::test_append, test/distributed/test_store.py::LibUvTCPStoreTest::test_compare_set, test/distributed/test_store.py::LibUvTCPStoreTest::test_init_pg_and_rpc_with_same_socket, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_get, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_set, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_fixed_world_size, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_nonfixed_world_size, test/distributed/test_store.py::LibUvTCPStoreTest::test_multitenancy, test/distributed/test_store.py::LibUvTCPStoreTest::test_numkeys_delkeys, test/distributed/test_store.py::LibUvTCPStoreTest::test_set_get_check, test/distributed/test_store.py::LibUvTCPStoreTest::test_simple_wait, test/distributed/test_store.py::LibUvTCPStoreTest::test_store_timeout_on_missing_clients, test/distributed/test_store.py::LibUvTCPStoreTest::test_take_over_listen_socket, test/distributed/test_store.py::PrefixTCPStoreTest::test_append, test/distributed/test_store.py::PrefixTCPStoreTest::test_compare_set, test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_get, test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_set, test/distributed/test_store.py::PrefixTCPStoreTest::test_set_get_check, test/distributed/test_store.py::PrefixTCPStoreTest::test_simple_wait, test/distributed/test_store.py::PrefixTCPStoreTest::test_underlying_non_prefix_store, test/distributed/test_store.py::PythonStoreTest::test_set_get, test/distributed/test_store.py::RendezvousTest::test_unknown_handler, test/distributed/test_store.py::RendezvousTest::test_url_with_node_params, test/distributed/test_store.py::RendezvousEnvTest::test_nominal, test/distributed/test_store.py::RendezvousFileTest::test_common_errors, test/distributed/test_store.py::RendezvousFileTest::test_nominal, test/distributed/test_store.py::RendezvousTCPTest::test_common_errors, test/distributed/test_store.py::RendezvousTCPTest::test_dns_timeout, test/distributed/test_store.py::RendezvousTCPTest::test_nominal, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_doest_break_client, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_set, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_url_with_libuv, test/distributed/test_store.py::TestPythonStore::test_append_roundtrip, test/distributed/test_store.py::TestPythonStore::test_extended_methods_fallbacks, test/distributed/test_store.py::TestPythonStore::test_has_extended_api_passthrough, test/distributed/test_store.py::TestPythonStore::test_has_extended_api_roundtrip, test/distributed/test_store.py::TestPythonStore::test_multi_get_roundtrip, test/distributed/test_store.py::TestPythonStore::test_multi_set_roundtrip, test/distributed/test_store.py::TestPythonStore::test_optional_methods_fail, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_0, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_1, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_2, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_3, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_4, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_5, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_6, test/distributed/test_store.py::TimeoutTest::test_interrupt_doesnt_break_wait, test/distributed/test_store.py::InitPgWithUvStore::test_with_env_var, test/distributed/test_store.py::InitPgWithUvStore::test_with_url_param 2024-04-03T18:26:44.3067901Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_append 2024-04-03T18:26:44.3069178Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_compare_set 2024-04-03T18:26:44.3070545Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_init_pg_and_rpc_with_same_file 2024-04-03T18:26:44.3071909Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_multi_get 2024-04-03T18:26:44.3073148Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_multi_set 2024-04-03T18:26:44.3074469Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_refcount 2024-04-03T18:26:44.3075651Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_set_get_check 2024-04-03T18:26:44.3076920Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_simple_wait 2024-04-03T18:26:44.3078146Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_append 2024-04-03T18:26:44.3079370Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_compare_set 2024-04-03T18:26:44.3080586Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_multi_get 2024-04-03T18:26:44.3081868Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_multi_set 2024-04-03T18:26:44.3083075Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_set_get_check 2024-04-03T18:26:44.3084322Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_simple_wait 2024-04-03T18:26:44.3085780Z Running 1 items in this shard: test/distributed/test_store.py::PrefixStoreTest::test_get_underlying_store 2024-04-03T18:26:44.3086999Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_append 2024-04-03T18:26:44.3088318Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_compare_set 2024-04-03T18:26:44.3089631Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_multi_get 2024-04-03T18:26:44.3090947Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_multi_set 2024-04-03T18:26:44.3092289Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_set_get_check 2024-04-03T18:26:44.3093775Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_simple_wait 2024-04-03T18:26:44.3095140Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_address_already_in_use 2024-04-03T18:26:44.3096414Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_append 2024-04-03T18:26:44.3097620Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_compare_set 2024-04-03T18:26:44.3098979Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_init_pg_and_rpc_with_same_socket 2024-04-03T18:26:44.3100334Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_get 2024-04-03T18:26:44.3101543Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_set 2024-04-03T18:26:44.3102898Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_fixed_world_size 2024-04-03T18:26:44.3104543Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_nonfixed_world_size 2024-04-03T18:26:44.3106295Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multitenancy 2024-04-03T18:26:44.3107592Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_numkeys_delkeys 2024-04-03T18:26:44.3108849Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_set_get_check 2024-04-03T18:26:44.3110228Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_simple_wait 2024-04-03T18:26:44.3111632Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_store_timeout_on_missing_clients 2024-04-03T18:26:44.3113079Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_take_over_listen_socket 2024-04-03T18:26:44.3114482Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_address_already_in_use 2024-04-03T18:26:44.3115847Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_append 2024-04-03T18:26:44.3117126Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_compare_set 2024-04-03T18:26:44.3118553Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_init_pg_and_rpc_with_same_socket 2024-04-03T18:26:44.3119988Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_get 2024-04-03T18:26:44.3121271Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_set 2024-04-03T18:26:44.3122715Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_fixed_world_size 2024-04-03T18:26:44.3124316Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_nonfixed_world_size 2024-04-03T18:26:44.3125811Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multitenancy 2024-04-03T18:26:44.3127160Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_numkeys_delkeys 2024-04-03T18:26:44.3128499Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_set_get_check 2024-04-03T18:26:44.3129822Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_simple_wait 2024-04-03T18:26:44.3131266Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_store_timeout_on_missing_clients 2024-04-03T18:26:44.3132782Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_take_over_listen_socket 2024-04-03T18:26:44.3134125Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_append 2024-04-03T18:26:44.3135427Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_compare_set 2024-04-03T18:26:44.3136903Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_get 2024-04-03T18:26:44.3138180Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_set 2024-04-03T18:26:44.3139500Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_set_get_check 2024-04-03T18:26:44.3140837Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_simple_wait 2024-04-03T18:26:44.3142274Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_underlying_non_prefix_store 2024-04-03T18:26:44.3143637Z Running 1 items in this shard: test/distributed/test_store.py::PythonStoreTest::test_set_get 2024-04-03T18:26:44.3144914Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTest::test_unknown_handler 2024-04-03T18:26:44.3146531Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTest::test_url_with_node_params 2024-04-03T18:26:44.3147843Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousEnvTest::test_nominal 2024-04-03T18:26:44.3149143Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousFileTest::test_common_errors 2024-04-03T18:26:44.3150456Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousFileTest::test_nominal 2024-04-03T18:26:44.3151765Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_common_errors 2024-04-03T18:26:44.3153184Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_dns_timeout 2024-04-03T18:26:44.3154534Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_nominal 2024-04-03T18:26:44.3155973Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_doest_break_client 2024-04-03T18:26:44.3157491Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_set 2024-04-03T18:26:44.3158953Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_url_with_libuv 2024-04-03T18:26:44.3160367Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_append_roundtrip 2024-04-03T18:26:44.3161777Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_extended_methods_fallbacks 2024-04-03T18:26:44.3163248Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_has_extended_api_passthrough 2024-04-03T18:26:44.3164733Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_has_extended_api_roundtrip 2024-04-03T18:26:44.3166155Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_multi_get_roundtrip 2024-04-03T18:26:44.3167535Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_multi_set_roundtrip 2024-04-03T18:26:44.3168920Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_optional_methods_fail 2024-04-03T18:26:44.3170294Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_0 2024-04-03T18:26:44.3171625Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_1 2024-04-03T18:26:44.3172946Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_2 2024-04-03T18:26:44.3174278Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_3 2024-04-03T18:26:44.3175730Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_4 2024-04-03T18:26:44.3176931Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_5 2024-04-03T18:26:44.3178238Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_6 2024-04-03T18:26:44.3179747Z Running 1 items in this shard: test/distributed/test_store.py::TimeoutTest::test_interrupt_doesnt_break_wait 2024-04-03T18:26:44.3181119Z Running 1 items in this shard: test/distributed/test_store.py::InitPgWithUvStore::test_with_env_var 2024-04-03T18:26:44.3182474Z Running 1 items in this shard: test/distributed/test_store.py::InitPgWithUvStore::test_with_url_param 2024-04-03T18:26:44.3183227Z 2024-04-03T18:26:44.3183906Z Running distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 ... [2024-04-03 18:26:44.301006] 2024-04-03T18:26:44.3186701Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:26:44.301501] 2024-04-03T18:30:17.8536433Z 2024-04-03T18:30:17.8538474Z distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_1.1_c5f8719403171f06_.log 2024-04-03T18:30:17.8591464Z Running 67 items in this shard: test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorMetadata::test_serialize_and_deserialize, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestCreateTensorFromParams::test_empty, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardParameter::test_shard_parameter, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardParameter::test_shard_parameter_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardTensor::test_shard_tensor, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardTensor::test_shard_tensor_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardTensor::test_shard_tensor_with_empty_shard, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestModuleHookApi::test_collect_local_shard, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestModuleHookApi::test_reshard_output, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestLocalTensor::test_local_tensor, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestLocalTensor::test_local_tensor_error, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_cleanup, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_complete_world_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_like, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_full, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_ones, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_rand, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_zeros, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_gather_even, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_gather_uneven, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_insufficient_sharding_dims, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_invalid_pg_rpc_ranks, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_invalid_sharding, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_load_state_dict_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_multiple_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_partial_world_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_sharded_tensor_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_sharded_tensor_sizes, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_sharding_columns, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_state_dict, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_state_dict_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_state_dict_no_sharded_tensors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_create_sharded_tensor_with_ones, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_gather_even, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_gather_uneven, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_grid_sharding, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_multiple_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_partial_world_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_device, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_to_cpu, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_to_cuda, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_to_test, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_uneven_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_with_rpc_names, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalTensor::test_init_from_local_tensor, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalTensor::test_init_from_local_tensor_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_and_global_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_and_global_metadata_invalid_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_pin_memory, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_property_cross_ranks, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_shards_gaps, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_shards_overlap, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_st_base_init_from_local_shards_and_global_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorCustomOps::test_custom_op, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorCustomOps::test_custom_op_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorCustomOps::test_custom_op_override, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardMetadata::test_create_shard_with_no_placement, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardMetadata::test_shard_metadata_init, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestCreateTensorNoProcessGroupMode::test_init_from_local_shards_and_global_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestCreateTensorNoProcessGroupMode::test_non_contiguous_local_shards 2024-04-03T18:30:17.8643100Z 2024-04-03T18:30:17.8643593Z Running distributed/test_c10d_nccl 2/2 ... [2024-04-03 18:30:17.854051] 2024-04-03T18:30:17.8645976Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=2', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:30:17.854544] 2024-04-03T18:46:13.8836399Z 2024-04-03T18:46:13.8838085Z distributed/test_c10d_nccl 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_2.2_fa9bfe56bb0b5164_.log 2024-04-03T18:46:13.8917083Z Running 116 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_abort_in_destroy_mixed_empty_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_allgather_base_basics, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_allgather_base_ops, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_allgather_ops, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_alltoall_ops_with_cudafree_race, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_barrier, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_comm_split_optimization, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_file_store_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_gather_ops, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_gather_stress, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_init_process_group_nccl_timeout, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_nccl_dist_backend_error, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_nccl_watchdog_cudagraph, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_reduce_scatter_base_basics, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_reduce_scatter_ops, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_scatter_checks, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_scatter_ops, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_send_recv, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_send_recv_complex, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_tensor_register_hook, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_terminate_before_destruct_pg, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info, test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_nccl.py::CompilerTest::test_allgather_work_wait_gpu, test/distributed/test_c10d_nccl.py::CompilerTest::test_allreduce_work_wait_gpu, test/distributed/test_c10d_nccl.py::CompilerTest::test_broadcast_work_wait_gpu, test/distributed/test_c10d_nccl.py::CompilerTest::test_consecutive_comm_work_wait_gpu, test/distributed/test_c10d_nccl.py::CompilerTest::test_nested_comm_tensor_wrapping, test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_tensor_work_wait_gpu, test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_work_wait_gpu, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check, test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_False, test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long, test/distributed/test_c10d_nccl.py::NCCLTraceTestTimeoutDumpOnStuckRanks::test_timeout_dumps_on_stuck_ranks 2024-04-03T18:46:13.8988196Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl 2024-04-03T18:46:13.8989795Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_abort_in_destroy_mixed_empty_pgs 2024-04-03T18:46:13.8991418Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_allgather_base_basics 2024-04-03T18:46:13.8992970Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_allgather_base_ops 2024-04-03T18:46:13.8994448Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_allgather_ops 2024-04-03T18:46:13.8995983Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_alltoall_ops_with_cudafree_race 2024-04-03T18:46:13.8997776Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_barrier 2024-04-03T18:46:13.8999279Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_comm_split_optimization 2024-04-03T18:46:13.9000789Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_file_store_check 2024-04-03T18:46:13.9002232Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_gather_ops 2024-04-03T18:46:13.9003661Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_gather_stress 2024-04-03T18:46:13.9005209Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_init_process_group_nccl_timeout 2024-04-03T18:46:13.9006808Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_nccl_dist_backend_error 2024-04-03T18:46:13.9008384Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_nccl_watchdog_cudagraph 2024-04-03T18:46:13.9009975Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_reduce_scatter_base_basics 2024-04-03T18:46:13.9011527Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_reduce_scatter_ops 2024-04-03T18:46:13.9012982Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_scatter_checks 2024-04-03T18:46:13.9014417Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_scatter_ops 2024-04-03T18:46:13.9015823Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_send_recv 2024-04-03T18:46:13.9017242Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_send_recv_complex 2024-04-03T18:46:13.9018803Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_tensor_register_hook 2024-04-03T18:46:13.9020392Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_terminate_before_destruct_pg 2024-04-03T18:46:13.9022076Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module 2024-04-03T18:46:13.9023888Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view 2024-04-03T18:46:13.9026078Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl 2024-04-03T18:46:13.9027811Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl 2024-04-03T18:46:13.9029466Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output 2024-04-03T18:46:13.9031125Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param 2024-04-03T18:46:13.9032898Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2024-04-03T18:46:13.9034751Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2024-04-03T18:46:13.9036650Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2024-04-03T18:46:13.9038624Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False 2024-04-03T18:46:13.9040619Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2024-04-03T18:46:13.9042657Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2024-04-03T18:46:13.9044644Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False 2024-04-03T18:46:13.9046633Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2024-04-03T18:46:13.9048655Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2024-04-03T18:46:13.9050639Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view 2024-04-03T18:46:13.9052581Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph 2024-04-03T18:46:13.9054344Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params 2024-04-03T18:46:13.9055963Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence 2024-04-03T18:46:13.9057614Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters 2024-04-03T18:46:13.9059296Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl 2024-04-03T18:46:13.9061070Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view 2024-04-03T18:46:13.9062921Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail 2024-04-03T18:46:13.9064749Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view 2024-04-03T18:46:13.9066635Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl 2024-04-03T18:46:13.9068294Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view 2024-04-03T18:46:13.9070065Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess 2024-04-03T18:46:13.9072030Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule 2024-04-03T18:46:13.9073764Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward 2024-04-03T18:46:13.9075650Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view 2024-04-03T18:46:13.9077621Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list 2024-04-03T18:46:13.9079623Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list 2024-04-03T18:46:13.9081458Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module 2024-04-03T18:46:13.9083153Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module 2024-04-03T18:46:13.9084937Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed 2024-04-03T18:46:13.9086882Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids 2024-04-03T18:46:13.9088843Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason 2024-04-03T18:46:13.9090460Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad 2024-04-03T18:46:13.9092060Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error 2024-04-03T18:46:13.9093711Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg 2024-04-03T18:46:13.9095336Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl 2024-04-03T18:46:13.9097131Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view 2024-04-03T18:46:13.9098938Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2024-04-03T18:46:13.9100669Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input 2024-04-03T18:46:13.9102371Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object 2024-04-03T18:46:13.9103928Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast 2024-04-03T18:46:13.9105446Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops 2024-04-03T18:46:13.9107076Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq 2024-04-03T18:46:13.9108638Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier 2024-04-03T18:46:13.9110294Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort 2024-04-03T18:46:13.9111944Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit 2024-04-03T18:46:13.9113473Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout 2024-04-03T18:46:13.9114903Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl 2024-04-03T18:46:13.9116468Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl 2024-04-03T18:46:13.9117923Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail 2024-04-03T18:46:13.9119487Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info 2024-04-03T18:46:13.9120926Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership 2024-04-03T18:46:13.9122312Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config 2024-04-03T18:46:13.9123787Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream 2024-04-03T18:46:13.9125356Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default 2024-04-03T18:46:13.9126888Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group 2024-04-03T18:46:13.9128304Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex 2024-04-03T18:46:13.9129651Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch 2024-04-03T18:46:13.9131074Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_allgather_work_wait_gpu 2024-04-03T18:46:13.9132645Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_allreduce_work_wait_gpu 2024-04-03T18:46:13.9134094Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_broadcast_work_wait_gpu 2024-04-03T18:46:13.9135587Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_consecutive_comm_work_wait_gpu 2024-04-03T18:46:13.9137104Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_nested_comm_tensor_wrapping 2024-04-03T18:46:13.9138642Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_tensor_work_wait_gpu 2024-04-03T18:46:13.9140167Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_work_wait_gpu 2024-04-03T18:46:13.9141946Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single 2024-04-03T18:46:13.9143881Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced 2024-04-03T18:46:13.9146070Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-04-03T18:46:13.9148210Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-04-03T18:46:13.9149984Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup 2024-04-03T18:46:13.9151412Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup 2024-04-03T18:46:13.9152819Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup 2024-04-03T18:46:13.9154202Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync 2024-04-03T18:46:13.9155710Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg 2024-04-03T18:46:13.9157298Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check 2024-04-03T18:46:13.9158744Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup 2024-04-03T18:46:13.9160357Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup 2024-04-03T18:46:13.9161909Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_False 2024-04-03T18:46:13.9163450Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata 2024-04-03T18:46:13.9165151Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False 2024-04-03T18:46:13.9167076Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False 2024-04-03T18:46:13.9168988Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True 2024-04-03T18:46:13.9170843Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True 2024-04-03T18:46:13.9172361Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe 2024-04-03T18:46:13.9173912Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False 2024-04-03T18:46:13.9175827Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True 2024-04-03T18:46:13.9177643Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True 2024-04-03T18:46:13.9179129Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long 2024-04-03T18:46:13.9180682Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTestTimeoutDumpOnStuckRanks::test_timeout_dumps_on_stuck_ranks 2024-04-03T18:46:13.9181732Z 2024-04-03T18:46:14.4714980Z 2024-04-03T18:46:14.4715764Z real 142m48.888s 2024-04-03T18:46:14.4716237Z user 244m12.488s 2024-04-03T18:46:14.4716562Z sys 99m42.324s 2024-04-03T18:46:14.4716941Z + assert_git_not_dirty 2024-04-03T18:46:14.4717643Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-04-03T18:46:14.4718412Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-04-03T18:46:14.4721155Z ++ git status --porcelain 2024-04-03T18:46:14.4722096Z ++ grep -v '?? third_party' 2024-04-03T18:46:15.9493553Z ++ true 2024-04-03T18:46:15.9494467Z + git_status= 2024-04-03T18:46:15.9494963Z + [[ -n '' ]] 2024-04-03T18:46:15.9495526Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-04-03T18:46:15.9496117Z + [[ 3 == 1 ]] 2024-04-03T18:46:15.9496467Z + [[ 3 == 1 ]] 2024-04-03T18:46:15.9579304Z ##[group]Run cat test/**/*_toprint.log || true 2024-04-03T18:46:15.9579969Z cat test/**/*_toprint.log || true 2024-04-03T18:46:15.9589224Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:46:15.9589858Z env: 2024-04-03T18:46:15.9590196Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:46:15.9590744Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:46:15.9591689Z DOCKER_CONTAINER_ID: e451a0adc73818f3a047cb83ff91d039adc3cd55a8f5a0cb827c973fea5bd896 2024-04-03T18:46:15.9592514Z ##[endgroup] 2024-04-03T18:46:15.9665303Z cat: test/**/*_toprint.log: No such file or directory 2024-04-03T18:46:15.9704515Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-04-03T18:46:15.9705135Z kill "$MONITOR_SCRIPT_PID" 2024-04-03T18:46:15.9713459Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:46:15.9714086Z env: 2024-04-03T18:46:15.9714405Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:46:15.9714973Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:46:15.9715918Z DOCKER_CONTAINER_ID: e451a0adc73818f3a047cb83ff91d039adc3cd55a8f5a0cb827c973fea5bd896 2024-04-03T18:46:15.9716953Z MONITOR_SCRIPT_PID: 55246 2024-04-03T18:46:15.9717397Z ##[endgroup] 2024-04-03T18:46:15.9931961Z Prepare all required actions 2024-04-03T18:46:15.9932553Z Getting action download info 2024-04-03T18:46:16.1848060Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-04-03T18:46:16.3841974Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-04-03T18:46:16.3842541Z with: 2024-04-03T18:46:16.3843149Z file-suffix: test-distributed-3-3-linux.8xlarge.nvidia.gpu_23402706518 2024-04-03T18:46:16.3843921Z s3-bucket: gha-artifacts 2024-04-03T18:46:16.3844316Z env: 2024-04-03T18:46:16.3844645Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:46:16.3845201Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:46:16.3846125Z DOCKER_CONTAINER_ID: e451a0adc73818f3a047cb83ff91d039adc3cd55a8f5a0cb827c973fea5bd896 2024-04-03T18:46:16.3846941Z ##[endgroup] 2024-04-03T18:46:16.3900905Z ##[group]Run # Remove any previous test jsons if they exist 2024-04-03T18:46:16.3901748Z # Remove any previous test jsons if they exist 2024-04-03T18:46:16.3902371Z rm -f test-jsons-*.zip 2024-04-03T18:46:16.3903025Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-04-03T18:46:16.3913191Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:46:16.3913852Z env: 2024-04-03T18:46:16.3914451Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:46:16.3915034Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:46:16.3915981Z DOCKER_CONTAINER_ID: e451a0adc73818f3a047cb83ff91d039adc3cd55a8f5a0cb827c973fea5bd896 2024-04-03T18:46:16.3917080Z FILE_SUFFIX: test-distributed-3-3-linux.8xlarge.nvidia.gpu_23402706518 2024-04-03T18:46:16.3917805Z ##[endgroup] 2024-04-03T18:46:16.4221073Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-04-03T18:46:16.4256452Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-04-03T18:46:16.4257329Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-04-03T18:46:16.4264590Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-04-03T18:46:16.4265889Z adding: test/.pytorch-slow-tests.json (deflated 59%) 2024-04-03T18:46:16.4274492Z adding: test/.pytorch-disabled-tests.json (deflated 84%) 2024-04-03T18:46:16.4308571Z ##[group]Run # Remove any previous test reports if they exist 2024-04-03T18:46:16.4309380Z # Remove any previous test reports if they exist 2024-04-03T18:46:16.4310031Z rm -f test-reports-*.zip 2024-04-03T18:46:16.4310759Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-04-03T18:46:16.4318944Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:46:16.4319545Z env: 2024-04-03T18:46:16.4319879Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:46:16.4320442Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:46:16.4321361Z DOCKER_CONTAINER_ID: e451a0adc73818f3a047cb83ff91d039adc3cd55a8f5a0cb827c973fea5bd896 2024-04-03T18:46:16.4322465Z FILE_SUFFIX: test-distributed-3-3-linux.8xlarge.nvidia.gpu_23402706518 2024-04-03T18:46:16.4323210Z ##[endgroup] 2024-04-03T18:46:16.4748402Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_logger/distributed._shard.sharded_tensor.test_logger-2a511f415f2bf12d.xml (deflated 37%) 2024-04-03T18:46:16.4750651Z adding: test/test-reports/python-pytest/distributed._tensor.test_init/distributed._tensor.test_init-87d934b5e541c600.xml (deflated 81%) 2024-04-03T18:46:16.4752794Z adding: test/test-reports/python-pytest/distributed._composable.test_checkpoint/distributed._composable.test_checkpoint-72fbd35f0e85b0a5.xml (deflated 69%) 2024-04-03T18:46:16.4755076Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_compile/distributed._tensor.test_dtensor_compile-e25474a445a01234.xml (deflated 90%) 2024-04-03T18:46:16.4757314Z adding: test/test-reports/python-pytest/distributed.checkpoint.e2e.test_fsdp_ep/distributed.checkpoint.e2e.test_fsdp_ep-73e0f9243be4ff50.xml (deflated 50%) 2024-04-03T18:46:16.4760061Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_save_load_api/distributed.checkpoint.test_save_load_api-8ed126878d82efe8.xml (deflated 50%) 2024-04-03T18:46:16.4762340Z adding: test/test-reports/python-pytest/distributed._composable.test_compose/distributed._composable.test_compose-85c4ae13d1f6b1b4.xml (deflated 76%) 2024-04-03T18:46:16.4764569Z adding: test/test-reports/python-pytest/distributed._tensor.debug.test_comm_mode/distributed._tensor.debug.test_comm_mode-ce0934584518d724.xml (deflated 50%) 2024-04-03T18:46:16.4766943Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_examples/distributed.tensor.parallel.test_tp_examples-2f1c171addebde38.xml (deflated 75%) 2024-04-03T18:46:16.4769236Z adding: test/test-reports/python-pytest/distributed.test_inductor_collectives/distributed.test_inductor_collectives-038b4c9cfe263fb1.xml (deflated 93%) 2024-04-03T18:46:16.4771594Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_backward_prefetch/distributed.fsdp.test_fsdp_backward_prefetch-5798cccfe211133c.xml (deflated 41%) 2024-04-03T18:46:16.4773868Z adding: test/test-reports/python-pytest/distributed._composable.test_replicate/distributed._composable.test_replicate-14333bb815cab379.xml (deflated 76%) 2024-04-03T18:46:16.4776268Z adding: test/test-reports/python-pytest/distributed._shard.sharding_plan.test_sharding_plan/distributed._shard.sharding_plan.test_sharding_plan-38d1aa843d41211c.xml (deflated 73%) 2024-04-03T18:46:16.4779190Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_clip_grad_norm_/distributed._composable.fsdp.test_fully_shard_clip_grad_norm_-2a0cc7dac2a4faf2.xml (deflated 55%) 2024-04-03T18:46:16.4782028Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion-09180be6deffaa05.xml (deflated 41%) 2024-04-03T18:46:16.4784452Z adding: test/test-reports/python-pytest/distributed._tensor.test_optimizers/distributed._tensor.test_optimizers-71a3e62ed098e5ea.xml (deflated 79%) 2024-04-03T18:46:16.4787349Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_frozen/distributed._composable.fsdp.test_fully_shard_frozen-5908e2716fdaff7c.xml (deflated 59%) 2024-04-03T18:46:16.4789798Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_flatten_params/distributed.fsdp.test_fsdp_flatten_params-cf90625a516f3fbd.xml (deflated 79%) 2024-04-03T18:46:16.4791978Z adding: test/test-reports/python-pytest/distributed.test_functional_api/distributed.test_functional_api-da35dfbb194a8084.xml (deflated 93%) 2024-04-03T18:46:16.4794173Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_ignored_modules/distributed.fsdp.test_fsdp_ignored_modules-60aea965e8b57034.xml (deflated 83%) 2024-04-03T18:46:16.4796496Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_checkpoint/distributed.checkpoint.test_checkpoint-da00563b252b2317.xml (deflated 79%) 2024-04-03T18:46:16.4798683Z adding: test/test-reports/python-pytest/distributed.rpc.test_share_memory/distributed.rpc.test_share_memory-64ea4421edf56a98.xml (deflated 48%) 2024-04-03T18:46:16.4800941Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_freezing_weights/distributed.fsdp.test_fsdp_freezing_weights-83c2a98ae40f3c9a.xml (deflated 94%) 2024-04-03T18:46:16.4803224Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-0a06dec9a0dd704e.xml (deflated 40%) 2024-04-03T18:46:16.4805369Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-b38f7e46af32acf3.xml (deflated 50%) 2024-04-03T18:46:16.4807462Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-9c2f8c172ff302c1.xml (deflated 38%) 2024-04-03T18:46:16.4810015Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_multiple_wrapping/distributed.fsdp.test_fsdp_multiple_wrapping-eb0f62d4c5f74c27.xml (deflated 40%) 2024-04-03T18:46:16.4812632Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_runtime/distributed._composable.fully_shard.test_fully_shard_runtime-7ec14a0d148c7ae7.xml (deflated 51%) 2024-04-03T18:46:16.4815163Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-b2fbdb29a574ce1a.xml (deflated 61%) 2024-04-03T18:46:16.4817432Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_exec_order/distributed.fsdp.test_fsdp_exec_order-2e2b55d29575b5e2.xml (deflated 78%) 2024-04-03T18:46:16.4819683Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_pipe/distributed.pipeline.sync.test_pipe-5e192fc5699db6b0.xml (deflated 86%) 2024-04-03T18:46:16.4821897Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_fine_tune/distributed.fsdp.test_fsdp_fine_tune-1f9c5899ba27e713.xml (deflated 60%) 2024-04-03T18:46:16.4824018Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-092963bf6783450f.xml (deflated 82%) 2024-04-03T18:46:16.4826595Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-90c1ca99928cb9e8.xml (deflated 88%) 2024-04-03T18:46:16.4828873Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_worker/distributed.pipeline.sync.test_worker-a0f215b4810aa617.xml (deflated 71%) 2024-04-03T18:46:16.4831244Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_transparency/distributed.pipeline.sync.test_transparency-378484c4d7b4039f.xml (deflated 41%) 2024-04-03T18:46:16.4833995Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_optim_checkpoint/distributed._composable.fully_shard.test_fully_shard_optim_checkpoint-31e8eafb0f44ad89.xml (deflated 54%) 2024-04-03T18:46:16.4836627Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_unshard_params/distributed.fsdp.test_fsdp_unshard_params-c41034c68a060565.xml (deflated 81%) 2024-04-03T18:46:16.4838977Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_hybrid_shard/distributed.fsdp.test_fsdp_hybrid_shard-75f677ca73eb7baa.xml (deflated 74%) 2024-04-03T18:46:16.4841267Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_inplace/distributed.pipeline.sync.test_inplace-b9decec9da64c352.xml (deflated 57%) 2024-04-03T18:46:16.4843620Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-395d67943d270a9e.xml (deflated 38%) 2024-04-03T18:46:16.4846410Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-90d2887497cd5131.xml (deflated 38%) 2024-04-03T18:46:16.4848813Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-70f487c620ee78bd.xml (deflated 37%) 2024-04-03T18:46:16.4851196Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-06cbeff3f66f1407.xml (deflated 38%) 2024-04-03T18:46:16.4853559Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-444b750773dafcfb.xml (deflated 37%) 2024-04-03T18:46:16.4855896Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-24c4e4a0ee4ac8c7.xml (deflated 37%) 2024-04-03T18:46:16.4858317Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ec0e0d98e0b78f43.xml (deflated 47%) 2024-04-03T18:46:16.4860859Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e5423d11fab58742.xml (deflated 47%) 2024-04-03T18:46:16.4863394Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-18930f272d765b1b.xml (deflated 47%) 2024-04-03T18:46:16.4866231Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-97c52798cc9685c0.xml (deflated 47%) 2024-04-03T18:46:16.4868612Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-38759cd6eebe2d29.xml (deflated 47%) 2024-04-03T18:46:16.4871008Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-11b47b250ddcc128.xml (deflated 38%) 2024-04-03T18:46:16.4873350Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1dcf6fbf91e2a893.xml (deflated 39%) 2024-04-03T18:46:16.4875758Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-be2d2447c1aa1ac6.xml (deflated 39%) 2024-04-03T18:46:16.4878150Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-f1b8f1ce715b6f30.xml (deflated 39%) 2024-04-03T18:46:16.4880515Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1342ed451edc2b24.xml (deflated 39%) 2024-04-03T18:46:16.4882847Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a9541b8cc41e4694.xml (deflated 39%) 2024-04-03T18:46:16.4885249Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-be0cf7d13de785a7.xml (deflated 39%) 2024-04-03T18:46:16.4887618Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a8cce37bcde90fe6.xml (deflated 38%) 2024-04-03T18:46:16.4889998Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-006fe66ec34450de.xml (deflated 39%) 2024-04-03T18:46:16.4892365Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-7984fadb99fdbadd.xml (deflated 38%) 2024-04-03T18:46:16.4894738Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-f9b05b77f331537b.xml (deflated 38%) 2024-04-03T18:46:16.4897106Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9d71cbf91df6d1f5.xml (deflated 39%) 2024-04-03T18:46:16.4899476Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-86099f3b6c73361a.xml (deflated 38%) 2024-04-03T18:46:16.4901855Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-cef5071eadfcb35d.xml (deflated 38%) 2024-04-03T18:46:16.4904221Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-7b1f1b5efcb502d3.xml (deflated 38%) 2024-04-03T18:46:16.4907019Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5295f891964815f4.xml (deflated 39%) 2024-04-03T18:46:16.4909376Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d0bcf665f8932939.xml (deflated 38%) 2024-04-03T18:46:16.4911865Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9633dad7c0b95158.xml (deflated 38%) 2024-04-03T18:46:16.4914392Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-96626f07ea91d6d0.xml (deflated 39%) 2024-04-03T18:46:16.4916754Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d00fb86bcdf9c53d.xml (deflated 38%) 2024-04-03T18:46:16.4919164Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1b147824f96eaa7c.xml (deflated 39%) 2024-04-03T18:46:16.4921535Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-46d515ba96f6b4f2.xml (deflated 39%) 2024-04-03T18:46:16.4923868Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ff9e3438583aec5c.xml (deflated 38%) 2024-04-03T18:46:16.4926272Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-db845ed9a2fcc97c.xml (deflated 39%) 2024-04-03T18:46:16.4928640Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-cc5eecc8376fd24b.xml (deflated 39%) 2024-04-03T18:46:16.4931013Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-31cded38aa552c56.xml (deflated 39%) 2024-04-03T18:46:16.4933369Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-8e5618658c0db332.xml (deflated 38%) 2024-04-03T18:46:16.4935724Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e96676bafa4c53a1.xml (deflated 39%) 2024-04-03T18:46:16.4938100Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0557b4b012374f57.xml (deflated 48%) 2024-04-03T18:46:16.4940460Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0fe3a9c51a858f43.xml (deflated 48%) 2024-04-03T18:46:16.4942834Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1221b446c15a55dd.xml (deflated 38%) 2024-04-03T18:46:16.4945177Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2b782ba19479ce17.xml (deflated 39%) 2024-04-03T18:46:16.4948008Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-8cf15f593187e7f1.xml (deflated 39%) 2024-04-03T18:46:16.4950383Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c214d22ec956a1d9.xml (deflated 38%) 2024-04-03T18:46:16.4952759Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c49658f5155ca679.xml (deflated 38%) 2024-04-03T18:46:16.4955098Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-35eec6cc24726d33.xml (deflated 38%) 2024-04-03T18:46:16.4957470Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9be5859bfb2a6e7a.xml (deflated 37%) 2024-04-03T18:46:16.4959845Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-47b422563706dfc5.xml (deflated 38%) 2024-04-03T18:46:16.4962211Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d5cefa322f2904e8.xml (deflated 49%) 2024-04-03T18:46:16.4964836Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-52c53a04a25d1add.xml (deflated 48%) 2024-04-03T18:46:16.4967010Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-e870a6893e7d2b0d.xml (deflated 37%) 2024-04-03T18:46:16.4968943Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-81cedce05421faf1.xml (deflated 37%) 2024-04-03T18:46:16.4970871Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-909c0463ed14fa1e.xml (deflated 37%) 2024-04-03T18:46:16.4972790Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-419000eb8d3684e1.xml (deflated 36%) 2024-04-03T18:46:16.4974698Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-05b9c2a51b544386.xml (deflated 37%) 2024-04-03T18:46:16.4976651Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-db4824859b9d4c2d.xml (deflated 37%) 2024-04-03T18:46:16.4978563Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-768d3b26d7776599.xml (deflated 38%) 2024-04-03T18:46:16.4980460Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-725a5c6e0f3c4a08.xml (deflated 49%) 2024-04-03T18:46:16.4982374Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-3c5368f80b023d49.xml (deflated 48%) 2024-04-03T18:46:16.4984295Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-14c3104a99931991.xml (deflated 37%) 2024-04-03T18:46:16.4986600Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-c9bdf9cbe8e6b05e.xml (deflated 37%) 2024-04-03T18:46:16.4988593Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-71c84f445c1cca9e.xml (deflated 38%) 2024-04-03T18:46:16.4990500Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-11021251a1bd98fb.xml (deflated 49%) 2024-04-03T18:46:16.4992417Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-f8135fb084e39d95.xml (deflated 48%) 2024-04-03T18:46:16.4994435Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-b2253581fed78f83.xml (deflated 37%) 2024-04-03T18:46:16.4996713Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_tensor_ops/distributed._shard.sharded_tensor.ops.test_tensor_ops-997edd9b1f1165ce.xml (deflated 80%) 2024-04-03T18:46:16.4999017Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-e992587980cd5b1e.xml (deflated 48%) 2024-04-03T18:46:16.5001017Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-9f28d3ef90923638.xml (deflated 35%) 2024-04-03T18:46:16.5003041Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-40b8cd8e36d52bd2.xml (deflated 36%) 2024-04-03T18:46:16.5005055Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-3dab3fde1177e516.xml (deflated 35%) 2024-04-03T18:46:16.5007068Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-094f7986dea9ec7b.xml (deflated 35%) 2024-04-03T18:46:16.5009057Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-16638e33f57a26a9.xml (deflated 36%) 2024-04-03T18:46:16.5011179Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_balance/distributed.pipeline.sync.test_balance-ee30ae50568fc227.xml (deflated 82%) 2024-04-03T18:46:16.5014070Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/distributed._shard.sharded_tensor.test_sharded_tensor_reshard-f15cfaa6495fbff0.xml (deflated 68%) 2024-04-03T18:46:16.5016701Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_dependency/distributed.pipeline.sync.test_dependency-10c3bbefc8e2635c.xml (deflated 74%) 2024-04-03T18:46:16.5019130Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_api/distributed.pipeline.sync.skip.test_api-bc1c4eb108b719f4.xml (deflated 60%) 2024-04-03T18:46:16.5021635Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_deferred_batch_norm/distributed.pipeline.sync.test_deferred_batch_norm-7874b625d1b40113.xml (deflated 79%) 2024-04-03T18:46:16.5024048Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_state_dict/distributed.checkpoint.test_state_dict-4951119aa0f4a32c.xml (deflated 76%) 2024-04-03T18:46:16.5027034Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_verify_skippables/distributed.pipeline.sync.skip.test_verify_skippables-54fee72ae92bf51b.xml (deflated 79%) 2024-04-03T18:46:16.5029392Z adding: test/test-reports/python-pytest/distributed.test_data_parallel/distributed.test_data_parallel-488fbf96568d8688.xml (deflated 85%) 2024-04-03T18:46:16.5031283Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5894e9555d12b310.xml (deflated 50%) 2024-04-03T18:46:16.5033066Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-732aab402fc2cf1c.xml (deflated 37%) 2024-04-03T18:46:16.5034863Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-ad36d38424dd8608.xml (deflated 41%) 2024-04-03T18:46:16.5036633Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-42fd6f29a649cbaf.xml (deflated 50%) 2024-04-03T18:46:16.5038417Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-b746bd5c5a910b2b.xml (deflated 50%) 2024-04-03T18:46:16.5040219Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-67a6056e5ea31452.xml (deflated 38%) 2024-04-03T18:46:16.5042006Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e7d2d40f9400c5e6.xml (deflated 38%) 2024-04-03T18:46:16.5043770Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e90bd1fa5a5e545a.xml (deflated 38%) 2024-04-03T18:46:16.5045558Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5c7bbe4e74a81414.xml (deflated 38%) 2024-04-03T18:46:16.5047404Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5e7c1af5f85d2a3c.xml (deflated 38%) 2024-04-03T18:46:16.5049194Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-a0189f53fb34a013.xml (deflated 38%) 2024-04-03T18:46:16.5050970Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-704f968c0c425081.xml (deflated 38%) 2024-04-03T18:46:16.5052749Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-815c7ca2cbc3cc6e.xml (deflated 38%) 2024-04-03T18:46:16.5054540Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5d730fce5df80d27.xml (deflated 38%) 2024-04-03T18:46:16.5056281Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-bfe77c5ecbc7e73d.xml (deflated 36%) 2024-04-03T18:46:16.5058103Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-8e68b0ea308f2658.xml (deflated 50%) 2024-04-03T18:46:16.5059880Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-2534e98622479d6b.xml (deflated 37%) 2024-04-03T18:46:16.5061649Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-75a039824e93a93f.xml (deflated 50%) 2024-04-03T18:46:16.5063669Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5a32aba4979cfb2b.xml (deflated 50%) 2024-04-03T18:46:16.5065455Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-ad1d864dd4c9d28d.xml (deflated 38%) 2024-04-03T18:46:16.5067783Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5a8715b225d0c4b2.xml (deflated 37%) 2024-04-03T18:46:16.5069572Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-a18b3ebce4d80c3c.xml (deflated 38%) 2024-04-03T18:46:16.5071332Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-bfcf17296383d6b9.xml (deflated 38%) 2024-04-03T18:46:16.5073112Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-763ce824e193bd08.xml (deflated 38%) 2024-04-03T18:46:16.5074903Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e92eab3b9b2bd362.xml (deflated 41%) 2024-04-03T18:46:16.5076709Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-43bbc475fc2233c6.xml (deflated 38%) 2024-04-03T18:46:16.5078467Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-7d49cb69ac08df74.xml (deflated 38%) 2024-04-03T18:46:16.5080215Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-b3cd069373b96ea2.xml (deflated 37%) 2024-04-03T18:46:16.5082026Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-8802b6a143741f33.xml (deflated 37%) 2024-04-03T18:46:16.5083786Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-1b9a7dfdd08b6b3b.xml (deflated 37%) 2024-04-03T18:46:16.5085578Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-04922c9994eeaef6.xml (deflated 38%) 2024-04-03T18:46:16.5087376Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-f8f75fdc104b8495.xml (deflated 38%) 2024-04-03T18:46:16.5089124Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-2646654abdea6801.xml (deflated 38%) 2024-04-03T18:46:16.5090909Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-aa86ccbd75816c57.xml (deflated 38%) 2024-04-03T18:46:16.5092696Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-78088c22c46edefa.xml (deflated 37%) 2024-04-03T18:46:16.5094448Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-a306b6b311a45bce.xml (deflated 37%) 2024-04-03T18:46:16.5096265Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-ecd756fe1e406f48.xml (deflated 38%) 2024-04-03T18:46:16.5098030Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-34741f40d8cf82a5.xml (deflated 37%) 2024-04-03T18:46:16.5099824Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d9f9fda8afa1b67f.xml (deflated 41%) 2024-04-03T18:46:16.5101633Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-0f80fe746d134f6f.xml (deflated 37%) 2024-04-03T18:46:16.5103392Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-49f3440522a6d5c8.xml (deflated 37%) 2024-04-03T18:46:16.5105131Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-baaa6fe45cb84220.xml (deflated 36%) 2024-04-03T18:46:16.5107404Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5b1ab3da5ba2ab99.xml (deflated 36%) 2024-04-03T18:46:16.5109190Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-28b3c924c87c728d.xml (deflated 37%) 2024-04-03T18:46:16.5111087Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-9043b374c1bc2e69.xml (deflated 37%) 2024-04-03T18:46:16.5113003Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-6169483d48256b90.xml (deflated 37%) 2024-04-03T18:46:16.5114762Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-2628c72528989df6.xml (deflated 37%) 2024-04-03T18:46:16.5116525Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-376fb194209a7654.xml (deflated 37%) 2024-04-03T18:46:16.5118294Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-46c75b36c3d76b37.xml (deflated 37%) 2024-04-03T18:46:16.5120133Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-edec6e6c380740ee.xml (deflated 37%) 2024-04-03T18:46:16.5121912Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-ab886c06f0d56da7.xml (deflated 37%) 2024-04-03T18:46:16.5123719Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-3612927fd38b1b7d.xml (deflated 37%) 2024-04-03T18:46:16.5125470Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-799502181d6fdd93.xml (deflated 38%) 2024-04-03T18:46:16.5127240Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-f40b9f46929b42ca.xml (deflated 38%) 2024-04-03T18:46:16.5129024Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-c223b1979253855f.xml (deflated 38%) 2024-04-03T18:46:16.5130771Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-da194c5595d85c25.xml (deflated 38%) 2024-04-03T18:46:16.5132552Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-2d902e11089f8e15.xml (deflated 38%) 2024-04-03T18:46:16.5134316Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-0d4e0fb232866095.xml (deflated 38%) 2024-04-03T18:46:16.5136130Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5f3f7ecf3f46441c.xml (deflated 37%) 2024-04-03T18:46:16.5137898Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-1e393543a3a0ba59.xml (deflated 37%) 2024-04-03T18:46:16.5139673Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-38e3a10e17df9055.xml (deflated 38%) 2024-04-03T18:46:16.5141442Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-6fb96473acb69432.xml (deflated 38%) 2024-04-03T18:46:16.5143229Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d36ae2055a1d78cf.xml (deflated 38%) 2024-04-03T18:46:16.5145026Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-36c8584160aff107.xml (deflated 38%) 2024-04-03T18:46:16.5147355Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-4bdca6801b129f51.xml (deflated 37%) 2024-04-03T18:46:16.5149144Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-6e86c28cd38c4ef6.xml (deflated 38%) 2024-04-03T18:46:16.5150914Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-0845e3b2b792a91f.xml (deflated 38%) 2024-04-03T18:46:16.5152649Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d6b27967ac46c73a.xml (deflated 37%) 2024-04-03T18:46:16.5154471Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-33d1becefe71e446.xml (deflated 38%) 2024-04-03T18:46:16.5156249Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-7951fbb893273d2d.xml (deflated 37%) 2024-04-03T18:46:16.5158028Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-abdb82bd90881439.xml (deflated 37%) 2024-04-03T18:46:16.5160055Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-a09f994d73abb376.xml (deflated 37%) 2024-04-03T18:46:16.5161848Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-051105d25515facb.xml (deflated 38%) 2024-04-03T18:46:16.5163615Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-de255521ff907f8d.xml (deflated 37%) 2024-04-03T18:46:16.5165413Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-bc80a5816a24b0a6.xml (deflated 38%) 2024-04-03T18:46:16.5167155Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-10c8585125e39e76.xml (deflated 38%) 2024-04-03T18:46:16.5168927Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-1f3904edeb67620e.xml (deflated 38%) 2024-04-03T18:46:16.5170718Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-b0b14be3012f5fc7.xml (deflated 37%) 2024-04-03T18:46:16.5172495Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-32bf4f628b66e5e7.xml (deflated 59%) 2024-04-03T18:46:16.5174272Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-524e5c860f7b9ca0.xml (deflated 59%) 2024-04-03T18:46:16.5176063Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-9e0895acb1c73c60.xml (deflated 59%) 2024-04-03T18:46:16.5177838Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e7d66c94ec85b975.xml (deflated 59%) 2024-04-03T18:46:16.5179629Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-0c6e824dd00eba7f.xml (deflated 38%) 2024-04-03T18:46:16.5181385Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-b1ee6227008bedcd.xml (deflated 37%) 2024-04-03T18:46:16.5183158Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-062861c33f31bf25.xml (deflated 37%) 2024-04-03T18:46:16.5185392Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_sharded_tensor/distributed._shard.sharded_tensor.test_sharded_tensor-0aa48676e9587f63.xml (deflated 93%) 2024-04-03T18:46:16.5188032Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-40e7b0a043a4d53c.xml (deflated 38%) 2024-04-03T18:46:16.5189925Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-42c71a28b72918f4.xml (deflated 34%) 2024-04-03T18:46:16.5191814Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-05dcf348c84e5e21.xml (deflated 34%) 2024-04-03T18:46:16.5193654Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fd650b230883ac56.xml (deflated 34%) 2024-04-03T18:46:16.5195547Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d8bfd485d6751309.xml (deflated 34%) 2024-04-03T18:46:16.5197455Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ad1e1419e88fab7b.xml (deflated 34%) 2024-04-03T18:46:16.5199344Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-47debacb64f43384.xml (deflated 35%) 2024-04-03T18:46:16.5201244Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-87f0b8a3c61c5de0.xml (deflated 34%) 2024-04-03T18:46:16.5203097Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1e1421a38c7b1ed3.xml (deflated 34%) 2024-04-03T18:46:16.5204990Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-44dc2a78d5624790.xml (deflated 35%) 2024-04-03T18:46:16.5206865Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-445a68f56bbb4d9b.xml (deflated 35%) 2024-04-03T18:46:16.5209018Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-38cdb800a1a28704.xml (deflated 36%) 2024-04-03T18:46:16.5210864Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f7afc16360095be3.xml (deflated 35%) 2024-04-03T18:46:16.5212773Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d2053c8512e988ff.xml (deflated 34%) 2024-04-03T18:46:16.5214671Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fea08b78856089ea.xml (deflated 35%) 2024-04-03T18:46:16.5216535Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1fb3b552d985b9a1.xml (deflated 34%) 2024-04-03T18:46:16.5218390Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8f88a47dd736809d.xml (deflated 34%) 2024-04-03T18:46:16.5220322Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8ef306677948af5f.xml (deflated 35%) 2024-04-03T18:46:16.5222200Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-afe70ee9da1307df.xml (deflated 35%) 2024-04-03T18:46:16.5224072Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5b73a3fb5e73e305.xml (deflated 34%) 2024-04-03T18:46:16.5226422Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4bae7ad1ccc58b8a.xml (deflated 34%) 2024-04-03T18:46:16.5228325Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2cbf353f821f912f.xml (deflated 34%) 2024-04-03T18:46:16.5230205Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-23d9fd32e4f8ca7c.xml (deflated 36%) 2024-04-03T18:46:16.5232064Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9cb80b563da4e414.xml (deflated 36%) 2024-04-03T18:46:16.5233960Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1e9195658f8d90c7.xml (deflated 37%) 2024-04-03T18:46:16.5235828Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9571507c1e3b1b24.xml (deflated 37%) 2024-04-03T18:46:16.5237698Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-dbb2383a5af7358d.xml (deflated 37%) 2024-04-03T18:46:16.5239568Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c9989d73e49e06f5.xml (deflated 37%) 2024-04-03T18:46:16.5241447Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-742907ba85adf3a9.xml (deflated 36%) 2024-04-03T18:46:16.5243316Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-db90a867d0084461.xml (deflated 36%) 2024-04-03T18:46:16.5245196Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9371a05d74f89aff.xml (deflated 36%) 2024-04-03T18:46:16.5247056Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-eab6755b61816e90.xml (deflated 36%) 2024-04-03T18:46:16.5248948Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-bbbcba43ead3660d.xml (deflated 36%) 2024-04-03T18:46:16.5250841Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b2b8ede831e8c53f.xml (deflated 36%) 2024-04-03T18:46:16.5252703Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-664b7fe9640fe9fe.xml (deflated 36%) 2024-04-03T18:46:16.5254539Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0bdf24f1cf141589.xml (deflated 36%) 2024-04-03T18:46:16.5256561Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3cfbbf15d5c8db3b.xml (deflated 36%) 2024-04-03T18:46:16.5258596Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c6a1074069c53219.xml (deflated 37%) 2024-04-03T18:46:16.5260459Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0399c575b5111024.xml (deflated 37%) 2024-04-03T18:46:16.5262316Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fd347adcfd2fb202.xml (deflated 36%) 2024-04-03T18:46:16.5264202Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1900cdfc33f7354a.xml (deflated 37%) 2024-04-03T18:46:16.5266561Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-adc925fbd12e23c5.xml (deflated 36%) 2024-04-03T18:46:16.5268484Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-eb029139c183423a.xml (deflated 37%) 2024-04-03T18:46:16.5270357Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-27ddeaf266052280.xml (deflated 36%) 2024-04-03T18:46:16.5272227Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1c9c8eb623702f67.xml (deflated 36%) 2024-04-03T18:46:16.5274084Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b6d3c2ee42199993.xml (deflated 35%) 2024-04-03T18:46:16.5275987Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-97ffb43d062aa350.xml (deflated 37%) 2024-04-03T18:46:16.5277907Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-49f9364498a63c86.xml (deflated 35%) 2024-04-03T18:46:16.5279889Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3c5648e29fbb097c.xml (deflated 36%) 2024-04-03T18:46:16.5281813Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b0af1d638670ab57.xml (deflated 44%) 2024-04-03T18:46:16.5283681Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-44bb57503f634260.xml (deflated 38%) 2024-04-03T18:46:16.5285544Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d2cb517c511695d1.xml (deflated 37%) 2024-04-03T18:46:16.5287418Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-71199f09a4919f64.xml (deflated 36%) 2024-04-03T18:46:16.5289281Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-35a2d28b859e59ae.xml (deflated 37%) 2024-04-03T18:46:16.5291157Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-14e4c7a9a00612f5.xml (deflated 44%) 2024-04-03T18:46:16.5293027Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c655dac4d9913986.xml (deflated 44%) 2024-04-03T18:46:16.5294888Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b258593d24a3f9da.xml (deflated 36%) 2024-04-03T18:46:16.5296783Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1c4c99cc2845f21f.xml (deflated 37%) 2024-04-03T18:46:16.5298627Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-231b3cf3b355406d.xml (deflated 38%) 2024-04-03T18:46:16.5300509Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-66964319bc8aa7e7.xml (deflated 36%) 2024-04-03T18:46:16.5302367Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6d81089c117f3d65.xml (deflated 37%) 2024-04-03T18:46:16.5304246Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5037f1dc13707ffe.xml (deflated 36%) 2024-04-03T18:46:16.5306826Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5da16fdf4454cf5e.xml (deflated 36%) 2024-04-03T18:46:16.5308748Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1aaf271aadc25873.xml (deflated 35%) 2024-04-03T18:46:16.5310625Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-04ff31939bcfe8e2.xml (deflated 36%) 2024-04-03T18:46:16.5312499Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e18e77a98ed8f297.xml (deflated 36%) 2024-04-03T18:46:16.5314354Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5234251d6dc742bf.xml (deflated 34%) 2024-04-03T18:46:16.5316217Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1478f27b35844c30.xml (deflated 35%) 2024-04-03T18:46:16.5318115Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-bfe1f2ee17321c06.xml (deflated 35%) 2024-04-03T18:46:16.5320037Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-04b9d8c346b22721.xml (deflated 35%) 2024-04-03T18:46:16.5321893Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3c720b950c121d47.xml (deflated 43%) 2024-04-03T18:46:16.5323758Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-70c1079e68f4ac25.xml (deflated 48%) 2024-04-03T18:46:16.5325606Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6f0a8ab3267cc2a1.xml (deflated 43%) 2024-04-03T18:46:16.5327516Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b8fb7549d5e37223.xml (deflated 44%) 2024-04-03T18:46:16.5329373Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2a2c5de0dbc55354.xml (deflated 36%) 2024-04-03T18:46:16.5331259Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-42dc02d568761d0f.xml (deflated 37%) 2024-04-03T18:46:16.5333139Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f774feca370c8acd.xml (deflated 36%) 2024-04-03T18:46:16.5335028Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6b9dd31718d57c05.xml (deflated 36%) 2024-04-03T18:46:16.5336881Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-272d9dd04433020f.xml (deflated 36%) 2024-04-03T18:46:16.5338754Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c3a83685247730aa.xml (deflated 36%) 2024-04-03T18:46:16.5340628Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2aff2a6c72632e44.xml (deflated 36%) 2024-04-03T18:46:16.5342504Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d9c59a9d65362bb4.xml (deflated 36%) 2024-04-03T18:46:16.5344382Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6f840c618cf69eae.xml (deflated 35%) 2024-04-03T18:46:16.5346767Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a17d7bf3c1eba7f2.xml (deflated 35%) 2024-04-03T18:46:16.5348636Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9c26b39d86e89d34.xml (deflated 36%) 2024-04-03T18:46:16.5350519Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f467f751d39e5324.xml (deflated 35%) 2024-04-03T18:46:16.5352397Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-aeade740c9f27992.xml (deflated 35%) 2024-04-03T18:46:16.5354413Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0f1b97cf70ac1e5d.xml (deflated 36%) 2024-04-03T18:46:16.5356461Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e47068552b73b287.xml (deflated 35%) 2024-04-03T18:46:16.5358344Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d9d9919c65d0c9cd.xml (deflated 35%) 2024-04-03T18:46:16.5360217Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b4c40c4bf167f088.xml (deflated 35%) 2024-04-03T18:46:16.5362090Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-86d62130a81d1027.xml (deflated 35%) 2024-04-03T18:46:16.5363965Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b659fe0ad8ea8b23.xml (deflated 34%) 2024-04-03T18:46:16.5365826Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3e1118940a7a4b63.xml (deflated 34%) 2024-04-03T18:46:16.5367701Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9a1278de26925bc6.xml (deflated 34%) 2024-04-03T18:46:16.5369586Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9fbc28cb8330aec2.xml (deflated 35%) 2024-04-03T18:46:16.5371472Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-26daf6c4056bf402.xml (deflated 43%) 2024-04-03T18:46:16.5373323Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e75cc574ff912606.xml (deflated 43%) 2024-04-03T18:46:16.5375206Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fea6d566f14a0976.xml (deflated 43%) 2024-04-03T18:46:16.5377080Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-48714a7a0bf82c02.xml (deflated 43%) 2024-04-03T18:46:16.5378939Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2b64001a59c6aef7.xml (deflated 43%) 2024-04-03T18:46:16.5380822Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7623fe061a3e73d0.xml (deflated 42%) 2024-04-03T18:46:16.5382692Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-bb41235138af2f03.xml (deflated 43%) 2024-04-03T18:46:16.5384563Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3fd7b646a0dc8b5c.xml (deflated 43%) 2024-04-03T18:46:16.5386909Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ab848f1b26218e55.xml (deflated 42%) 2024-04-03T18:46:16.5388790Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-74cd9a0e9884dd6a.xml (deflated 36%) 2024-04-03T18:46:16.5390674Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b04b7bf29a7916d7.xml (deflated 34%) 2024-04-03T18:46:16.5392559Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1eff43886fb50805.xml (deflated 34%) 2024-04-03T18:46:16.5394379Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b42c425e17860f23.xml (deflated 35%) 2024-04-03T18:46:16.5396286Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b963c9a27449849c.xml (deflated 35%) 2024-04-03T18:46:16.5398163Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-792e990d4dbe5340.xml (deflated 35%) 2024-04-03T18:46:16.5400038Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d4b10cf1d1c41b38.xml (deflated 34%) 2024-04-03T18:46:16.5401897Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-abedb82644945e36.xml (deflated 34%) 2024-04-03T18:46:16.5404031Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4d81a4c71d5c07c4.xml (deflated 34%) 2024-04-03T18:46:16.5405930Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5b46dd46c8b0359e.xml (deflated 35%) 2024-04-03T18:46:16.5407781Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-581c613dab79eef8.xml (deflated 35%) 2024-04-03T18:46:16.5409757Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cfd5da86fad062b1.xml (deflated 58%) 2024-04-03T18:46:16.5411809Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-546dda12fc4d2561.xml (deflated 37%) 2024-04-03T18:46:16.5413848Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7b4ab24e123a99c3.xml (deflated 37%) 2024-04-03T18:46:16.5415906Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3be89d765fe911c3.xml (deflated 38%) 2024-04-03T18:46:16.5417934Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c3834d0aa7b79a15.xml (deflated 43%) 2024-04-03T18:46:16.5420004Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70321de64d604aad.xml (deflated 38%) 2024-04-03T18:46:16.5422290Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ac186adc78f231ac.xml (deflated 38%) 2024-04-03T18:46:16.5424319Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3f042a3bb172a87e.xml (deflated 37%) 2024-04-03T18:46:16.5426729Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-09fd8d3c0f77b24e.xml (deflated 49%) 2024-04-03T18:46:16.5428839Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-40eda4a69a4af21b.xml (deflated 44%) 2024-04-03T18:46:16.5430881Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-04af2dc85dafe47e.xml (deflated 37%) 2024-04-03T18:46:16.5432978Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a294679e62007e05.xml (deflated 37%) 2024-04-03T18:46:16.5435039Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7bb31df009485d7b.xml (deflated 37%) 2024-04-03T18:46:16.5437071Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7fbc1ce0871c797c.xml (deflated 37%) 2024-04-03T18:46:16.5439152Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-11c996fd0cc65859.xml (deflated 38%) 2024-04-03T18:46:16.5441169Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4fd13bc9d4947e89.xml (deflated 37%) 2024-04-03T18:46:16.5443237Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-49604819908eaf18.xml (deflated 38%) 2024-04-03T18:46:16.5445219Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2bc55170f96f2dc3.xml (deflated 37%) 2024-04-03T18:46:16.5447248Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5ece11912da1903.xml (deflated 37%) 2024-04-03T18:46:16.5449293Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-25e447b3f2d22122.xml (deflated 38%) 2024-04-03T18:46:16.5451325Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3087e864510345ac.xml (deflated 37%) 2024-04-03T18:46:16.5453609Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ba87b3cab524273.xml (deflated 37%) 2024-04-03T18:46:16.5455677Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a9dea00acd51b13.xml (deflated 47%) 2024-04-03T18:46:16.5457741Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3728239d68c94325.xml (deflated 37%) 2024-04-03T18:46:16.5459744Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e8105436b528a7af.xml (deflated 37%) 2024-04-03T18:46:16.5461758Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96382c3c51e126f2.xml (deflated 38%) 2024-04-03T18:46:16.5463798Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd58a4b1d9b74d01.xml (deflated 38%) 2024-04-03T18:46:16.5466296Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-736c19fb741f5cf8.xml (deflated 37%) 2024-04-03T18:46:16.5468374Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6755f889d63ad4ff.xml (deflated 45%) 2024-04-03T18:46:16.5470418Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eaa3c1183cd3a86b.xml (deflated 44%) 2024-04-03T18:46:16.5472444Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-85b854bbf9436021.xml (deflated 37%) 2024-04-03T18:46:16.5474476Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9178b8fe7009e7e2.xml (deflated 43%) 2024-04-03T18:46:16.5476523Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5902a405ed728cf4.xml (deflated 43%) 2024-04-03T18:46:16.5478567Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6bdd2fee6167bd7c.xml (deflated 38%) 2024-04-03T18:46:16.5480619Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bffeed23c601df22.xml (deflated 46%) 2024-04-03T18:46:16.5482698Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1a17a071b891bcd.xml (deflated 43%) 2024-04-03T18:46:16.5484695Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6b00134215974140.xml (deflated 37%) 2024-04-03T18:46:16.5486713Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b9e74103187f26e3.xml (deflated 37%) 2024-04-03T18:46:16.5488754Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d4c5cf8124796834.xml (deflated 58%) 2024-04-03T18:46:16.5490800Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-216be37fb82dc797.xml (deflated 37%) 2024-04-03T18:46:16.5492845Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-50d6e483223dbcf9.xml (deflated 37%) 2024-04-03T18:46:16.5494892Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-295c730ba3197b1e.xml (deflated 37%) 2024-04-03T18:46:16.5496912Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ffd2d49c5a70458.xml (deflated 43%) 2024-04-03T18:46:16.5498949Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-74391a61b19f34cf.xml (deflated 37%) 2024-04-03T18:46:16.5501119Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f80cd86f91166ffb.xml (deflated 38%) 2024-04-03T18:46:16.5503272Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1ff0b79e4e806ba2.xml (deflated 37%) 2024-04-03T18:46:16.5505324Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b31aa7f41349276a.xml (deflated 49%) 2024-04-03T18:46:16.5507816Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-382c993c2e7aa2af.xml (deflated 44%) 2024-04-03T18:46:16.5509842Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ba10882f53f0a607.xml (deflated 37%) 2024-04-03T18:46:16.5511887Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3204bc5dbcf7f249.xml (deflated 37%) 2024-04-03T18:46:16.5514155Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e433c74fb416186b.xml (deflated 38%) 2024-04-03T18:46:16.5516226Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-64c4dfd972eec5c6.xml (deflated 37%) 2024-04-03T18:46:16.5518546Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17b20fb24c29244e.xml (deflated 38%) 2024-04-03T18:46:16.5520603Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e1e1de3d8aad300.xml (deflated 37%) 2024-04-03T18:46:16.5522647Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eaf888027ce8dde6.xml (deflated 38%) 2024-04-03T18:46:16.5524695Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dedd58f301322f69.xml (deflated 37%) 2024-04-03T18:46:16.5526744Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3cb10b8437465b7b.xml (deflated 37%) 2024-04-03T18:46:16.5528791Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-328942e3e092c01a.xml (deflated 38%) 2024-04-03T18:46:16.5530831Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0cbaeac32462e868.xml (deflated 38%) 2024-04-03T18:46:16.5532882Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9da75b7464b7ba27.xml (deflated 38%) 2024-04-03T18:46:16.5534909Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf47f5eac509749d.xml (deflated 47%) 2024-04-03T18:46:16.5536963Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1d1bfbe4f0dca792.xml (deflated 37%) 2024-04-03T18:46:16.5539021Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9c40a17c76239f76.xml (deflated 37%) 2024-04-03T18:46:16.5541060Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2fd4af645129a0cc.xml (deflated 38%) 2024-04-03T18:46:16.5543082Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-45e5ce6503935046.xml (deflated 38%) 2024-04-03T18:46:16.5545120Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-86fc9e0d732c7fd9.xml (deflated 37%) 2024-04-03T18:46:16.5547615Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ab14af3d75ab7190.xml (deflated 45%) 2024-04-03T18:46:16.5549661Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-897218038517fa62.xml (deflated 44%) 2024-04-03T18:46:16.5551987Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-092ab9df5bc0a0db.xml (deflated 36%) 2024-04-03T18:46:16.5554055Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fa3716233a5ecd44.xml (deflated 43%) 2024-04-03T18:46:16.5556101Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b2ddd54bdac6e889.xml (deflated 43%) 2024-04-03T18:46:16.5558156Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-52b0bfcb88a0b91f.xml (deflated 37%) 2024-04-03T18:46:16.5560185Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-afc9d3ae96687e48.xml (deflated 46%) 2024-04-03T18:46:16.5562229Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a65a922d89d5274.xml (deflated 44%) 2024-04-03T18:46:16.5564293Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d7d02bbdd4d84d8e.xml (deflated 37%) 2024-04-03T18:46:16.5566347Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d7257965cc6ab50d.xml (deflated 37%) 2024-04-03T18:46:16.5568368Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-79a41fc65398c06b.xml (deflated 44%) 2024-04-03T18:46:16.5570417Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5bf93953fe1c08e.xml (deflated 38%) 2024-04-03T18:46:16.5572463Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1f07bf0dc1e6e78.xml (deflated 38%) 2024-04-03T18:46:16.5574505Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4851a8db9391a03.xml (deflated 46%) 2024-04-03T18:46:16.5576541Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d5a6fcc872732b8e.xml (deflated 46%) 2024-04-03T18:46:16.5579084Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b17a4315dab32205.xml (deflated 45%) 2024-04-03T18:46:16.5581176Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f68b4213fefdaba7.xml (deflated 37%) 2024-04-03T18:46:16.5583214Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6e62ada39b253488.xml (deflated 43%) 2024-04-03T18:46:16.5585233Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-703cf7e09618b0ff.xml (deflated 43%) 2024-04-03T18:46:16.5587731Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f1aaee3bdb86613.xml (deflated 43%) 2024-04-03T18:46:16.5589802Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a6b89ea538755464.xml (deflated 44%) 2024-04-03T18:46:16.5591847Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c57ae746950b02d9.xml (deflated 43%) 2024-04-03T18:46:16.5593872Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0f2eeb4eb2a405bd.xml (deflated 43%) 2024-04-03T18:46:16.5595913Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-09d485c864a451d5.xml (deflated 44%) 2024-04-03T18:46:16.5597960Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cc4e46cf9f2c164c.xml (deflated 37%) 2024-04-03T18:46:16.5599998Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5fe428c6c8156455.xml (deflated 45%) 2024-04-03T18:46:16.5602543Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-58db31e7416b2be1.xml (deflated 37%) 2024-04-03T18:46:16.5604989Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-85093348298eb65e.xml (deflated 49%) 2024-04-03T18:46:16.5607437Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ed524ea5c212803d.xml (deflated 44%) 2024-04-03T18:46:16.5609747Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-50faddf0dee698f2.xml (deflated 58%) 2024-04-03T18:46:16.5611896Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b25d6b8d78047ea6.xml (deflated 49%) 2024-04-03T18:46:16.5614236Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f432540e9d7b39cf.xml (deflated 44%) 2024-04-03T18:46:16.5616715Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4861c4be1bcdb240.xml (deflated 42%) 2024-04-03T18:46:16.5619151Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ed3956564647fd8f.xml (deflated 38%) 2024-04-03T18:46:16.5621202Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-661dc323a182d292.xml (deflated 38%) 2024-04-03T18:46:16.5623252Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b8ae323f858f42ef.xml (deflated 37%) 2024-04-03T18:46:16.5625297Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2bc60ebfd646ed46.xml (deflated 49%) 2024-04-03T18:46:16.5627888Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d7d5f32a40b6b113.xml (deflated 37%) 2024-04-03T18:46:16.5629924Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c168948d24a6007.xml (deflated 43%) 2024-04-03T18:46:16.5631968Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ced1d8a3170ddd14.xml (deflated 37%) 2024-04-03T18:46:16.5634012Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a846f0bd24704916.xml (deflated 43%) 2024-04-03T18:46:16.5636047Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3464c8c448a9586a.xml (deflated 38%) 2024-04-03T18:46:16.5638059Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-514977a82c0f5f7d.xml (deflated 37%) 2024-04-03T18:46:16.5640121Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fef89317b402b96c.xml (deflated 43%) 2024-04-03T18:46:16.5642175Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c71ce5559fb7985c.xml (deflated 37%) 2024-04-03T18:46:16.5644225Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2e0bad6c83cd9ae6.xml (deflated 38%) 2024-04-03T18:46:16.5646253Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6661a4da06a83db1.xml (deflated 38%) 2024-04-03T18:46:16.5648350Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b377955516a50230.xml (deflated 38%) 2024-04-03T18:46:16.5650384Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6796d2aea4732d86.xml (deflated 44%) 2024-04-03T18:46:16.5652567Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b12cf07e0efae34c.xml (deflated 43%) 2024-04-03T18:46:16.5654750Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-60c0a92745cb8efc.xml (deflated 38%) 2024-04-03T18:46:16.5656792Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f5f9a0d66e14df63.xml (deflated 44%) 2024-04-03T18:46:16.5658835Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b40f784eb33b5876.xml (deflated 38%) 2024-04-03T18:46:16.5660873Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b92abdf2c97bf8af.xml (deflated 38%) 2024-04-03T18:46:16.5662891Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e73822928224143f.xml (deflated 46%) 2024-04-03T18:46:16.5664931Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b8f8ca2d9040a824.xml (deflated 46%) 2024-04-03T18:46:16.5667215Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-027d0ec7b70b61a8.xml (deflated 46%) 2024-04-03T18:46:16.5669709Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-db9e69d7e02d4d4c.xml (deflated 37%) 2024-04-03T18:46:16.5672091Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ad2d76e8db0738e1.xml (deflated 43%) 2024-04-03T18:46:16.5674567Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-056a621fad98ffdf.xml (deflated 43%) 2024-04-03T18:46:16.5677008Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1dab8432bfff332d.xml (deflated 43%) 2024-04-03T18:46:16.5679293Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b6749c26b6b9139c.xml (deflated 44%) 2024-04-03T18:46:16.5681537Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bb0f5b0c73cdb83f.xml (deflated 43%) 2024-04-03T18:46:16.5683589Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c2a533fd1221f22e.xml (deflated 43%) 2024-04-03T18:46:16.5685631Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f040f0adb5965ba.xml (deflated 44%) 2024-04-03T18:46:16.5687675Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e2757b0b695a2ec1.xml (deflated 37%) 2024-04-03T18:46:16.5689690Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-04c9216df832b6d8.xml (deflated 44%) 2024-04-03T18:46:16.5691760Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc4b5f5eaf0b4c9c.xml (deflated 37%) 2024-04-03T18:46:16.5693811Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8bf683996eb153b1.xml (deflated 49%) 2024-04-03T18:46:16.5695854Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5bd3cd7498a4fafb.xml (deflated 44%) 2024-04-03T18:46:16.5697887Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-46b563bf8afaa7a5.xml (deflated 58%) 2024-04-03T18:46:16.5699933Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0d9c83e677f06cb4.xml (deflated 49%) 2024-04-03T18:46:16.5701974Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d5656d63380b87e5.xml (deflated 44%) 2024-04-03T18:46:16.5704305Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96ed6fb4dffd9bfe.xml (deflated 42%) 2024-04-03T18:46:16.5706551Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e41d2aa1e2470e2c.xml (deflated 37%) 2024-04-03T18:46:16.5708599Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7512d9905387b946.xml (deflated 37%) 2024-04-03T18:46:16.5710637Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-47041b1363c6253d.xml (deflated 37%) 2024-04-03T18:46:16.5712670Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b1e62239bbe4416f.xml (deflated 49%) 2024-04-03T18:46:16.5714681Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24830d825579d448.xml (deflated 37%) 2024-04-03T18:46:16.5716740Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dea7248d4e609917.xml (deflated 43%) 2024-04-03T18:46:16.5718815Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bab80f108da013a9.xml (deflated 37%) 2024-04-03T18:46:16.5720837Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2422f156f6ea7f48.xml (deflated 43%) 2024-04-03T18:46:16.5722868Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e7a944287e88731.xml (deflated 38%) 2024-04-03T18:46:16.5724902Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3a8676ed4648f223.xml (deflated 37%) 2024-04-03T18:46:16.5726929Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-55d103025a538bca.xml (deflated 43%) 2024-04-03T18:46:16.5728958Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7382591d6198e3c1.xml (deflated 37%) 2024-04-03T18:46:16.5730999Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-edbd567f4be50a39.xml (deflated 38%) 2024-04-03T18:46:16.5733097Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3da8c28960dd951c.xml (deflated 38%) 2024-04-03T18:46:16.5735142Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bad7b6518357ff17.xml (deflated 38%) 2024-04-03T18:46:16.5737183Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a74225fbd45bed78.xml (deflated 43%) 2024-04-03T18:46:16.5739206Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ad1d7a793cf8eb04.xml (deflated 43%) 2024-04-03T18:46:16.5741257Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-633967d78be7538b.xml (deflated 37%) 2024-04-03T18:46:16.5743300Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89e2a5ec935db3b4.xml (deflated 44%) 2024-04-03T18:46:16.5745320Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f82a591e767398cd.xml (deflated 38%) 2024-04-03T18:46:16.5747528Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d5b0bd05074038d7.xml (deflated 38%) 2024-04-03T18:46:16.5749573Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d8112f2c8e119bc5.xml (deflated 37%) 2024-04-03T18:46:16.5751610Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a8b925261601e47d.xml (deflated 38%) 2024-04-03T18:46:16.5753901Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-df579f4710cf29ec.xml (deflated 46%) 2024-04-03T18:46:16.5755930Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a4ced4ed636c80b9.xml (deflated 43%) 2024-04-03T18:46:16.5757969Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-144fe17ee452186b.xml (deflated 44%) 2024-04-03T18:46:16.5760008Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-87be2a829e895d0d.xml (deflated 44%) 2024-04-03T18:46:16.5762035Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4240fff6f5043aba.xml (deflated 44%) 2024-04-03T18:46:16.5764073Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-851570db481eb69a.xml (deflated 43%) 2024-04-03T18:46:16.5766124Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34bf643124da9b0c.xml (deflated 38%) 2024-04-03T18:46:16.5768160Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0cf1a71517364bf3.xml (deflated 49%) 2024-04-03T18:46:16.5770179Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5f821249affa80c.xml (deflated 37%) 2024-04-03T18:46:16.5772219Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-38e020924e6ff807.xml (deflated 37%) 2024-04-03T18:46:16.5774255Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ccf09b3a957df932.xml (deflated 37%) 2024-04-03T18:46:16.5776306Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca34b62335557be6.xml (deflated 37%) 2024-04-03T18:46:16.5778337Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c0e023ba20f6a1ea.xml (deflated 38%) 2024-04-03T18:46:16.5780384Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-42cf47b6ccdc16ff.xml (deflated 38%) 2024-04-03T18:46:16.5782428Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-954917bb4f03d814.xml (deflated 37%) 2024-04-03T18:46:16.5784467Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e8dc3af7e5c6d4c2.xml (deflated 38%) 2024-04-03T18:46:16.5786807Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51dc42687889e199.xml (deflated 38%) 2024-04-03T18:46:16.5788852Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-04517bfaf85b66fe.xml (deflated 38%) 2024-04-03T18:46:16.5790900Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-08593359ed90f042.xml (deflated 38%) 2024-04-03T18:46:16.5792940Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fc2ce4367eae7b2d.xml (deflated 44%) 2024-04-03T18:46:16.5794970Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af040bb5ff9fda8a.xml (deflated 38%) 2024-04-03T18:46:16.5797005Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30836c5322db493e.xml (deflated 43%) 2024-04-03T18:46:16.5799039Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e73214dddf907d28.xml (deflated 37%) 2024-04-03T18:46:16.5801208Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0328c1b06acafabc.xml (deflated 43%) 2024-04-03T18:46:16.5803361Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ebe5fbf27c5393ba.xml (deflated 47%) 2024-04-03T18:46:16.5805409Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-735afebcce8b9784.xml (deflated 43%) 2024-04-03T18:46:16.5807454Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0271daec69d4cbcd.xml (deflated 38%) 2024-04-03T18:46:16.5809504Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-db9b4b7f0a8b474f.xml (deflated 45%) 2024-04-03T18:46:16.5811533Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d3d6ffb1d943e3b7.xml (deflated 38%) 2024-04-03T18:46:16.5813577Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c88f570e1733124.xml (deflated 44%) 2024-04-03T18:46:16.5815628Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb51d41cab838ed7.xml (deflated 38%) 2024-04-03T18:46:16.5817671Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ef7329da1a334a25.xml (deflated 38%) 2024-04-03T18:46:16.5819745Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-216a9dafde7839be.xml (deflated 37%) 2024-04-03T18:46:16.5821788Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e7cf55058bc0eb80.xml (deflated 38%) 2024-04-03T18:46:16.5823827Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-007f3d99c339d78d.xml (deflated 46%) 2024-04-03T18:46:16.5826053Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-78be81487ad462b5.xml (deflated 43%) 2024-04-03T18:46:16.5828094Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b16f7dea031b53f6.xml (deflated 44%) 2024-04-03T18:46:16.5830143Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2a1b60c5b6e6af7.xml (deflated 44%) 2024-04-03T18:46:16.5832191Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2dc77d0f6cc8af2f.xml (deflated 44%) 2024-04-03T18:46:16.5834237Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c4791e0735416c1.xml (deflated 43%) 2024-04-03T18:46:16.5836257Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fef683c26f8d1b04.xml (deflated 38%) 2024-04-03T18:46:16.5838317Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5b5dd0ada386a107.xml (deflated 49%) 2024-04-03T18:46:16.5840362Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cd5372ea5b24046f.xml (deflated 38%) 2024-04-03T18:46:16.5842387Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2ab3256837022803.xml (deflated 37%) 2024-04-03T18:46:16.5844399Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d86d8932a82f47d.xml (deflated 37%) 2024-04-03T18:46:16.5846447Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cdd195ba6c8c89e4.xml (deflated 37%) 2024-04-03T18:46:16.5848494Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-627fbf730b898bf0.xml (deflated 38%) 2024-04-03T18:46:16.5850796Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d20ee8a382c23fdc.xml (deflated 38%) 2024-04-03T18:46:16.5852839Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-905061cfe9a60e88.xml (deflated 37%) 2024-04-03T18:46:16.5854874Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-57886b5b122a149b.xml (deflated 38%) 2024-04-03T18:46:16.5856914Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a000b35148df35e1.xml (deflated 39%) 2024-04-03T18:46:16.5858956Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a032e4530cc11ee1.xml (deflated 38%) 2024-04-03T18:46:16.5860979Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ce866d9edf85827c.xml (deflated 38%) 2024-04-03T18:46:16.5863039Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-713b84492b6479f6.xml (deflated 44%) 2024-04-03T18:46:16.5865085Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4b7b7655bba8245c.xml (deflated 38%) 2024-04-03T18:46:16.5867301Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-05a5f554fc112ad8.xml (deflated 43%) 2024-04-03T18:46:16.5869328Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-117e602e6e9198e0.xml (deflated 38%) 2024-04-03T18:46:16.5871365Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4041b799ba8383c5.xml (deflated 43%) 2024-04-03T18:46:16.5873402Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0805c7edc21b336f.xml (deflated 47%) 2024-04-03T18:46:16.5875461Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1cd0b740cfdbb73.xml (deflated 43%) 2024-04-03T18:46:16.5877486Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-99e521d2c83c6bf5.xml (deflated 38%) 2024-04-03T18:46:16.5879535Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b18ade6d54a18a4.xml (deflated 45%) 2024-04-03T18:46:16.5881577Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-639550190004ef81.xml (deflated 38%) 2024-04-03T18:46:16.5883967Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-5ab534df133e6148.xml (deflated 49%) 2024-04-03T18:46:16.5886663Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-fb993d02402a939c.xml (deflated 49%) 2024-04-03T18:46:16.5889387Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-cd5fefd494b4f044.xml (deflated 39%) 2024-04-03T18:46:16.5892098Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-d35611a3b0525ae9.xml (deflated 39%) 2024-04-03T18:46:16.5894805Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-ee50f8425e3a1958.xml (deflated 39%) 2024-04-03T18:46:16.5897494Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-ff79f60925b004f9.xml (deflated 39%) 2024-04-03T18:46:16.5900340Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-84da9e60a49a3b77.xml (deflated 49%) 2024-04-03T18:46:16.5903170Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-589faf6c43c999dc.xml (deflated 49%) 2024-04-03T18:46:16.5906065Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-ce4845657b607dc4.xml (deflated 39%) 2024-04-03T18:46:16.5908763Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-95348a359756643a.xml (deflated 39%) 2024-04-03T18:46:16.5911442Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-932aa8a43ce4e5a0.xml (deflated 39%) 2024-04-03T18:46:16.5914150Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-bb241c49079bcca9.xml (deflated 39%) 2024-04-03T18:46:16.5916525Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2762012eedf5a0c7.xml (deflated 58%) 2024-04-03T18:46:16.5918574Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bbf34142b061b1f9.xml (deflated 37%) 2024-04-03T18:46:16.5920648Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fbb115eca5a1ec05.xml (deflated 43%) 2024-04-03T18:46:16.5922703Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-92c1584a5beff0d1.xml (deflated 44%) 2024-04-03T18:46:16.5924754Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b2d80f1d1daa9897.xml (deflated 37%) 2024-04-03T18:46:16.5926826Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3d6e3e46a6b95aeb.xml (deflated 38%) 2024-04-03T18:46:16.5928861Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3b0bfd4cfea287fa.xml (deflated 43%) 2024-04-03T18:46:16.5930915Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-278b5fc9a17186ba.xml (deflated 43%) 2024-04-03T18:46:16.5932961Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e7b84ddb2c830efb.xml (deflated 44%) 2024-04-03T18:46:16.5935016Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-570fac3aa315c2a0.xml (deflated 49%) 2024-04-03T18:46:16.5937038Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2da97453a906c20e.xml (deflated 42%) 2024-04-03T18:46:16.5939095Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-20128205b75b4fed.xml (deflated 47%) 2024-04-03T18:46:16.5941138Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-71a967bffd40ed39.xml (deflated 38%) 2024-04-03T18:46:16.5943205Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-559436e4ff35a2ae.xml (deflated 37%) 2024-04-03T18:46:16.5945205Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b40ab3591844b057.xml (deflated 38%) 2024-04-03T18:46:16.5947459Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-48b5d6999138e4ab.xml (deflated 38%) 2024-04-03T18:46:16.5949481Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-849b0ba8150c49ac.xml (deflated 38%) 2024-04-03T18:46:16.5951767Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-beb9fc75325e45d0.xml (deflated 37%) 2024-04-03T18:46:16.5953812Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ebaea020b75ae83c.xml (deflated 38%) 2024-04-03T18:46:16.5955858Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9d85c10f1366e1ab.xml (deflated 38%) 2024-04-03T18:46:16.5957901Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4cbc2e9cf9a6280a.xml (deflated 38%) 2024-04-03T18:46:16.5959938Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8b7900e4328be42d.xml (deflated 38%) 2024-04-03T18:46:16.5961960Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-be3ad838890bf944.xml (deflated 47%) 2024-04-03T18:46:16.5964022Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-217f69d05d2d1400.xml (deflated 37%) 2024-04-03T18:46:16.5966062Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8ebbf121b55f5d68.xml (deflated 38%) 2024-04-03T18:46:16.5968109Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-77ea6028eb8b1a6c.xml (deflated 37%) 2024-04-03T18:46:16.5970137Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cacdbfcfc8f85fc0.xml (deflated 38%) 2024-04-03T18:46:16.5972196Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1402cfbd9968035a.xml (deflated 37%) 2024-04-03T18:46:16.5974235Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-09c63be8802a308a.xml (deflated 49%) 2024-04-03T18:46:16.5976288Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a245f973ec075e65.xml (deflated 48%) 2024-04-03T18:46:16.5978315Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-59d24cf4985e1aba.xml (deflated 37%) 2024-04-03T18:46:16.5980359Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e18ee6e2193c31c.xml (deflated 49%) 2024-04-03T18:46:16.5982406Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8fa510fb1a8826a4.xml (deflated 37%) 2024-04-03T18:46:16.5984450Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-27be235c8f19da35.xml (deflated 42%) 2024-04-03T18:46:16.5986649Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d0d9114407d1de45.xml (deflated 37%) 2024-04-03T18:46:16.5988701Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-136ce3516730fd59.xml (deflated 37%) 2024-04-03T18:46:16.5990739Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a5129108a9c8c096.xml (deflated 38%) 2024-04-03T18:46:16.5992776Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7dd5d7a256539ae0.xml (deflated 37%) 2024-04-03T18:46:16.5994800Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dbf628318f56281d.xml (deflated 58%) 2024-04-03T18:46:16.5996840Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-838a2f858628547e.xml (deflated 37%) 2024-04-03T18:46:16.5998994Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15c4aa0b5372daa1.xml (deflated 43%) 2024-04-03T18:46:16.6001158Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d0e0cae6577e85ad.xml (deflated 44%) 2024-04-03T18:46:16.6003183Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4d91b6576f6bf57.xml (deflated 38%) 2024-04-03T18:46:16.6005224Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ac708e917e8ea57.xml (deflated 38%) 2024-04-03T18:46:16.6007274Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2ad6eabae9c8e960.xml (deflated 43%) 2024-04-03T18:46:16.6009301Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-541bbc9a4819b3d5.xml (deflated 43%) 2024-04-03T18:46:16.6011344Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b96392462ccaf573.xml (deflated 44%) 2024-04-03T18:46:16.6013395Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c7c4d286721d3a6.xml (deflated 49%) 2024-04-03T18:46:16.6015433Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9911a942426e6c46.xml (deflated 42%) 2024-04-03T18:46:16.6017449Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6a130b14afd14821.xml (deflated 47%) 2024-04-03T18:46:16.6019531Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c6b7ac4e6415325.xml (deflated 38%) 2024-04-03T18:46:16.6021577Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-50b308aa94cdd670.xml (deflated 37%) 2024-04-03T18:46:16.6023628Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1e64b8c798b37ed.xml (deflated 38%) 2024-04-03T18:46:16.6025807Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-71a5c3f8a7e787d0.xml (deflated 38%) 2024-04-03T18:46:16.6027868Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7bcf8c73dcb02927.xml (deflated 38%) 2024-04-03T18:46:16.6029910Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cd95a60622114183.xml (deflated 37%) 2024-04-03T18:46:16.6031949Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5b31ae276fb7c811.xml (deflated 37%) 2024-04-03T18:46:16.6033973Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b07d9f02bc8dfbb0.xml (deflated 39%) 2024-04-03T18:46:16.6036035Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-704d3d24a4cbcdaf.xml (deflated 38%) 2024-04-03T18:46:16.6038083Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8dd850bed154df8f.xml (deflated 38%) 2024-04-03T18:46:16.6040136Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd83eb04f24b90df.xml (deflated 47%) 2024-04-03T18:46:16.6042160Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3908ea35cf7a493.xml (deflated 37%) 2024-04-03T18:46:16.6044199Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c233d29f602f70de.xml (deflated 37%) 2024-04-03T18:46:16.6046239Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c2e966c25902edff.xml (deflated 37%) 2024-04-03T18:46:16.6048531Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3f31dce142084912.xml (deflated 38%) 2024-04-03T18:46:16.6050575Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1c28a2872e7ddf3.xml (deflated 37%) 2024-04-03T18:46:16.6052611Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-056d4422560ed609.xml (deflated 49%) 2024-04-03T18:46:16.6054652Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17e6f5d17a5ff81f.xml (deflated 49%) 2024-04-03T18:46:16.6056698Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ab9bbda405d64357.xml (deflated 37%) 2024-04-03T18:46:16.6058726Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-55305fb0cc618cc3.xml (deflated 49%) 2024-04-03T18:46:16.6060783Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c3c2b4d0d955d2e4.xml (deflated 38%) 2024-04-03T18:46:16.6062832Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b0d4d77825e39d4d.xml (deflated 43%) 2024-04-03T18:46:16.6064877Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9d8a621faf7d8515.xml (deflated 37%) 2024-04-03T18:46:16.6067144Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51365bfbec6e2f05.xml (deflated 37%) 2024-04-03T18:46:16.6069185Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d28b9a58395d58e7.xml (deflated 37%) 2024-04-03T18:46:16.6071229Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34b41e271bdd3a3c.xml (deflated 37%) 2024-04-03T18:46:16.6073292Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a7f4fcf47c9a74e0.xml (deflated 39%) 2024-04-03T18:46:16.6075320Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca9a95bc4a620d6f.xml (deflated 38%) 2024-04-03T18:46:16.6077368Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7b539c0eb54be277.xml (deflated 39%) 2024-04-03T18:46:16.6079407Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c66a37f28025bb3.xml (deflated 37%) 2024-04-03T18:46:16.6081452Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a16e2cdf6ff01a70.xml (deflated 37%) 2024-04-03T18:46:16.6083476Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2af7fabb7192432c.xml (deflated 37%) 2024-04-03T18:46:16.6085519Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f3710237920311bf.xml (deflated 45%) 2024-04-03T18:46:16.6087554Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-785c4fd3b008d12b.xml (deflated 49%) 2024-04-03T18:46:16.6089594Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4488b662a993a49.xml (deflated 49%) 2024-04-03T18:46:16.6091616Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-38d8c7fad05423a3.xml (deflated 38%) 2024-04-03T18:46:16.6093649Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-42c172e3a28e1b29.xml (deflated 44%) 2024-04-03T18:46:16.6095681Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-173076444962cdcd.xml (deflated 43%) 2024-04-03T18:46:16.6097953Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-54f83b5efc016e0b.xml (deflated 44%) 2024-04-03T18:46:16.6099995Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0f7cd020eb3667ee.xml (deflated 44%) 2024-04-03T18:46:16.6102038Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51976ba2957e2740.xml (deflated 44%) 2024-04-03T18:46:16.6104078Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2f2ffd651c065f66.xml (deflated 44%) 2024-04-03T18:46:16.6106279Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e458276b45e55c5.xml (deflated 44%) 2024-04-03T18:46:16.6108311Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2300292c6fd36c76.xml (deflated 45%) 2024-04-03T18:46:16.6110374Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e7da2a1161a01b96.xml (deflated 44%) 2024-04-03T18:46:16.6112422Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cde2c6beefea9b85.xml (deflated 58%) 2024-04-03T18:46:16.6114469Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c444c64e89017efd.xml (deflated 49%) 2024-04-03T18:46:16.6116483Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4835936bb79f357f.xml (deflated 49%) 2024-04-03T18:46:16.6118526Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7016b36dee2a6eca.xml (deflated 38%) 2024-04-03T18:46:16.6120618Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3f037cca036b7abc.xml (deflated 37%) 2024-04-03T18:46:16.6122678Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-438c08ea14c141fb.xml (deflated 38%) 2024-04-03T18:46:16.6124703Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d7f102183f3a5529.xml (deflated 37%) 2024-04-03T18:46:16.6126745Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-057f35e44317e052.xml (deflated 49%) 2024-04-03T18:46:16.6128786Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7626a125a3d89dc3.xml (deflated 37%) 2024-04-03T18:46:16.6130826Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5af9e2033d779dc1.xml (deflated 43%) 2024-04-03T18:46:16.6132840Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7c6a75160600b102.xml (deflated 37%) 2024-04-03T18:46:16.6134894Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d7b69bf9565b53a1.xml (deflated 37%) 2024-04-03T18:46:16.6136945Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1a4b9a80cb7d1baa.xml (deflated 37%) 2024-04-03T18:46:16.6138988Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-049682f458fefbdd.xml (deflated 37%) 2024-04-03T18:46:16.6141007Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b5f440eea6985f53.xml (deflated 37%) 2024-04-03T18:46:16.6143045Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4147950d5827964f.xml (deflated 37%) 2024-04-03T18:46:16.6145192Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1911e94ba5dcb231.xml (deflated 45%) 2024-04-03T18:46:16.6147516Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aecec9c95c3582fb.xml (deflated 43%) 2024-04-03T18:46:16.6149537Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-214f4e81405df607.xml (deflated 43%) 2024-04-03T18:46:16.6151579Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f12a78da0bb16f5f.xml (deflated 38%) 2024-04-03T18:46:16.6153624Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f52cf0e93f5e06ff.xml (deflated 37%) 2024-04-03T18:46:16.6155669Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0a2ddedc1535a395.xml (deflated 41%) 2024-04-03T18:46:16.6157703Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-36c34c97ac0e5b40.xml (deflated 38%) 2024-04-03T18:46:16.6159751Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-71ede68940d5f499.xml (deflated 38%) 2024-04-03T18:46:16.6161798Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-edb2b4f00f6c1d62.xml (deflated 38%) 2024-04-03T18:46:16.6163848Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fec56e5c84f7d6ea.xml (deflated 37%) 2024-04-03T18:46:16.6165871Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4bc9135a1d56b1c2.xml (deflated 38%) 2024-04-03T18:46:16.6167913Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7fb436c73bc4f791.xml (deflated 37%) 2024-04-03T18:46:16.6169963Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-afd2554147dee9a5.xml (deflated 45%) 2024-04-03T18:46:16.6172005Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-038fb5711b270350.xml (deflated 49%) 2024-04-03T18:46:16.6174010Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2300e094d81f1760.xml (deflated 49%) 2024-04-03T18:46:16.6176036Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6512593620203c33.xml (deflated 38%) 2024-04-03T18:46:16.6178067Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dd828a07b8477bb5.xml (deflated 43%) 2024-04-03T18:46:16.6180107Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-99fc6639ffa19569.xml (deflated 43%) 2024-04-03T18:46:16.6182133Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-28e8eb42d4ec44d1.xml (deflated 43%) 2024-04-03T18:46:16.6184175Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2446781188e97d41.xml (deflated 44%) 2024-04-03T18:46:16.6186359Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-303edaa40e331420.xml (deflated 45%) 2024-04-03T18:46:16.6188413Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-191e6cf76ef806a8.xml (deflated 44%) 2024-04-03T18:46:16.6190431Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c6047efa546c8008.xml (deflated 44%) 2024-04-03T18:46:16.6192457Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-428732010118e963.xml (deflated 45%) 2024-04-03T18:46:16.6194733Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5b788aca28b69df.xml (deflated 44%) 2024-04-03T18:46:16.6196790Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-84f05d14d64f3a0b.xml (deflated 58%) 2024-04-03T18:46:16.6198809Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6c8b547762e56e97.xml (deflated 48%) 2024-04-03T18:46:16.6200847Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c969774207d17ad1.xml (deflated 49%) 2024-04-03T18:46:16.6202885Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-05a8bda61038d81a.xml (deflated 38%) 2024-04-03T18:46:16.6204905Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-698a64b7d91b6f30.xml (deflated 38%) 2024-04-03T18:46:16.6206957Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-354379a04df84b43.xml (deflated 38%) 2024-04-03T18:46:16.6209010Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6eeca1c08bd6da42.xml (deflated 37%) 2024-04-03T18:46:16.6211055Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b5345514be742177.xml (deflated 49%) 2024-04-03T18:46:16.6213072Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dbdc00ce3a532d27.xml (deflated 37%) 2024-04-03T18:46:16.6215115Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-62b25634d7d0184c.xml (deflated 44%) 2024-04-03T18:46:16.6217157Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2cec343cf6e43b63.xml (deflated 37%) 2024-04-03T18:46:16.6219261Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1a3b70b09478f5eb.xml (deflated 37%) 2024-04-03T18:46:16.6221283Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ef3c83e67743f224.xml (deflated 37%) 2024-04-03T18:46:16.6223326Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ce633de44bd703e.xml (deflated 37%) 2024-04-03T18:46:16.6225369Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1e71749b2159ec1.xml (deflated 37%) 2024-04-03T18:46:16.6227565Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a1ae2b306438f705.xml (deflated 37%) 2024-04-03T18:46:16.6229581Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-11030810c2d56272.xml (deflated 45%) 2024-04-03T18:46:16.6231625Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-749d52cc5890f51a.xml (deflated 43%) 2024-04-03T18:46:16.6233665Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-728fef47d11dae7e.xml (deflated 43%) 2024-04-03T18:46:16.6235708Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-be0ef31c11bdce16.xml (deflated 37%) 2024-04-03T18:46:16.6237737Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6fbf30963a37a28.xml (deflated 37%) 2024-04-03T18:46:16.6239773Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3d2b8074faf10483.xml (deflated 41%) 2024-04-03T18:46:16.6241807Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-165f49a38660b5ec.xml (deflated 37%) 2024-04-03T18:46:16.6244093Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fe4617a7a13e209d.xml (deflated 38%) 2024-04-03T18:46:16.6246129Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aa913e53ff52278c.xml (deflated 37%) 2024-04-03T18:46:16.6248211Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-18d5f73b6692105c.xml (deflated 37%) 2024-04-03T18:46:16.6250255Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6b79370edefaf90a.xml (deflated 38%) 2024-04-03T18:46:16.6252294Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a24e915d019db467.xml (deflated 48%) 2024-04-03T18:46:16.6254306Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-501262573e765d58.xml (deflated 49%) 2024-04-03T18:46:16.6256355Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d23067a9f0fd5a40.xml (deflated 37%) 2024-04-03T18:46:16.6258394Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f34171968daff240.xml (deflated 48%) 2024-04-03T18:46:16.6260429Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3fbb97e8a43d479c.xml (deflated 49%) 2024-04-03T18:46:16.6262447Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-364f9f8c463d266e.xml (deflated 37%) 2024-04-03T18:46:16.6264481Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-998883fe8ab8c6cd.xml (deflated 38%) 2024-04-03T18:46:16.6266678Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0a7d23a02770f4ea.xml (deflated 49%) 2024-04-03T18:46:16.6268742Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-540937fa6afe8387.xml (deflated 42%) 2024-04-03T18:46:16.6270758Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24f8c19b904a78be.xml (deflated 37%) 2024-04-03T18:46:16.6272805Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-871f3c6abf6eb33e.xml (deflated 37%) 2024-04-03T18:46:16.6274845Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-daa123c93b731937.xml (deflated 37%) 2024-04-03T18:46:16.6276889Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-beb604f948ed8962.xml (deflated 38%) 2024-04-03T18:46:16.6278909Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a06d5c812163e1a1.xml (deflated 38%) 2024-04-03T18:46:16.6280972Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-99a07e8a7b2ca422.xml (deflated 37%) 2024-04-03T18:46:16.6283008Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-57018d155c02968d.xml (deflated 38%) 2024-04-03T18:46:16.6285041Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-65ba97011382ea12.xml (deflated 38%) 2024-04-03T18:46:16.6287063Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8b6c63c97bc4f363.xml (deflated 37%) 2024-04-03T18:46:16.6289107Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-86ea6267eacb965c.xml (deflated 37%) 2024-04-03T18:46:16.6291262Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7aa411a161afa62d.xml (deflated 37%) 2024-04-03T18:46:16.6293413Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f599002a136903b.xml (deflated 45%) 2024-04-03T18:46:16.6295429Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-381192eddecb15e2.xml (deflated 43%) 2024-04-03T18:46:16.6297472Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c9d3402bf78d24a.xml (deflated 37%) 2024-04-03T18:46:16.6299518Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-83cdcfbe09a57e26.xml (deflated 43%) 2024-04-03T18:46:16.6301550Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2022c832441961a9.xml (deflated 47%) 2024-04-03T18:46:16.6303572Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3948c389abdd6cfd.xml (deflated 38%) 2024-04-03T18:46:16.6305771Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-71b9ad73b41a51b7.xml (deflated 45%) 2024-04-03T18:46:16.6307831Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f78f409d8bbf290a.xml (deflated 37%) 2024-04-03T18:46:16.6309875Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-efa43a231c1fde15.xml (deflated 38%) 2024-04-03T18:46:16.6311899Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-976f2d461ff97f78.xml (deflated 38%) 2024-04-03T18:46:16.6313956Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-edfec2de1de6fa7f.xml (deflated 38%) 2024-04-03T18:46:16.6316019Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7dbab5d577b260da.xml (deflated 37%) 2024-04-03T18:46:16.6318080Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-063833a0d3a6b238.xml (deflated 37%) 2024-04-03T18:46:16.6320134Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3b8fb86e2b5745a1.xml (deflated 38%) 2024-04-03T18:46:16.6322184Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7b2350ef468d82a2.xml (deflated 48%) 2024-04-03T18:46:16.6324223Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e4ef88b89811968.xml (deflated 49%) 2024-04-03T18:46:16.6326254Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f95212411b16546.xml (deflated 37%) 2024-04-03T18:46:16.6328266Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-278ce293ea3502db.xml (deflated 48%) 2024-04-03T18:46:16.6330317Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7896e699f91990d6.xml (deflated 48%) 2024-04-03T18:46:16.6332361Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-243dae8e5e0e326d.xml (deflated 37%) 2024-04-03T18:46:16.6334407Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d87127f5e34167dd.xml (deflated 38%) 2024-04-03T18:46:16.6336499Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a5a79b8e5743bc1f.xml (deflated 49%) 2024-04-03T18:46:16.6338556Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-16ea11a02e7cfd76.xml (deflated 42%) 2024-04-03T18:46:16.6340847Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-175df30b91d43609.xml (deflated 37%) 2024-04-03T18:46:16.6342899Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5641598f2fc34713.xml (deflated 37%) 2024-04-03T18:46:16.6344917Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e7afb93ef7cb26ff.xml (deflated 37%) 2024-04-03T18:46:16.6347140Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-42a1fe37277b834e.xml (deflated 38%) 2024-04-03T18:46:16.6349189Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f168f227e81aa7a1.xml (deflated 38%) 2024-04-03T18:46:16.6351224Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-18f19a81a259e11f.xml (deflated 38%) 2024-04-03T18:46:16.6353259Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-309e08e404d7e0a8.xml (deflated 38%) 2024-04-03T18:46:16.6355302Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a0b46f3863ea2513.xml (deflated 38%) 2024-04-03T18:46:16.6357344Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6b2549633ee35e70.xml (deflated 38%) 2024-04-03T18:46:16.6359384Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-668c21ff125b457a.xml (deflated 38%) 2024-04-03T18:46:16.6361407Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4d0b404105523a3d.xml (deflated 37%) 2024-04-03T18:46:16.6363449Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a63246dd464b539.xml (deflated 45%) 2024-04-03T18:46:16.6365502Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d3c96034049cdef.xml (deflated 43%) 2024-04-03T18:46:16.6367544Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9988550b5cad4ffb.xml (deflated 37%) 2024-04-03T18:46:16.6369566Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-80213bb4d2da7fa8.xml (deflated 43%) 2024-04-03T18:46:16.6371610Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ec539a3b75c602f1.xml (deflated 47%) 2024-04-03T18:46:16.6373648Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96584ea03a777261.xml (deflated 37%) 2024-04-03T18:46:16.6375679Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b3566aef34f1fd9c.xml (deflated 45%) 2024-04-03T18:46:16.6377712Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-14971e69a3dc59b6.xml (deflated 38%) 2024-04-03T18:46:16.6379757Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1cf649c986df5339.xml (deflated 38%) 2024-04-03T18:46:16.6382148Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-ac06a233a1b851d8.xml (deflated 39%) 2024-04-03T18:46:16.6404675Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-6ec3c57580a43bff.xml (deflated 39%) 2024-04-03T18:46:16.6407413Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-d14dc47be513c3ae.xml (deflated 50%) 2024-04-03T18:46:16.6410469Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-8ca1444f9207f756.xml (deflated 50%) 2024-04-03T18:46:16.6413194Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-171b2c0875c996de.xml (deflated 51%) 2024-04-03T18:46:16.6415874Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-dcef6b454d434c39.xml (deflated 51%) 2024-04-03T18:46:16.6418567Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-62e3e290cda3be69.xml (deflated 39%) 2024-04-03T18:46:16.6421307Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-5d57a0f6f973f1b9.xml (deflated 39%) 2024-04-03T18:46:16.6424013Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-03ce0ba50c8f2f60.xml (deflated 50%) 2024-04-03T18:46:16.6426904Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-676d6fffd85e5fed.xml (deflated 50%) 2024-04-03T18:46:16.6429601Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-1ab1434bb0976e69.xml (deflated 51%) 2024-04-03T18:46:16.6432271Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-fbe632195b933a98.xml (deflated 51%) 2024-04-03T18:46:16.6434613Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24d87a9e5d522079.xml (deflated 58%) 2024-04-03T18:46:16.6436657Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-77c0b7081fbdc18a.xml (deflated 44%) 2024-04-03T18:46:16.6438677Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b2797c5d6375eb9c.xml (deflated 43%) 2024-04-03T18:46:16.6440701Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9def4666eddb0081.xml (deflated 44%) 2024-04-03T18:46:16.6442715Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-668917a04d29b783.xml (deflated 38%) 2024-04-03T18:46:16.6444718Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cb6005045112a494.xml (deflated 44%) 2024-04-03T18:46:16.6446720Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3b2eb02bae69d2ce.xml (deflated 44%) 2024-04-03T18:46:16.6448764Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b5f9caa0f1667c65.xml (deflated 43%) 2024-04-03T18:46:16.6450786Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f91621be06139701.xml (deflated 44%) 2024-04-03T18:46:16.6452803Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-066a984be3aeac21.xml (deflated 43%) 2024-04-03T18:46:16.6454800Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-284e1ece1c266bfb.xml (deflated 42%) 2024-04-03T18:46:16.6456805Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-579537277a9445fd.xml (deflated 47%) 2024-04-03T18:46:16.6458818Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a973a4da51f6e5d2.xml (deflated 37%) 2024-04-03T18:46:16.6461084Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1e1464b43bc4a47.xml (deflated 37%) 2024-04-03T18:46:16.6463107Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-001f2ee2b7be2c80.xml (deflated 38%) 2024-04-03T18:46:16.6465129Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ff69b632a8a03d98.xml (deflated 38%) 2024-04-03T18:46:16.6467335Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5f8dd7cbfe0db925.xml (deflated 38%) 2024-04-03T18:46:16.6469358Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b3938da39c2159f.xml (deflated 37%) 2024-04-03T18:46:16.6471355Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-74a6ba79d9c37155.xml (deflated 37%) 2024-04-03T18:46:16.6473386Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d2d5ea7896ea0098.xml (deflated 38%) 2024-04-03T18:46:16.6475410Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3797c42a1a137899.xml (deflated 38%) 2024-04-03T18:46:16.6477410Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-970ef895f1f6fee3.xml (deflated 38%) 2024-04-03T18:46:16.6479424Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-da54bdabd7fbd430.xml (deflated 47%) 2024-04-03T18:46:16.6481447Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1cda4af5a3800037.xml (deflated 37%) 2024-04-03T18:46:16.6483463Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-00f86f60cb97901d.xml (deflated 37%) 2024-04-03T18:46:16.6485471Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f88ac6875e49a3f.xml (deflated 38%) 2024-04-03T18:46:16.6487488Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2168c8038e6c0546.xml (deflated 38%) 2024-04-03T18:46:16.6489496Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-97dfd2479c700644.xml (deflated 37%) 2024-04-03T18:46:16.6491488Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-340587657f0b2990.xml (deflated 44%) 2024-04-03T18:46:16.6493470Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e7553420daa42cc.xml (deflated 44%) 2024-04-03T18:46:16.6495485Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c42c448bfea6732a.xml (deflated 36%) 2024-04-03T18:46:16.6497514Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ba08b8426fcd5baf.xml (deflated 49%) 2024-04-03T18:46:16.6499540Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ad2688f571946a1f.xml (deflated 38%) 2024-04-03T18:46:16.6501530Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6be651412bbdee7a.xml (deflated 43%) 2024-04-03T18:46:16.6503553Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc067a6ede98e857.xml (deflated 37%) 2024-04-03T18:46:16.6505757Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1f5ab06e0be47f7b.xml (deflated 38%) 2024-04-03T18:46:16.6507778Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-50a419ed53794628.xml (deflated 45%) 2024-04-03T18:46:16.6510001Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dd56d9a19095a946.xml (deflated 37%) 2024-04-03T18:46:16.6512036Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2e07129b946c218e.xml (deflated 58%) 2024-04-03T18:46:16.6514053Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e1f2534b40a7ea21.xml (deflated 44%) 2024-04-03T18:46:16.6516066Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8c90af34cdbcf126.xml (deflated 43%) 2024-04-03T18:46:16.6518067Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c6c16576b6732be1.xml (deflated 44%) 2024-04-03T18:46:16.6520125Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1bf4d94c14bb69f1.xml (deflated 38%) 2024-04-03T18:46:16.6522153Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9d49688599b0cb56.xml (deflated 44%) 2024-04-03T18:46:16.6524168Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1954ea3b9a74242d.xml (deflated 44%) 2024-04-03T18:46:16.6526152Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13a395a32759867e.xml (deflated 43%) 2024-04-03T18:46:16.6528155Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7485c7b02324d95c.xml (deflated 44%) 2024-04-03T18:46:16.6530171Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7bc0e4355c2e4e3e.xml (deflated 44%) 2024-04-03T18:46:16.6532193Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-98e6cf1be9ac2b1f.xml (deflated 42%) 2024-04-03T18:46:16.6534209Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-93f4b3d3813cc1b5.xml (deflated 47%) 2024-04-03T18:46:16.6536230Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d2da8557f1cc6207.xml (deflated 37%) 2024-04-03T18:46:16.6538246Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-42a2a99c1d71b407.xml (deflated 37%) 2024-04-03T18:46:16.6540256Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-402248781f36f2b3.xml (deflated 38%) 2024-04-03T18:46:16.6542250Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2782fcad278ce1fa.xml (deflated 37%) 2024-04-03T18:46:16.6544256Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f8012c439c3f518f.xml (deflated 38%) 2024-04-03T18:46:16.6546429Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8843a1645e9c9fdc.xml (deflated 37%) 2024-04-03T18:46:16.6548490Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-558c55028599e7ad.xml (deflated 37%) 2024-04-03T18:46:16.6550468Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5480e6e630873fd2.xml (deflated 38%) 2024-04-03T18:46:16.6552469Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7c861e56e7a8d68d.xml (deflated 37%) 2024-04-03T18:46:16.6554462Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-97d615b23e981e17.xml (deflated 37%) 2024-04-03T18:46:16.6556471Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc3f34d1c87086e8.xml (deflated 47%) 2024-04-03T18:46:16.6558726Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-600c7a18d37c5e9f.xml (deflated 38%) 2024-04-03T18:46:16.6560741Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3b7027910a2c53b7.xml (deflated 37%) 2024-04-03T18:46:16.6562742Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6168259c669a88d6.xml (deflated 38%) 2024-04-03T18:46:16.6564747Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6ef6b16ab778ccfb.xml (deflated 38%) 2024-04-03T18:46:16.6566736Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5558f88169fb28d5.xml (deflated 37%) 2024-04-03T18:46:16.6568746Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6f4000dcdffa8cf.xml (deflated 44%) 2024-04-03T18:46:16.6570781Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ca3f830d70e6e6a.xml (deflated 44%) 2024-04-03T18:46:16.6572790Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5672425e7a7730d4.xml (deflated 37%) 2024-04-03T18:46:16.6574767Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ac3e182974f5076.xml (deflated 49%) 2024-04-03T18:46:16.6576770Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-690a614948e271de.xml (deflated 38%) 2024-04-03T18:46:16.6578774Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1205fdfc18d77736.xml (deflated 43%) 2024-04-03T18:46:16.6580783Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-43ebdd08c93067d7.xml (deflated 38%) 2024-04-03T18:46:16.6582796Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6bf1eb688077f7fd.xml (deflated 37%) 2024-04-03T18:46:16.6584801Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f3e712906b607b42.xml (deflated 45%) 2024-04-03T18:46:16.6586991Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-726f6be9999b77b2.xml (deflated 37%) 2024-04-03T18:46:16.6588996Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b3eb83312c575a01.xml (deflated 38%) 2024-04-03T18:46:16.6590995Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0b94a9a20801ad26.xml (deflated 38%) 2024-04-03T18:46:16.6593011Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-16c2be9916a1f14e.xml (deflated 38%) 2024-04-03T18:46:16.6595037Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0e85a61db2dafc30.xml (deflated 45%) 2024-04-03T18:46:16.6597035Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a674076e151c739f.xml (deflated 45%) 2024-04-03T18:46:16.6599030Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7c47a4e9eb4b868e.xml (deflated 45%) 2024-04-03T18:46:16.6601038Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-147c1f7f408fc381.xml (deflated 45%) 2024-04-03T18:46:16.6603047Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-477fb5954f4fa174.xml (deflated 49%) 2024-04-03T18:46:16.6605168Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fd229846a1a7f75c.xml (deflated 49%) 2024-04-03T18:46:16.6607270Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c120f85180840517.xml (deflated 38%) 2024-04-03T18:46:16.6609275Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fc5e27f13bff206f.xml (deflated 43%) 2024-04-03T18:46:16.6611294Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-df797cbe3d5928bb.xml (deflated 43%) 2024-04-03T18:46:16.6613305Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-86bf8d593ac0353b.xml (deflated 43%) 2024-04-03T18:46:16.6615305Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b8c07c6cbfdb3c87.xml (deflated 44%) 2024-04-03T18:46:16.6617314Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-015418fe54502451.xml (deflated 44%) 2024-04-03T18:46:16.6619376Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e37c2a92e8d92af.xml (deflated 44%) 2024-04-03T18:46:16.6621387Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bbb3c130a5ed6352.xml (deflated 44%) 2024-04-03T18:46:16.6623372Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f41559307f086c0a.xml (deflated 44%) 2024-04-03T18:46:16.6625372Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-18f9cdc722038e66.xml (deflated 44%) 2024-04-03T18:46:16.6627552Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2fd8aaf7f1c0468c.xml (deflated 58%) 2024-04-03T18:46:16.6629573Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e993539aa085b968.xml (deflated 49%) 2024-04-03T18:46:16.6631573Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b31506579bbde003.xml (deflated 44%) 2024-04-03T18:46:16.6633577Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-032c1bd06fee13c8.xml (deflated 41%) 2024-04-03T18:46:16.6635580Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ace038f50a7488a3.xml (deflated 37%) 2024-04-03T18:46:16.6637572Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-336658f018b7caf7.xml (deflated 38%) 2024-04-03T18:46:16.6639555Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a0f4c5a7f1486da.xml (deflated 37%) 2024-04-03T18:46:16.6641570Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4cf2ae3a023c8a40.xml (deflated 49%) 2024-04-03T18:46:16.6643590Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-21333a43a176646c.xml (deflated 37%) 2024-04-03T18:46:16.6645583Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1abaae2ff1945908.xml (deflated 43%) 2024-04-03T18:46:16.6647574Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d9a14769a7aa76ca.xml (deflated 37%) 2024-04-03T18:46:16.6649579Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a9150ad6e830b7c.xml (deflated 43%) 2024-04-03T18:46:16.6651589Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1ce3e306bdb246c7.xml (deflated 46%) 2024-04-03T18:46:16.6653709Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bf75ba95937feb35.xml (deflated 37%) 2024-04-03T18:46:16.6655809Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-da076dad38ee1f8f.xml (deflated 38%) 2024-04-03T18:46:16.6657818Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a45d39a219f7ce6.xml (deflated 37%) 2024-04-03T18:46:16.6659829Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e01eecb3da080fa.xml (deflated 45%) 2024-04-03T18:46:16.6661820Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5b4d332fb75fce48.xml (deflated 43%) 2024-04-03T18:46:16.6663834Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0cb7de40a1dca0d9.xml (deflated 43%) 2024-04-03T18:46:16.6666208Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8eeb126e069eb53b.xml (deflated 44%) 2024-04-03T18:46:16.6668228Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1378f8c23d1b29a5.xml (deflated 43%) 2024-04-03T18:46:16.6670235Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f666f106d23fde8f.xml (deflated 41%) 2024-04-03T18:46:16.6672237Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-95eb64fd20c89859.xml (deflated 39%) 2024-04-03T18:46:16.6674229Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-805dfbd42e996d19.xml (deflated 38%) 2024-04-03T18:46:16.6676244Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d103d28032842bd.xml (deflated 38%) 2024-04-03T18:46:16.6678269Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-335461259905de1d.xml (deflated 46%) 2024-04-03T18:46:16.6680270Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-031bc60dd6b16d67.xml (deflated 46%) 2024-04-03T18:46:16.6682257Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c3714d6be44d7be.xml (deflated 45%) 2024-04-03T18:46:16.6684272Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-500fee3c4bfbc986.xml (deflated 45%) 2024-04-03T18:46:16.6686287Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9869c9fd13f073e2.xml (deflated 49%) 2024-04-03T18:46:16.6688274Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-859c84440d11572d.xml (deflated 49%) 2024-04-03T18:46:16.6690273Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-041ff209e7dc2b58.xml (deflated 38%) 2024-04-03T18:46:16.6692266Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5d413167b610d383.xml (deflated 44%) 2024-04-03T18:46:16.6694275Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6a746fa512aac20c.xml (deflated 43%) 2024-04-03T18:46:16.6696269Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2b81390db063054.xml (deflated 44%) 2024-04-03T18:46:16.6698275Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e726af8e4d1d5703.xml (deflated 44%) 2024-04-03T18:46:16.6700288Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51ed370c2e3adaa1.xml (deflated 45%) 2024-04-03T18:46:16.6702552Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6ef67965904b0dbb.xml (deflated 44%) 2024-04-03T18:46:16.6704562Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bcb08b522cea35ce.xml (deflated 44%) 2024-04-03T18:46:16.6706741Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e03a00003bf4f18.xml (deflated 45%) 2024-04-03T18:46:16.6708750Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-37d2895e985e1207.xml (deflated 44%) 2024-04-03T18:46:16.6710758Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3821ac0ff73d92af.xml (deflated 58%) 2024-04-03T18:46:16.6712751Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a5f01db110af74a3.xml (deflated 49%) 2024-04-03T18:46:16.6714779Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-07090eec480d2bec.xml (deflated 44%) 2024-04-03T18:46:16.6716778Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eeb0ec7821069318.xml (deflated 42%) 2024-04-03T18:46:16.6718779Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7e2637be0e14dd45.xml (deflated 38%) 2024-04-03T18:46:16.6720817Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-80ba9060e94ec380.xml (deflated 38%) 2024-04-03T18:46:16.6722823Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-287d77a10ee146a7.xml (deflated 37%) 2024-04-03T18:46:16.6724833Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3c11da17a94742d4.xml (deflated 49%) 2024-04-03T18:46:16.6726863Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-45f80ecb37fdd0c3.xml (deflated 37%) 2024-04-03T18:46:16.6728857Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a78a91afddaac34c.xml (deflated 43%) 2024-04-03T18:46:16.6730863Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a135059b502395f.xml (deflated 37%) 2024-04-03T18:46:16.6732851Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ae6abc4e8345640.xml (deflated 43%) 2024-04-03T18:46:16.6734834Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3551f2499299d9dd.xml (deflated 46%) 2024-04-03T18:46:16.6736822Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c3b3799202d255f5.xml (deflated 37%) 2024-04-03T18:46:16.6738827Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-841131e172b5dc75.xml (deflated 37%) 2024-04-03T18:46:16.6740832Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0e6fd1d6195ab33f.xml (deflated 37%) 2024-04-03T18:46:16.6742835Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0dcf382971f3d0e5.xml (deflated 45%) 2024-04-03T18:46:16.6744827Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e5ebd68229a3be4.xml (deflated 43%) 2024-04-03T18:46:16.6746965Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-50643221746606b8.xml (deflated 43%) 2024-04-03T18:46:16.6748956Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c0d9937905883e4d.xml (deflated 43%) 2024-04-03T18:46:16.6751203Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2b77a78ac89ce70c.xml (deflated 43%) 2024-04-03T18:46:16.6753208Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17bd1604c60a4b55.xml (deflated 42%) 2024-04-03T18:46:16.6755205Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dfe178ba9f228bd4.xml (deflated 38%) 2024-04-03T18:46:16.6757210Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-97a4c17ed8df8b68.xml (deflated 38%) 2024-04-03T18:46:16.6759219Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3aad3e8724659a6.xml (deflated 45%) 2024-04-03T18:46:16.6761205Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca66b62e53ff02a3.xml (deflated 44%) 2024-04-03T18:46:16.6763233Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f03f9e46e72436b7.xml (deflated 44%) 2024-04-03T18:46:16.6765232Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-599a0a4ce2b105e6.xml (deflated 46%) 2024-04-03T18:46:16.6767235Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2cc042fb1481c607.xml (deflated 49%) 2024-04-03T18:46:16.6769223Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e74101d95ae161a.xml (deflated 44%) 2024-04-03T18:46:16.6771217Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c54157c187f862e.xml (deflated 44%) 2024-04-03T18:46:16.6773213Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-81e16cedb77ae914.xml (deflated 44%) 2024-04-03T18:46:16.6775227Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-459ee43407b3b2d2.xml (deflated 37%) 2024-04-03T18:46:16.6777213Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-749307f565becb6b.xml (deflated 47%) 2024-04-03T18:46:16.6779218Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dee8458f26d4aae7.xml (deflated 49%) 2024-04-03T18:46:16.6781225Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-25c205f71ad4b20d.xml (deflated 42%) 2024-04-03T18:46:16.6783209Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e53c8102d488824d.xml (deflated 37%) 2024-04-03T18:46:16.6785189Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6a8bbf70844ea3f.xml (deflated 37%) 2024-04-03T18:46:16.6787382Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7d5a3d6a5a33053d.xml (deflated 37%) 2024-04-03T18:46:16.6789393Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1224ea7dbc89d7af.xml (deflated 38%) 2024-04-03T18:46:16.6791400Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3bb0a99b9cd632df.xml (deflated 38%) 2024-04-03T18:46:16.6793387Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b5913fa551804528.xml (deflated 38%) 2024-04-03T18:46:16.6795378Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-74f59b6684bc2187.xml (deflated 38%) 2024-04-03T18:46:16.6797386Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-722fa7b7a77d16bf.xml (deflated 38%) 2024-04-03T18:46:16.6799640Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e618653d6ee2f14b.xml (deflated 38%) 2024-04-03T18:46:16.6801650Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7031b6f546a21f66.xml (deflated 38%) 2024-04-03T18:46:16.6803647Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a7871993bb8cba53.xml (deflated 44%) 2024-04-03T18:46:16.6805650Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dfae5fa7d30d25e6.xml (deflated 45%) 2024-04-03T18:46:16.6807646Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7bdeb7334bd22244.xml (deflated 43%) 2024-04-03T18:46:16.6809623Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-19b4506f54087325.xml (deflated 37%) 2024-04-03T18:46:16.6811632Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-03a7e9ecff928012.xml (deflated 43%) 2024-04-03T18:46:16.6813640Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-569b554b97b3997f.xml (deflated 47%) 2024-04-03T18:46:16.6815633Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3699fa1734013b78.xml (deflated 38%) 2024-04-03T18:46:16.6816612Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-abdcfb7c77a8d219.xml (deflated 45%) 2024-04-03T18:46:16.6817573Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-21344fbc5f749276.xml (deflated 45%) 2024-04-03T18:46:16.6818551Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd3ea6f90e59c420.xml (deflated 38%) 2024-04-03T18:46:16.6819584Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a06b4c856573d7bd.xml (deflated 38%) 2024-04-03T18:46:16.6820564Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-82772dac39287a3d.xml (deflated 38%) 2024-04-03T18:46:16.6821532Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2740c8a320be9d18.xml (deflated 45%) 2024-04-03T18:46:16.6822488Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4af59ea4b7869230.xml (deflated 44%) 2024-04-03T18:46:16.6823462Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-20d00514d66bdc71.xml (deflated 44%) 2024-04-03T18:46:16.6824431Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e6d13455896fee21.xml (deflated 46%) 2024-04-03T18:46:16.6825413Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-664d2ec5b9327136.xml (deflated 49%) 2024-04-03T18:46:16.6826543Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c7d07e6ef6e9052e.xml (deflated 44%) 2024-04-03T18:46:16.6827526Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f0eb8fa1f2db7fb7.xml (deflated 44%) 2024-04-03T18:46:16.6828502Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-08da73ad9a3c5de0.xml (deflated 44%) 2024-04-03T18:46:16.6829467Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c8a8e74182b01bb4.xml (deflated 38%) 2024-04-03T18:46:16.6830557Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-46f97aef89a0f8ac.xml (deflated 47%) 2024-04-03T18:46:16.6831637Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e69fcf515ea98bf9.xml (deflated 49%) 2024-04-03T18:46:16.6832605Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8d562fbdb31f13e2.xml (deflated 42%) 2024-04-03T18:46:16.6833581Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-290580ae5eff3fd0.xml (deflated 37%) 2024-04-03T18:46:16.6834542Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c3a3dd1426979609.xml (deflated 37%) 2024-04-03T18:46:16.6835520Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cb7fd129b8670caf.xml (deflated 37%) 2024-04-03T18:46:16.6836490Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-26dbf84bd19ecf53.xml (deflated 38%) 2024-04-03T18:46:16.6837461Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2d096a69f1212093.xml (deflated 38%) 2024-04-03T18:46:16.6838437Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6868941ffa8cd43a.xml (deflated 38%) 2024-04-03T18:46:16.6839406Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-310ebdf8cf5175fc.xml (deflated 38%) 2024-04-03T18:46:16.6840376Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5453d8410b4ec96b.xml (deflated 38%) 2024-04-03T18:46:16.6841334Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2657631ccfcf2087.xml (deflated 37%) 2024-04-03T18:46:16.6842310Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1d670780cef420fd.xml (deflated 37%) 2024-04-03T18:46:16.6843297Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a44128c4ba828c0d.xml (deflated 44%) 2024-04-03T18:46:16.6844262Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1cd2d37178e90eab.xml (deflated 45%) 2024-04-03T18:46:16.6845234Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-121ada2a2d728267.xml (deflated 43%) 2024-04-03T18:46:16.6846199Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a0a4d0496d52f6f8.xml (deflated 37%) 2024-04-03T18:46:16.6847211Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-93d9c26edbea9944.xml (deflated 43%) 2024-04-03T18:46:16.6848193Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e74a1d9e800f26ac.xml (deflated 47%) 2024-04-03T18:46:16.6849170Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-73088e7d2dcb84ff.xml (deflated 37%) 2024-04-03T18:46:16.6850148Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d9912fef87add3d7.xml (deflated 45%) 2024-04-03T18:46:16.6851110Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-11d76bf5688692f8.xml (deflated 45%) 2024-04-03T18:46:16.6852069Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c0dd03595d85e4ce.xml (deflated 38%) 2024-04-03T18:46:16.6882473Z ##[group]Run # Remove any previous test reports if they exist 2024-04-03T18:46:16.6882801Z # Remove any previous test reports if they exist 2024-04-03T18:46:16.6882954Z rm -f logs-*.zip 2024-04-03T18:46:16.6883718Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-04-03T18:46:16.6884002Z # so check to see if the file exists first 2024-04-03T18:46:16.6884184Z if [ -f 'usage_log.txt' ]; then 2024-04-03T18:46:16.6884458Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-04-03T18:46:16.6884588Z fi 2024-04-03T18:46:16.6884793Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-04-03T18:46:16.6885080Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-04-03T18:46:16.6885205Z fi 2024-04-03T18:46:16.6892940Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:46:16.6893065Z env: 2024-04-03T18:46:16.6893221Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:46:16.6893500Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:46:16.6893962Z DOCKER_CONTAINER_ID: e451a0adc73818f3a047cb83ff91d039adc3cd55a8f5a0cb827c973fea5bd896 2024-04-03T18:46:16.6894383Z FILE_SUFFIX: test-distributed-3-3-linux.8xlarge.nvidia.gpu_23402706518 2024-04-03T18:46:16.6894527Z ##[endgroup] 2024-04-03T18:46:16.7483826Z adding: usage_log.txt (deflated 96%) 2024-04-03T18:46:16.7771644Z adding: test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_bdc5285b23114a1a_.log (deflated 55%) 2024-04-03T18:46:16.7779589Z adding: test/test-reports/distributed.test_distributed_spawn_7.7_b4a0fdb855a0c2d4_.log (deflated 92%) 2024-04-03T18:46:16.7781045Z adding: test/test-reports/distributed._tensor.test_init_1.1_5320e5fe4e0137ce_.log (deflated 69%) 2024-04-03T18:46:16.7782487Z adding: test/test-reports/distributed.test_distributed_spawn_7.7_2f45b07ac9dec14e_.log (deflated 12%) 2024-04-03T18:46:16.7783994Z adding: test/test-reports/distributed._composable.test_checkpoint_1.1_f592fa157770fb92_.log (deflated 67%) 2024-04-03T18:46:16.7788246Z adding: test/test-reports/distributed.test_distributed_spawn_7.7_b401ac7049deb526_.log (deflated 93%) 2024-04-03T18:46:16.7790043Z adding: test/test-reports/distributed._tensor.test_dtensor_compile_1.1_04f66d18ea139d04_.log (deflated 89%) 2024-04-03T18:46:16.7796139Z adding: test/test-reports/distributed.test_distributed_spawn_7.7_0afed1f42cf4ccc1_.log (deflated 93%) 2024-04-03T18:46:16.7797686Z adding: test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_7b2189c283a19011_.log (deflated 52%) 2024-04-03T18:46:16.7803589Z adding: test/test-reports/distributed.test_distributed_spawn_7.7_099c77c62c01a5cc_.log (deflated 93%) 2024-04-03T18:46:16.7805136Z adding: test/test-reports/distributed.checkpoint.test_save_load_api_1.1_ae554707cd6c3fa6_.log (deflated 53%) 2024-04-03T18:46:16.7811033Z adding: test/test-reports/distributed.test_distributed_spawn_7.7_369b067f6ab52d86_.log (deflated 93%) 2024-04-03T18:46:16.7812494Z adding: test/test-reports/distributed._spmd.test_data_parallel_1.1_38bc5c6b5868eb5b_.log (stored 0%) 2024-04-03T18:46:16.7818740Z adding: test/test-reports/distributed.test_distributed_spawn_7.7_8f3b0333eaf97b10_.log (deflated 93%) 2024-04-03T18:46:16.7820239Z adding: test/test-reports/distributed.test_distributed_spawn_1.7_0601af37cdbb41cd_.log (deflated 12%) 2024-04-03T18:46:16.7823115Z adding: test/test-reports/distributed.test_functional_api_1.1_ca0301fccc2e1f63_.log (deflated 92%) 2024-04-03T18:46:16.7824577Z adding: test/test-reports/distributed.test_distributed_spawn_1.7_edd3d9f7ae01ba50_.log (deflated 12%) 2024-04-03T18:46:16.7826202Z adding: test/test-reports/distributed.rpc.test_share_memory_1.1_0bbc2379a34af0dd_.log (deflated 57%) 2024-04-03T18:46:16.7832924Z adding: test/test-reports/distributed.test_distributed_spawn_1.7_53301db6d035a5c0_.log (deflated 92%) 2024-04-03T18:46:16.7834387Z adding: test/test-reports/distributed.fsdp.test_fsdp_input_1.1_dd481dbae4d04fa8_.log (deflated 67%) 2024-04-03T18:46:16.7841940Z adding: test/test-reports/distributed.test_distributed_spawn_1.7_dc23d305cc089c79_.log (deflated 93%) 2024-04-03T18:46:16.7843781Z adding: test/test-reports/distributed.fsdp.test_fsdp_uneven_1.1_91f5baa042f208ec_.log (deflated 53%) 2024-04-03T18:46:16.7850364Z adding: test/test-reports/distributed.test_distributed_spawn_1.7_00533d651908b9e9_.log (deflated 93%) 2024-04-03T18:46:16.7883237Z adding: test/test-reports/distributed.fsdp.test_fsdp_core_1.2_1251799597747aa0_.log (deflated 96%) 2024-04-03T18:46:16.7890870Z adding: test/test-reports/distributed.test_distributed_spawn_1.7_059406b002abc96e_.log (deflated 93%) 2024-04-03T18:46:16.7892324Z adding: test/test-reports/distributed.rpc.test_faulty_agent_1.1_9c6de7659bbb5721_.log (deflated 36%) 2024-04-03T18:46:16.7900936Z adding: test/test-reports/distributed.test_distributed_spawn_1.7_117a7f4f5f876735_.log (deflated 92%) 2024-04-03T18:46:16.7902986Z adding: test/test-reports/distributed.test_pg_wrapper_1.1_f19af57a9c7dd20f_.log (deflated 93%) 2024-04-03T18:46:16.7913270Z adding: test/test-reports/distributed.test_distributed_spawn_1.7_7f9804eb8b719486_.log (deflated 92%) 2024-04-03T18:46:16.7914746Z adding: test/test-reports/distributed.test_c10d_spawn_ucc_1.1_2d3b576963f0935b_.log (deflated 89%) 2024-04-03T18:46:16.7916179Z adding: test/test-reports/distributed.test_distributed_spawn_4.7_b0f9aba2f77ecf94_.log (deflated 12%) 2024-04-03T18:46:16.7917610Z adding: test/test-reports/distributed.test_data_parallel_1.1_259678cc80509e74_.log (deflated 87%) 2024-04-03T18:46:16.7919086Z adding: test/test-reports/distributed.test_distributed_spawn_4.7_99b891ef90ace298_.log (deflated 12%) 2024-04-03T18:46:16.7927865Z adding: test/test-reports/distributed.test_store_1.1_ba1e3246f41b86e1_.log (deflated 95%) 2024-04-03T18:46:16.7935376Z adding: test/test-reports/distributed.test_distributed_spawn_4.7_8446e0b7e6c8743d_.log (deflated 93%) 2024-04-03T18:46:16.7962779Z adding: test/test-reports/distributed.test_c10d_nccl_2.2_fa9bfe56bb0b5164_.log (deflated 95%) 2024-04-03T18:46:16.7970309Z adding: test/test-reports/distributed.test_distributed_spawn_4.7_20c512541393f1d1_.log (deflated 93%) 2024-04-03T18:46:16.7978390Z adding: test/test-reports/distributed.test_distributed_spawn_4.7_14afd39b8a1c0b1b_.log (deflated 94%) 2024-04-03T18:46:16.7986647Z adding: test/test-reports/distributed.test_distributed_spawn_4.7_dc8c6a2d91480ebb_.log (deflated 94%) 2024-04-03T18:46:16.7994700Z adding: test/test-reports/distributed.test_distributed_spawn_4.7_f9bb301952eeb94f_.log (deflated 93%) 2024-04-03T18:46:16.8002700Z adding: test/test-reports/distributed.test_distributed_spawn_4.7_30ec712cbfee1a74_.log (deflated 93%) 2024-04-03T18:46:16.8004157Z adding: test/test-reports/distributed.test_distributed_spawn_7.7_b3347970d7198181_.log (deflated 12%) 2024-04-03T18:46:16.8005598Z adding: test/test-reports/distributed.optim.test_named_optimizer_1.1_1774d2ff12840844_.log (stored 0%) 2024-04-03T18:46:16.8007091Z adding: test/test-reports/distributed.test_inductor_collectives_1.1_954504a6275e4c63_.log (deflated 93%) 2024-04-03T18:46:16.8008593Z adding: test/test-reports/distributed._composable.test_compose_1.1_72ec906ad1796bec_.log (deflated 90%) 2024-04-03T18:46:16.8010127Z adding: test/test-reports/distributed._tensor.debug.test_comm_mode_1.1_4b085a7970ce12c0_.log (deflated 55%) 2024-04-03T18:46:16.8011737Z adding: test/test-reports/distributed.tensor.parallel.test_tp_examples_1.1_f98e22abf68fffc5_.log (deflated 74%) 2024-04-03T18:46:16.8013365Z adding: test/test-reports/distributed.fsdp.test_fsdp_backward_prefetch_1.1_c01256328cc8beb8_.log (deflated 63%) 2024-04-03T18:46:16.8014933Z adding: test/test-reports/distributed._composable.test_replicate_1.1_4edbda9424142652_.log (deflated 74%) 2024-04-03T18:46:16.8016527Z adding: test/test-reports/distributed._shard.sharding_plan.test_sharding_plan_1.1_5c3f2919ac34196c_.log (deflated 65%) 2024-04-03T18:46:16.8018154Z adding: test/test-reports/distributed.pipeline.sync.test_inplace_1.1_0233b43f411056a4_.log (deflated 59%) 2024-04-03T18:46:16.8019896Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_clip_grad_norm__1.1_42c874edff863625_.log (deflated 61%) 2024-04-03T18:46:16.8021935Z adding: test/test-reports/distributed.pipeline.sync.test_dependency_1.1_d04a3ab63e20334a_.log (deflated 65%) 2024-04-03T18:46:16.8023653Z adding: test/test-reports/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion_1.1_e1832f98755cfebd_.log (deflated 70%) 2024-04-03T18:46:16.8025287Z adding: test/test-reports/distributed._tensor.test_optimizers_1.1_7b8a4ae776f35b9c_.log (deflated 76%) 2024-04-03T18:46:16.8027143Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_tensor_ops_1.1_90351e1710e031ff_.log (deflated 70%) 2024-04-03T18:46:16.8028865Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_edf706a973aa33a5_.log (deflated 66%) 2024-04-03T18:46:16.8030479Z adding: test/test-reports/distributed.fsdp.test_fsdp_flatten_params_1.1_08891095b96f30b5_.log (deflated 82%) 2024-04-03T18:46:16.8032065Z adding: test/test-reports/distributed.fsdp.test_fsdp_ignored_modules_1.1_cc8d154f55e6b3db_.log (deflated 91%) 2024-04-03T18:46:16.8033653Z adding: test/test-reports/distributed.checkpoint.test_checkpoint_1.1_763d4cf5bbe10aa5_.log (deflated 81%) 2024-04-03T18:46:16.8035236Z adding: test/test-reports/distributed.fsdp.test_fsdp_freezing_weights_1.1_94c0efac8066a46a_.log (deflated 93%) 2024-04-03T18:46:16.8036771Z adding: test/test-reports/distributed.fsdp.test_fsdp_traversal_1.1_907a08fff0ad5454_.log (deflated 53%) 2024-04-03T18:46:16.8038332Z adding: test/test-reports/distributed.fsdp.test_fsdp_multiple_wrapping_1.1_5c4c46e97cb340a3_.log (deflated 63%) 2024-04-03T18:46:16.8046991Z adding: test/test-reports/distributed.rpc.cuda.test_tensorpipe_agent_1.2_5efa29acd448dda8_.log (deflated 97%) 2024-04-03T18:46:16.8048716Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_runtime_1.1_531af965937050b9_.log (deflated 83%) 2024-04-03T18:46:16.8059440Z adding: test/test-reports/distributed.fsdp.test_fsdp_clip_grad_norm_1.1_8524e74f6a7070eb_.log (deflated 96%) 2024-04-03T18:46:16.8060985Z adding: test/test-reports/distributed.fsdp.test_fsdp_exec_order_1.1_20e98eb01b817e9c_.log (deflated 90%) 2024-04-03T18:46:16.8062694Z adding: test/test-reports/distributed.pipeline.sync.test_pipe_1.1_b65732e662869d46_.log (deflated 83%) 2024-04-03T18:46:16.8064941Z adding: test/test-reports/distributed.fsdp.test_fsdp_fine_tune_1.1_21ef5b58070f5c39_.log (deflated 92%) 2024-04-03T18:46:16.8066768Z adding: test/test-reports/distributed.pipeline.sync.test_balance_1.1_aff550061fccb10c_.log (deflated 75%) 2024-04-03T18:46:16.8068424Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_ebe94a5e0c5a3c86_.log (stored 0%) 2024-04-03T18:46:16.8070107Z adding: test/test-reports/distributed.pipeline.sync.skip.test_api_1.1_29082bd71ec9e6c8_.log (deflated 58%) 2024-04-03T18:46:16.8071797Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_ccb00e2d19b0a174_.log (stored 0%) 2024-04-03T18:46:16.8073560Z adding: test/test-reports/distributed.pipeline.sync.test_deferred_batch_norm_1.1_3cfb9c566e84850b_.log (deflated 72%) 2024-04-03T18:46:16.8075297Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_077cf16a76dd90ad_.log (deflated 90%) 2024-04-03T18:46:16.8076960Z adding: test/test-reports/distributed.checkpoint.test_state_dict_1.1_e315175c13bd7934_.log (deflated 84%) 2024-04-03T18:46:16.8078647Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_37f8c5c30eb53ad9_.log (deflated 90%) 2024-04-03T18:46:16.8080384Z adding: test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_1.1_c5f8719403171f06_.log (deflated 89%) 2024-04-03T18:46:16.8082139Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_a824c422a6708f46_.log (deflated 90%) 2024-04-03T18:46:16.8083906Z adding: test/test-reports/distributed.pipeline.sync.skip.test_verify_skippables_1.1_c971a164f0c8f629_.log (deflated 71%) 2024-04-03T18:46:16.8086010Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_af965abe4bb9486e_.log (deflated 90%) 2024-04-03T18:46:16.8087804Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_bec76d76296ef90b_.log (stored 0%) 2024-04-03T18:46:16.8089561Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_f26cc8364d311ba1_.log (stored 0%) 2024-04-03T18:46:16.8091193Z adding: test/test-reports/distributed.fsdp.test_fsdp_checkpoint_1.1_ed2fe2f341499fe6_.log (deflated 88%) 2024-04-03T18:46:16.8092733Z adding: test/test-reports/distributed.pipeline.sync.test_worker_1.1_d48c8e607e82038a_.log (deflated 64%) 2024-04-03T18:46:16.8094315Z adding: test/test-reports/distributed.pipeline.sync.test_transparency_1.1_7a7763c818d30bd8_.log (deflated 53%) 2024-04-03T18:46:16.8095911Z adding: test/test-reports/distributed.fsdp.test_fsdp_unshard_params_1.1_e901e7c4adde1487_.log (deflated 93%) 2024-04-03T18:46:16.8097473Z adding: test/test-reports/distributed.fsdp.test_fsdp_hybrid_shard_1.1_fab3fb41a1eb25a1_.log (deflated 90%) 2024-04-03T18:46:16.8099508Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_optim_checkpoint_1.1_2793910f1cf7594e_.log (deflated 76%) 2024-04-03T18:46:16.8101605Z adding: test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_reshard_1.1_3b5d71927f2416c8_.log (deflated 63%) 2024-04-03T18:46:16.8186462Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-04-03T18:46:16.8187022Z with: 2024-04-03T18:46:16.8187348Z s3-bucket: gha-artifacts 2024-04-03T18:46:16.8187878Z s3-prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T18:46:16.8188458Z retention-days: 14 2024-04-03T18:46:16.8188850Z if-no-files-found: warn 2024-04-03T18:46:16.8189282Z path: test-jsons-*.zip 2024-04-03T18:46:16.8189694Z name: artifact 2024-04-03T18:46:16.8190042Z region: us-east-1 2024-04-03T18:46:16.8190403Z env: 2024-04-03T18:46:16.8190729Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:46:16.8191287Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:46:16.8192232Z DOCKER_CONTAINER_ID: e451a0adc73818f3a047cb83ff91d039adc3cd55a8f5a0cb827c973fea5bd896 2024-04-03T18:46:16.8193057Z ##[endgroup] 2024-04-03T18:46:17.2616124Z NOTE: s3-prefix specified, ignoring name parameter 2024-04-03T18:46:17.2617081Z With the provided path, there will be 1 file uploaded 2024-04-03T18:46:17.2617848Z Uploading to s3 prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T18:46:17.4289282Z Starting upload of test-jsons-test-distributed-3-3-linux.8xlarge.nvidia.gpu_23402706518.zip 2024-04-03T18:46:17.5968226Z Finished upload of test-jsons-test-distributed-3-3-linux.8xlarge.nvidia.gpu_23402706518.zip 2024-04-03T18:46:17.6139099Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-04-03T18:46:17.6139632Z with: 2024-04-03T18:46:17.6139973Z s3-bucket: gha-artifacts 2024-04-03T18:46:17.6140494Z s3-prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T18:46:17.6141053Z retention-days: 14 2024-04-03T18:46:17.6141485Z if-no-files-found: error 2024-04-03T18:46:17.6141945Z path: test-reports-*.zip 2024-04-03T18:46:17.6142353Z name: artifact 2024-04-03T18:46:17.6142719Z region: us-east-1 2024-04-03T18:46:17.6143080Z env: 2024-04-03T18:46:17.6143388Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:46:17.6143943Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:46:17.6144881Z DOCKER_CONTAINER_ID: e451a0adc73818f3a047cb83ff91d039adc3cd55a8f5a0cb827c973fea5bd896 2024-04-03T18:46:17.6145969Z ##[endgroup] 2024-04-03T18:46:18.0605020Z NOTE: s3-prefix specified, ignoring name parameter 2024-04-03T18:46:18.0606137Z With the provided path, there will be 1 file uploaded 2024-04-03T18:46:18.0606916Z Uploading to s3 prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T18:46:18.0654537Z Starting upload of test-reports-test-distributed-3-3-linux.8xlarge.nvidia.gpu_23402706518.zip 2024-04-03T18:46:18.2525397Z Finished upload of test-reports-test-distributed-3-3-linux.8xlarge.nvidia.gpu_23402706518.zip 2024-04-03T18:46:18.2700232Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-04-03T18:46:18.2700782Z with: 2024-04-03T18:46:18.2701125Z s3-bucket: gha-artifacts 2024-04-03T18:46:18.2701654Z s3-prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T18:46:18.2702218Z retention-days: 14 2024-04-03T18:46:18.2702632Z if-no-files-found: ignore 2024-04-03T18:46:18.2703076Z path: logs-*.zip 2024-04-03T18:46:18.2703433Z name: artifact 2024-04-03T18:46:18.2703796Z region: us-east-1 2024-04-03T18:46:18.2704163Z env: 2024-04-03T18:46:18.2704475Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:46:18.2705036Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:46:18.2706437Z DOCKER_CONTAINER_ID: e451a0adc73818f3a047cb83ff91d039adc3cd55a8f5a0cb827c973fea5bd896 2024-04-03T18:46:18.2707254Z ##[endgroup] 2024-04-03T18:46:18.7128825Z NOTE: s3-prefix specified, ignoring name parameter 2024-04-03T18:46:18.7129990Z With the provided path, there will be 1 file uploaded 2024-04-03T18:46:18.7130843Z Uploading to s3 prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T18:46:18.7177462Z Starting upload of logs-test-distributed-3-3-linux.8xlarge.nvidia.gpu_23402706518.zip 2024-04-03T18:46:18.9300839Z Finished upload of logs-test-distributed-3-3-linux.8xlarge.nvidia.gpu_23402706518.zip 2024-04-03T18:46:18.9472384Z ##[group]Run # shellcheck disable=SC2156 2024-04-03T18:46:18.9473016Z # shellcheck disable=SC2156 2024-04-03T18:46:18.9474024Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-04-03T18:46:18.9482972Z shell: /usr/bin/bash -e {0} 2024-04-03T18:46:18.9483399Z env: 2024-04-03T18:46:18.9483737Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:46:18.9484303Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:46:18.9485232Z DOCKER_CONTAINER_ID: e451a0adc73818f3a047cb83ff91d039adc3cd55a8f5a0cb827c973fea5bd896 2024-04-03T18:46:18.9486054Z ##[endgroup] 2024-04-03T18:46:19.1957824Z GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2 2024-04-03T18:46:19.1959331Z Copyright (C) 2020 Free Software Foundation, Inc. 2024-04-03T18:46:19.1960221Z License GPLv3+: GNU GPL version 3 or later 2024-04-03T18:46:19.1961279Z This is free software: you are free to change and redistribute it. 2024-04-03T18:46:19.1962085Z There is NO WARRANTY, to the extent permitted by law. 2024-04-03T18:46:19.1962799Z Type "show copying" and "show warranty" for details. 2024-04-03T18:46:19.1963553Z This GDB was configured as "x86_64-linux-gnu". 2024-04-03T18:46:19.1964210Z Type "show configuration" for configuration details. 2024-04-03T18:46:19.1964875Z For bug reporting instructions, please see: 2024-04-03T18:46:19.1965473Z . 2024-04-03T18:46:19.1966180Z Find the GDB manual and other documentation resources online at: 2024-04-03T18:46:19.1966977Z . 2024-04-03T18:46:19.1967450Z 2024-04-03T18:46:19.1967715Z For help, type "help". 2024-04-03T18:46:19.1968957Z Type "apropos word" to search for commands related to "word"... 2024-04-03T18:46:19.2716639Z Reading symbols from python... 2024-04-03T18:46:19.9583379Z 2024-04-03T18:46:19.9584735Z warning: core file may not match specified executable file. 2024-04-03T18:46:19.9889519Z [New LWP 9558] 2024-04-03T18:46:19.9890280Z [New LWP 9524] 2024-04-03T18:46:19.9890997Z [New LWP 9528] 2024-04-03T18:46:19.9891784Z [New LWP 9526] 2024-04-03T18:46:19.9892262Z [New LWP 9531] 2024-04-03T18:46:19.9892596Z [New LWP 9533] 2024-04-03T18:46:19.9892955Z [New LWP 9544] 2024-04-03T18:46:19.9893311Z [New LWP 9543] 2024-04-03T18:46:19.9893647Z [New LWP 9521] 2024-04-03T18:46:19.9894001Z [New LWP 9522] 2024-04-03T18:46:19.9894351Z [New LWP 9523] 2024-04-03T18:46:19.9894681Z [New LWP 9525] 2024-04-03T18:46:19.9895035Z [New LWP 9527] 2024-04-03T18:46:19.9895389Z [New LWP 9529] 2024-04-03T18:46:19.9895722Z [New LWP 9530] 2024-04-03T18:46:19.9896072Z [New LWP 9488] 2024-04-03T18:46:19.9896806Z [New LWP 9532] 2024-04-03T18:46:19.9897138Z [New LWP 9535] 2024-04-03T18:46:19.9897487Z [New LWP 9534] 2024-04-03T18:46:19.9897836Z [New LWP 9536] 2024-04-03T18:46:19.9898169Z [New LWP 9537] 2024-04-03T18:46:19.9898523Z [New LWP 9538] 2024-04-03T18:46:19.9898875Z [New LWP 9541] 2024-04-03T18:46:19.9899219Z [New LWP 9542] 2024-04-03T18:46:19.9899574Z [New LWP 9539] 2024-04-03T18:46:19.9899924Z [New LWP 9540] 2024-04-03T18:46:19.9900256Z [New LWP 9545] 2024-04-03T18:46:19.9900604Z [New LWP 9546] 2024-04-03T18:46:19.9900941Z [New LWP 9548] 2024-04-03T18:46:19.9901291Z [New LWP 9547] 2024-04-03T18:46:19.9901643Z [New LWP 9549] 2024-04-03T18:46:19.9901978Z [New LWP 9569] 2024-04-03T18:46:19.9902325Z [New LWP 9556] 2024-04-03T18:46:19.9902676Z [New LWP 9557] 2024-04-03T18:46:19.9903010Z [New LWP 9567] 2024-04-03T18:46:19.9903358Z [New LWP 9554] 2024-04-03T18:46:19.9903707Z [New LWP 9572] 2024-04-03T18:46:19.9904041Z [New LWP 9550] 2024-04-03T18:46:19.9904390Z [New LWP 9551] 2024-04-03T18:46:19.9904753Z [New LWP 9559] 2024-04-03T18:46:19.9932714Z [Thread debugging using libthread_db enabled] 2024-04-03T18:46:19.9934294Z Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1". 2024-04-03T18:46:35.7914760Z 50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. 2024-04-03T18:46:35.7916583Z Core was generated by `/opt/conda/envs/py_3.10/bin/python -c from multiprocessing.spawn import spawn_m'. 2024-04-03T18:46:35.7917703Z Program terminated with signal SIGABRT, Aborted. 2024-04-03T18:46:35.7918503Z #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 2024-04-03T18:46:35.7919317Z [Current thread is 1 (Thread 0x7fc74a983700 (LWP 9558))] 2024-04-03T18:46:35.7920935Z warning: File "/var/lib/jenkins/workspace/.gdbinit" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load". 2024-04-03T18:46:35.8008756Z To enable execution of this file add 2024-04-03T18:46:35.8010466Z add-auto-load-safe-path /var/lib/jenkins/workspace/.gdbinit 2024-04-03T18:46:35.8011354Z line to your configuration file "/var/lib/jenkins/.gdbinit". 2024-04-03T18:46:35.8012115Z To completely disable this security protection add 2024-04-03T18:46:35.8012790Z set auto-load safe-path / 2024-04-03T18:46:35.8013382Z line to your configuration file "/var/lib/jenkins/.gdbinit". 2024-04-03T18:46:35.8014191Z For more information about this security protection see the 2024-04-03T18:46:35.8015200Z "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: 2024-04-03T18:46:35.8016083Z info "(gdb)Auto-loading safe path" 2024-04-03T18:46:35.8016750Z #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 2024-04-03T18:46:35.8017561Z #1 0x00007fc84ecfc859 in __GI_abort () at abort.c:79 2024-04-03T18:46:35.9104235Z #2 0x00007fc8293535e9 in c10::MessageLogger::~MessageLogger() [clone .cold] 2024-04-03T18:46:35.9104993Z () 2024-04-03T18:46:35.9106123Z from /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so 2024-04-03T18:46:35.9116845Z #3 0x00007fc82a6a0874 in c10d::ProcessGroupNCCL::terminateProcess(std::__cxx11::basic_string, std::allocator >) () 2024-04-03T18:46:35.9118474Z from /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so 2024-04-03T18:46:35.9128733Z #4 0x00007fc82a6a6b13 in c10d::ProcessGroupNCCL::heartbeatMonitor() () 2024-04-03T18:46:35.9129904Z from /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so 2024-04-03T18:46:35.9218548Z #5 0x00007fc84686baeb in std::execute_native_thread_routine ( 2024-04-03T18:46:35.9219810Z __p=) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 2024-04-03T18:46:35.9221550Z #6 0x00007fc84f02e609 in start_thread (arg=) 2024-04-03T18:46:35.9222315Z at pthread_create.c:477 2024-04-03T18:46:35.9222766Z #7 0x00007fc84edf9353 in clone () 2024-04-03T18:46:35.9223328Z at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 2024-04-03T18:46:36.1193124Z GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2 2024-04-03T18:46:36.1194505Z Copyright (C) 2020 Free Software Foundation, Inc. 2024-04-03T18:46:36.1195394Z License GPLv3+: GNU GPL version 3 or later 2024-04-03T18:46:36.1196441Z This is free software: you are free to change and redistribute it. 2024-04-03T18:46:36.1197273Z There is NO WARRANTY, to the extent permitted by law. 2024-04-03T18:46:36.1197977Z Type "show copying" and "show warranty" for details. 2024-04-03T18:46:36.1198741Z This GDB was configured as "x86_64-linux-gnu". 2024-04-03T18:46:36.1199416Z Type "show configuration" for configuration details. 2024-04-03T18:46:36.1200067Z For bug reporting instructions, please see: 2024-04-03T18:46:36.1200671Z . 2024-04-03T18:46:36.1201396Z Find the GDB manual and other documentation resources online at: 2024-04-03T18:46:36.1202179Z . 2024-04-03T18:46:36.1202659Z 2024-04-03T18:46:36.1202836Z For help, type "help". 2024-04-03T18:46:36.1203431Z Type "apropos word" to search for commands related to "word"... 2024-04-03T18:46:36.1953018Z Reading symbols from python... 2024-04-03T18:46:36.8946397Z 2024-04-03T18:46:36.9254003Z warning: core file may not match specified executable file. 2024-04-03T18:46:36.9255620Z [New LWP 9562] 2024-04-03T18:46:36.9256005Z [New LWP 9494] 2024-04-03T18:46:36.9256392Z [New LWP 9490] 2024-04-03T18:46:36.9256751Z [New LWP 9496] 2024-04-03T18:46:36.9257094Z [New LWP 9499] 2024-04-03T18:46:36.9257481Z [New LWP 9489] 2024-04-03T18:46:36.9257853Z [New LWP 9505] 2024-04-03T18:46:36.9258168Z [New LWP 9491] 2024-04-03T18:46:36.9258757Z [New LWP 9495] 2024-04-03T18:46:36.9259251Z [New LWP 9497] 2024-04-03T18:46:36.9259589Z [New LWP 9498] 2024-04-03T18:46:36.9259906Z [New LWP 9493] 2024-04-03T18:46:36.9260223Z [New LWP 9500] 2024-04-03T18:46:36.9260557Z [New LWP 9492] 2024-04-03T18:46:36.9260910Z [New LWP 9503] 2024-04-03T18:46:36.9261265Z [New LWP 9502] 2024-04-03T18:46:36.9261617Z [New LWP 9501] 2024-04-03T18:46:36.9261967Z [New LWP 9504] 2024-04-03T18:46:36.9262317Z [New LWP 9506] 2024-04-03T18:46:36.9262646Z [New LWP 9507] 2024-04-03T18:46:36.9262996Z [New LWP 9509] 2024-04-03T18:46:36.9263327Z [New LWP 9510] 2024-04-03T18:46:36.9263679Z [New LWP 9512] 2024-04-03T18:46:36.9264039Z [New LWP 9511] 2024-04-03T18:46:36.9264404Z [New LWP 9508] 2024-04-03T18:46:36.9264736Z [New LWP 9513] 2024-04-03T18:46:36.9265084Z [New LWP 9514] 2024-04-03T18:46:36.9265416Z [New LWP 9515] 2024-04-03T18:46:36.9266107Z [New LWP 9516] 2024-04-03T18:46:36.9266463Z [New LWP 9518] 2024-04-03T18:46:36.9266798Z [New LWP 9517] 2024-04-03T18:46:36.9267146Z [New LWP 9519] 2024-04-03T18:46:36.9267498Z [New LWP 9520] 2024-04-03T18:46:36.9267828Z [New LWP 9561] 2024-04-03T18:46:36.9268177Z [New LWP 9560] 2024-04-03T18:46:36.9268525Z [New LWP 9570] 2024-04-03T18:46:36.9268856Z [New LWP 9571] 2024-04-03T18:46:36.9269205Z [New LWP 9563] 2024-04-03T18:46:36.9269554Z [New LWP 9568] 2024-04-03T18:46:36.9269894Z [New LWP 9555] 2024-04-03T18:46:36.9296156Z [Thread debugging using libthread_db enabled] 2024-04-03T18:46:36.9297145Z Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1". 2024-04-03T18:46:52.7051618Z 50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. 2024-04-03T18:46:52.7053395Z warning: File "/var/lib/jenkins/workspace/.gdbinit" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load". 2024-04-03T18:46:52.7055257Z Core was generated by `/opt/conda/envs/py_3.10/bin/python -c from multiprocessing.spawn import spawn_m'. 2024-04-03T18:46:52.7056343Z Program terminated with signal SIGABRT, Aborted. 2024-04-03T18:46:52.7057157Z #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 2024-04-03T18:46:52.7057997Z [Current thread is 1 (Thread 0x7f0e1a983700 (LWP 9562))] 2024-04-03T18:46:52.7131388Z To enable execution of this file add 2024-04-03T18:46:52.7132276Z add-auto-load-safe-path /var/lib/jenkins/workspace/.gdbinit 2024-04-03T18:46:52.7133455Z line to your configuration file "/var/lib/jenkins/.gdbinit". 2024-04-03T18:46:52.7134204Z To completely disable this security protection add 2024-04-03T18:46:52.7134873Z set auto-load safe-path / 2024-04-03T18:46:52.7135493Z line to your configuration file "/var/lib/jenkins/.gdbinit". 2024-04-03T18:46:52.7136293Z For more information about this security protection see the 2024-04-03T18:46:52.7137302Z "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: 2024-04-03T18:46:52.7138154Z info "(gdb)Auto-loading safe path" 2024-04-03T18:46:52.7138854Z #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 2024-04-03T18:46:52.7139662Z #1 0x00007f0f1e38b859 in __GI_abort () at abort.c:79 2024-04-03T18:46:52.8237973Z #2 0x00007f0ef89e25e9 in c10::MessageLogger::~MessageLogger() [clone .cold] 2024-04-03T18:46:52.8238711Z () 2024-04-03T18:46:52.8239504Z from /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so 2024-04-03T18:46:52.8250074Z #3 0x00007f0ef9d2f874 in c10d::ProcessGroupNCCL::terminateProcess(std::__cxx11::basic_string, std::allocator >) () 2024-04-03T18:46:52.8251726Z from /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so 2024-04-03T18:46:52.8262249Z #4 0x00007f0ef9d35b13 in c10d::ProcessGroupNCCL::heartbeatMonitor() () 2024-04-03T18:46:52.8263451Z from /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so 2024-04-03T18:46:52.8354059Z #5 0x00007f0f15efaaeb in std::execute_native_thread_routine ( 2024-04-03T18:46:52.8355339Z __p=) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 2024-04-03T18:46:52.8356500Z #6 0x00007f0f1e6bd609 in start_thread (arg=) 2024-04-03T18:46:52.8357121Z at pthread_create.c:477 2024-04-03T18:46:52.8357591Z #7 0x00007f0f1e488353 in clone () 2024-04-03T18:46:52.8358130Z at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 2024-04-03T18:46:53.0703748Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-04-03T18:46:53.0704488Z with: 2024-04-03T18:46:53.0704779Z env: 2024-04-03T18:46:53.0705114Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:46:53.0705901Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:46:53.0706845Z DOCKER_CONTAINER_ID: e451a0adc73818f3a047cb83ff91d039adc3cd55a8f5a0cb827c973fea5bd896 2024-04-03T18:46:53.0707670Z ##[endgroup] 2024-04-03T18:46:53.2032040Z ##[group]Run set -eou pipefail 2024-04-03T18:46:53.2032555Z set -eou pipefail 2024-04-03T18:46:53.2032970Z  2024-04-03T18:46:53.2033605Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-04-03T18:46:53.2034415Z for _ in $(seq 1440); do 2024-04-03T18:46:53.2034984Z  # Break if no ssh session exists anymore 2024-04-03T18:46:53.2035606Z  if [ "$(who)" = "" ]; then 2024-04-03T18:46:53.2036114Z  break 2024-04-03T18:46:53.2036564Z  fi 2024-04-03T18:46:53.2036919Z  echo "." 2024-04-03T18:46:53.2037323Z  sleep 5 2024-04-03T18:46:53.2037710Z done 2024-04-03T18:46:53.2046216Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:46:53.2046829Z env: 2024-04-03T18:46:53.2047171Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:46:53.2047738Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:46:53.2048662Z DOCKER_CONTAINER_ID: e451a0adc73818f3a047cb83ff91d039adc3cd55a8f5a0cb827c973fea5bd896 2024-04-03T18:46:53.2049486Z ##[endgroup] 2024-04-03T18:46:53.2075971Z Holding runner for 2 hours until all ssh sessions have logged out 2024-04-03T18:46:53.4868546Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T18:46:53.4869531Z # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T18:46:53.4870278Z # shellcheck disable=SC2046 2024-04-03T18:46:53.4870875Z docker stop $(docker ps -q) || true 2024-04-03T18:46:53.4871674Z # Prune all of the docker images 2024-04-03T18:46:53.4872244Z docker system prune -af 2024-04-03T18:46:53.4880646Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:46:53.4881280Z env: 2024-04-03T18:46:53.4881603Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:46:53.4882170Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:46:53.4883118Z DOCKER_CONTAINER_ID: e451a0adc73818f3a047cb83ff91d039adc3cd55a8f5a0cb827c973fea5bd896 2024-04-03T18:46:53.4883951Z ##[endgroup] 2024-04-03T18:46:55.0650288Z e451a0adc738 2024-04-03T18:46:55.6232051Z Deleted Containers: 2024-04-03T18:46:55.6232977Z e451a0adc73818f3a047cb83ff91d039adc3cd55a8f5a0cb827c973fea5bd896 2024-04-03T18:46:55.6233599Z 2024-04-03T18:47:02.2571291Z Deleted Images: 2024-04-03T18:47:02.2573060Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T18:47:02.2577858Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9@sha256:68e0ca2e38253fb3bd10c84d380560df0602da1b16bccdfd7d4dd7ac22529759 2024-04-03T18:47:02.2581831Z deleted: sha256:f0aa5113e491b8f8ef3b51fc066bce798422ecc2bdeb4be54a8a4efa6914052e 2024-04-03T18:47:02.2583189Z deleted: sha256:c454db9d5e4e4efa0edf7c73a701618617c23c0f03a17d5e9c5f49ccbbfa0e73 2024-04-03T18:47:02.2584287Z deleted: sha256:b2dd02706fb0cdd8a4420189b04471a7739f76c16bb6c9e36747a62daaf28ad9 2024-04-03T18:47:02.2585344Z deleted: sha256:3e52b95dec2563c40922e1bfd58e5ec9813906f6958f9e481f3dad044b00c734 2024-04-03T18:47:02.2586922Z deleted: sha256:eeeb44278fe18f54afa2e7db8fd0210ec20683601770a2353cb53c27f7a1b5bc 2024-04-03T18:47:02.2588024Z deleted: sha256:3de3ee3c1c2cccb5001455ea6adcb75600111cc3fd52f8f80c11c495d6ee4e57 2024-04-03T18:47:02.2589699Z deleted: sha256:d6bf411cddfbc13bf76f9472a8b37a35ba76ffdccd5a2e21b14829368e18d3f7 2024-04-03T18:47:02.2591295Z deleted: sha256:8613c4bc44191f5e1cc622290ed8b516510d1ab876c25ccd377c36f51c2773f5 2024-04-03T18:47:02.2592419Z deleted: sha256:c3b2fbe550b9b298db4f89162757d368522e9cfede30c563e85dbf4440beb582 2024-04-03T18:47:02.2593511Z deleted: sha256:7a2ba3e1861525f15dcbdea7d5f2b39e6d14cc6a1a4d3a4669496c7a5c3a2d75 2024-04-03T18:47:02.2594599Z deleted: sha256:09e9c5c1b4f18382df5cddefe5c9b0dc27e5679be209c1e8622765d97df05bd2 2024-04-03T18:47:02.2595649Z deleted: sha256:3afff7cf2c94a194113068301442afd3f5a1b5c921841f28093ed5947ff17bf9 2024-04-03T18:47:02.2596692Z deleted: sha256:01421ca8d6d1e66f01d50067086730b4100f9a1966a2266332540cfd2fa67288 2024-04-03T18:47:02.2597740Z deleted: sha256:9d8429c72401fd7217175b98df195cdbe1dd4ff878348104c6a0be2c8fe87f07 2024-04-03T18:47:02.2598817Z deleted: sha256:b0ab639903fffd49ad6bda7fdf2fcd209dcbfce6f84ef28b9dac3dae321bb025 2024-04-03T18:47:02.2599929Z deleted: sha256:e1e2ba6eb5c7694469bb6fbd6820df749e354c265ab0b261960dec6a7a3a28be 2024-04-03T18:47:02.2601005Z deleted: sha256:9722a85cb366e43d512471e05004f0bba6c5a2c2dd592ba038ca03d96e348d92 2024-04-03T18:47:02.2602052Z deleted: sha256:59105bb9683ece65c0c62952242c8f582486ef48e62af263bdadbc6f61608fad 2024-04-03T18:47:02.2603111Z deleted: sha256:053162ab6c481eb96b5f661978e688dcc6bee984e7ec68e2d6ce6296bc891738 2024-04-03T18:47:02.2604175Z deleted: sha256:aa95ee7dea81036a66f9398d7a92020ea53c0de98fea9690a848c4887242c204 2024-04-03T18:47:02.2605242Z deleted: sha256:b00f90cd5628b07e6c16d09e2bdd195897085b1fd7c583800717b13fdb2bddba 2024-04-03T18:47:02.2606283Z deleted: sha256:d8764b2eab188f5b3c2c6573294c0e7d25cabe9e2206b51a1089bee5fdc9a099 2024-04-03T18:47:02.2607388Z deleted: sha256:2f02a119b01d7d4f2da3ae04e62eb82c6862630ad151910586b7d5fd54e3dd20 2024-04-03T18:47:02.2608462Z deleted: sha256:bb8ac4a8069a057d8ce651918a3f3a37f69cf070660ae23904dc9cd9675da101 2024-04-03T18:47:02.2609520Z deleted: sha256:20129be101ad623cea8a076f51abe873d22e6a1b9182dcffa89aefda79b6a7ec 2024-04-03T18:47:02.2610611Z deleted: sha256:ddc263e2d5206bdf8e3face17284fd07e61bedefc7820f3d15e43754688f56f6 2024-04-03T18:47:02.2611885Z deleted: sha256:331c5b9483ad593f5aa15565c0c1f483b7827a1ef011a7c0d9d40658f7e43a11 2024-04-03T18:47:02.2612958Z deleted: sha256:e2746ff342dbe1a36a1388a0bb33bc12e69b1627af1a7a13ffa6231db1645bd3 2024-04-03T18:47:02.2614021Z deleted: sha256:d2af2292b7bf3bd83e33811bdfceb98e128b901af3f13071e17ca8bb9a852a6f 2024-04-03T18:47:02.2615100Z deleted: sha256:a2ec7109ed947976900ebfb586b3d4e2552f6af6ce04fa04adda236eee7550ef 2024-04-03T18:47:02.2616176Z deleted: sha256:6b40bbee3bc6087e5c14ba5d43abde7796d62653a46181c6f87aa635f9ade9ac 2024-04-03T18:47:02.2617228Z deleted: sha256:377d27777d5b7ccd2c4a239e28d1d2ced864e35163e3ffc9131aee6b50ebfd44 2024-04-03T18:47:02.2618282Z deleted: sha256:139d166e407995d29a3d1c469c6af0eccb96895ec95721e668d7d1a96f1601e6 2024-04-03T18:47:02.2619312Z deleted: sha256:9e29d5432f1d387203719074a0d079463fc945d166c6fe9168342acd94d25565 2024-04-03T18:47:02.2620341Z deleted: sha256:7c66ee17d9810d8b654318ddadf226dc224d7ca4adbb64287aa4e1d378369d40 2024-04-03T18:47:02.2621491Z deleted: sha256:ca79b52f3eae23be700ee8aedcb485040d1918aacb1dcac027ff07a5e30d49e5 2024-04-03T18:47:02.2622594Z deleted: sha256:7fd608199fde74f258f5ff0cdb5708bbbb269c40c238f0941357570ceb3c0f97 2024-04-03T18:47:02.2623656Z deleted: sha256:953c2071c9b50052113d1dcbf64eaf51cd11e8dfa56e581b412bc075a784fb30 2024-04-03T18:47:02.2624686Z deleted: sha256:e397941ce9440cfbb931d0384718d8689325a0bca0c603e02412c5d28dc91891 2024-04-03T18:47:02.2626101Z deleted: sha256:a2f0692c36460bc69d0253b25333b5b17eb59f9f0cbc5325329b3823ac81025d 2024-04-03T18:47:02.2627473Z deleted: sha256:53a2bb00ca64b3972528438023e17333f36c3b85db48605df0f1111c08b6a91c 2024-04-03T18:47:02.2628500Z deleted: sha256:350185224c8f26322214dfdf6f1f72776da8088d3168d6f83723c98445e43f10 2024-04-03T18:47:02.2629542Z deleted: sha256:76385dca1ebd668f08aad474882b95d21679544054579c76ad79776a75b83e22 2024-04-03T18:47:02.2630592Z deleted: sha256:b11bcb556f21ebd78f10a9125d7f534ea59f86e81a006d815962a83602b0e662 2024-04-03T18:47:02.2631808Z deleted: sha256:7c6e21fb86aabd7cccb7531f8e7f4fa39607bf4136679b174638238ba2036339 2024-04-03T18:47:02.2632936Z deleted: sha256:49b69334db1af3decf2111f3474f9a7a76aeadafd34b647465bf35baf4496632 2024-04-03T18:47:02.2634013Z deleted: sha256:d6830305dc4ef0d8627ddc0fe0f1e49f5304333b7151a30b2aed904945e6a68f 2024-04-03T18:47:02.2635088Z deleted: sha256:21e3145f7deb8e4c8e4c359f495779f0b2a6ffd8a54ad879ec0bb6bcdbed9001 2024-04-03T18:47:02.2636129Z deleted: sha256:1c978ef9d890a352bfc23641fa000d332f042527032227c9715f297ce15a9d3a 2024-04-03T18:47:02.2637168Z deleted: sha256:5e2571ab32a3976828a3cf45d33bf651d60949687f8a81f3970c51c90a5a4a1d 2024-04-03T18:47:02.2638202Z deleted: sha256:547e1709b8e6365cc024934a2d46fe0b4029f4195b372a5ca63522747b09f9f5 2024-04-03T18:47:02.2639213Z deleted: sha256:d6f91263e804bd96920276a41b0538e01c33bf03b3ab32c4175588b5a9b5c217 2024-04-03T18:47:02.2640246Z deleted: sha256:ac317a820242b9e6c255f8065538f07519c66bbb5b40f313434cccb99ed34e56 2024-04-03T18:47:02.2641307Z deleted: sha256:6787bc9a8b82b4efce75e9e6d725ce95ff1e03a4f53768f96d420c5c0d5904d4 2024-04-03T18:47:02.2642371Z deleted: sha256:6a19981c4f76113029dbbcc0d87cdf2d324ec8d370b70ffb9d64623c804cfc3e 2024-04-03T18:47:02.2643447Z deleted: sha256:90ee54386e1cfb202860b4f78bfa7bb0db9722c0b8cc055c9f578130f09363d3 2024-04-03T18:47:02.2644508Z deleted: sha256:e039079492e83156c78b02baa6b4a38218c1c1529f7aec4dece116017320584d 2024-04-03T18:47:02.2645560Z deleted: sha256:488705e52ca1ed434cf0c3ee439eca1262115a69eb7492de82b7610234e2f6c5 2024-04-03T18:47:02.2646595Z deleted: sha256:5c567c50a43b1644e3211ef0e28dc73aa22a25d12bb1346308746375c9efbcfd 2024-04-03T18:47:02.2647640Z deleted: sha256:7c59d78b7f784e56d846633c9da755128c52d841ac70c649169b80efc9861e94 2024-04-03T18:47:02.2648670Z deleted: sha256:0702e7a9df3e7459b15b0f3d87c1029524539618e4676908b5934b923ed199ea 2024-04-03T18:47:02.2649675Z deleted: sha256:57a6f7d803492e9f768503b021ed8248e4d78d575185dd1938e57d18245667ad 2024-04-03T18:47:02.2650712Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-04-03T18:47:02.2651783Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-04-03T18:47:02.2652992Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-04-03T18:47:02.2654050Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-04-03T18:47:02.2655132Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-04-03T18:47:02.2656210Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-04-03T18:47:02.2657265Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-04-03T18:47:02.2658309Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-04-03T18:47:02.2659380Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-04-03T18:47:02.2660420Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-04-03T18:47:02.2661477Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-04-03T18:47:02.2662127Z 2024-04-03T18:47:02.2662333Z Total reclaimed space: 25.62GB 2024-04-03T18:47:02.2749975Z Post job cleanup. 2024-04-03T18:47:02.2816527Z Post job cleanup. 2024-04-03T18:47:02.3993280Z [command]/usr/bin/git version 2024-04-03T18:47:02.4048954Z git version 2.40.1 2024-04-03T18:47:02.4111828Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/ad7f2b2c-52c4-470b-8eeb-079d0aaf8949' before making global git config changes 2024-04-03T18:47:02.4113417Z Adding repository directory to the temporary git global config as a safe directory 2024-04-03T18:47:02.4119674Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T18:47:02.4163082Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-04-03T18:47:02.4202108Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-04-03T18:47:02.4536358Z Entering 'android/libs/fbjni' 2024-04-03T18:47:02.4589580Z Entering 'third_party/FP16' 2024-04-03T18:47:02.4643393Z Entering 'third_party/FXdiv' 2024-04-03T18:47:02.4697050Z Entering 'third_party/NNPACK' 2024-04-03T18:47:02.4750938Z Entering 'third_party/QNNPACK' 2024-04-03T18:47:02.4804480Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T18:47:02.4858405Z Entering 'third_party/XNNPACK' 2024-04-03T18:47:02.4933579Z Entering 'third_party/benchmark' 2024-04-03T18:47:02.4986067Z Entering 'third_party/cpuinfo' 2024-04-03T18:47:02.5040564Z Entering 'third_party/cub' 2024-04-03T18:47:02.5094183Z Entering 'third_party/cudnn_frontend' 2024-04-03T18:47:02.5149365Z Entering 'third_party/cutlass' 2024-04-03T18:47:02.5211923Z Entering 'third_party/eigen' 2024-04-03T18:47:02.5270209Z Entering 'third_party/fbgemm' 2024-04-03T18:47:02.5325079Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T18:47:02.5378455Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T18:47:02.5432764Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T18:47:02.5491647Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T18:47:02.5544357Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T18:47:02.5599587Z Entering 'third_party/flatbuffers' 2024-04-03T18:47:02.5656739Z Entering 'third_party/fmt' 2024-04-03T18:47:02.5711398Z Entering 'third_party/foxi' 2024-04-03T18:47:02.5764747Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T18:47:02.5819406Z Entering 'third_party/gloo' 2024-04-03T18:47:02.5874018Z Entering 'third_party/googletest' 2024-04-03T18:47:02.5928896Z Entering 'third_party/ideep' 2024-04-03T18:47:02.5981942Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T18:47:02.6045053Z Entering 'third_party/ios-cmake' 2024-04-03T18:47:02.6097802Z Entering 'third_party/ittapi' 2024-04-03T18:47:02.6152223Z Entering 'third_party/kineto' 2024-04-03T18:47:02.6206559Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T18:47:02.6259149Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T18:47:02.6314926Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T18:47:02.6367461Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T18:47:02.6419531Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T18:47:02.6472943Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T18:47:02.6526335Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T18:47:02.6579260Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T18:47:02.6631560Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T18:47:02.6686049Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T18:47:02.6741257Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T18:47:02.6794820Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T18:47:02.6848933Z Entering 'third_party/mimalloc' 2024-04-03T18:47:02.6903727Z Entering 'third_party/nccl/nccl' 2024-04-03T18:47:02.6957224Z Entering 'third_party/neon2sse' 2024-04-03T18:47:02.7010445Z Entering 'third_party/nlohmann' 2024-04-03T18:47:02.7064700Z Entering 'third_party/onnx' 2024-04-03T18:47:02.7136397Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T18:47:02.7191709Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T18:47:02.7246778Z Entering 'third_party/onnx-tensorrt' 2024-04-03T18:47:02.7299574Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T18:47:02.7357528Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T18:47:02.7410756Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T18:47:02.7463071Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T18:47:02.7522080Z Entering 'third_party/pocketfft' 2024-04-03T18:47:02.7574647Z Entering 'third_party/protobuf' 2024-04-03T18:47:02.7632302Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T18:47:02.7684174Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T18:47:02.7737807Z Entering 'third_party/psimd' 2024-04-03T18:47:02.7791295Z Entering 'third_party/pthreadpool' 2024-04-03T18:47:02.7846396Z Entering 'third_party/pybind11' 2024-04-03T18:47:02.7898711Z Entering 'third_party/python-peachpy' 2024-04-03T18:47:02.7952857Z Entering 'third_party/sleef' 2024-04-03T18:47:02.8005985Z Entering 'third_party/tbb' 2024-04-03T18:47:02.8061369Z Entering 'third_party/tensorpipe' 2024-04-03T18:47:02.8116019Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T18:47:02.8168254Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T18:47:02.8219293Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T18:47:02.8271901Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T18:47:02.8322295Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T18:47:02.8376953Z Entering 'third_party/zstd' 2024-04-03T18:47:02.8450554Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-04-03T18:47:02.8477256Z http.https://github.com/.extraheader 2024-04-03T18:47:02.8489146Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-04-03T18:47:02.8526577Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-04-03T18:47:02.8852499Z Entering 'android/libs/fbjni' 2024-04-03T18:47:02.8886981Z http.https://github.com/.extraheader 2024-04-03T18:47:02.8923592Z Entering 'third_party/FP16' 2024-04-03T18:47:02.8959017Z http.https://github.com/.extraheader 2024-04-03T18:47:02.8994674Z Entering 'third_party/FXdiv' 2024-04-03T18:47:02.9029460Z http.https://github.com/.extraheader 2024-04-03T18:47:02.9064080Z Entering 'third_party/NNPACK' 2024-04-03T18:47:02.9098054Z http.https://github.com/.extraheader 2024-04-03T18:47:02.9134438Z Entering 'third_party/QNNPACK' 2024-04-03T18:47:02.9168897Z http.https://github.com/.extraheader 2024-04-03T18:47:02.9204514Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T18:47:02.9238597Z http.https://github.com/.extraheader 2024-04-03T18:47:02.9273763Z Entering 'third_party/XNNPACK' 2024-04-03T18:47:02.9308403Z http.https://github.com/.extraheader 2024-04-03T18:47:02.9362190Z Entering 'third_party/benchmark' 2024-04-03T18:47:02.9397623Z http.https://github.com/.extraheader 2024-04-03T18:47:02.9433129Z Entering 'third_party/cpuinfo' 2024-04-03T18:47:02.9467257Z http.https://github.com/.extraheader 2024-04-03T18:47:02.9502524Z Entering 'third_party/cub' 2024-04-03T18:47:02.9537850Z http.https://github.com/.extraheader 2024-04-03T18:47:02.9573023Z Entering 'third_party/cudnn_frontend' 2024-04-03T18:47:02.9607352Z http.https://github.com/.extraheader 2024-04-03T18:47:02.9642015Z Entering 'third_party/cutlass' 2024-04-03T18:47:02.9675268Z http.https://github.com/.extraheader 2024-04-03T18:47:02.9719918Z Entering 'third_party/eigen' 2024-04-03T18:47:02.9753715Z http.https://github.com/.extraheader 2024-04-03T18:47:02.9793629Z Entering 'third_party/fbgemm' 2024-04-03T18:47:02.9826692Z http.https://github.com/.extraheader 2024-04-03T18:47:02.9861839Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T18:47:02.9895698Z http.https://github.com/.extraheader 2024-04-03T18:47:02.9930494Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T18:47:02.9963368Z http.https://github.com/.extraheader 2024-04-03T18:47:02.9998781Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T18:47:03.0031667Z http.https://github.com/.extraheader 2024-04-03T18:47:03.0074608Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T18:47:03.0106499Z http.https://github.com/.extraheader 2024-04-03T18:47:03.0141350Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T18:47:03.0174285Z http.https://github.com/.extraheader 2024-04-03T18:47:03.0210019Z Entering 'third_party/flatbuffers' 2024-04-03T18:47:03.0243983Z http.https://github.com/.extraheader 2024-04-03T18:47:03.0281726Z Entering 'third_party/fmt' 2024-04-03T18:47:03.0315775Z http.https://github.com/.extraheader 2024-04-03T18:47:03.0352855Z Entering 'third_party/foxi' 2024-04-03T18:47:03.0386771Z http.https://github.com/.extraheader 2024-04-03T18:47:03.0421619Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T18:47:03.0455549Z http.https://github.com/.extraheader 2024-04-03T18:47:03.0492573Z Entering 'third_party/gloo' 2024-04-03T18:47:03.0527056Z http.https://github.com/.extraheader 2024-04-03T18:47:03.0562198Z Entering 'third_party/googletest' 2024-04-03T18:47:03.0597526Z http.https://github.com/.extraheader 2024-04-03T18:47:03.0633518Z Entering 'third_party/ideep' 2024-04-03T18:47:03.0668380Z http.https://github.com/.extraheader 2024-04-03T18:47:03.0702928Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T18:47:03.0736889Z http.https://github.com/.extraheader 2024-04-03T18:47:03.0780831Z Entering 'third_party/ios-cmake' 2024-04-03T18:47:03.0814440Z http.https://github.com/.extraheader 2024-04-03T18:47:03.0849743Z Entering 'third_party/ittapi' 2024-04-03T18:47:03.0883798Z http.https://github.com/.extraheader 2024-04-03T18:47:03.0919073Z Entering 'third_party/kineto' 2024-04-03T18:47:03.0955264Z http.https://github.com/.extraheader 2024-04-03T18:47:03.0990480Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T18:47:03.1023999Z http.https://github.com/.extraheader 2024-04-03T18:47:03.1060177Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T18:47:03.1093582Z http.https://github.com/.extraheader 2024-04-03T18:47:03.1133944Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T18:47:03.1168018Z http.https://github.com/.extraheader 2024-04-03T18:47:03.1205964Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T18:47:03.1241559Z http.https://github.com/.extraheader 2024-04-03T18:47:03.1278041Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T18:47:03.1313376Z http.https://github.com/.extraheader 2024-04-03T18:47:03.1349302Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T18:47:03.1383397Z http.https://github.com/.extraheader 2024-04-03T18:47:03.1422359Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T18:47:03.1458253Z http.https://github.com/.extraheader 2024-04-03T18:47:03.1495435Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T18:47:03.1532242Z http.https://github.com/.extraheader 2024-04-03T18:47:03.1569590Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T18:47:03.1605078Z http.https://github.com/.extraheader 2024-04-03T18:47:03.1643358Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T18:47:03.1677753Z http.https://github.com/.extraheader 2024-04-03T18:47:03.1716624Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T18:47:03.1752255Z http.https://github.com/.extraheader 2024-04-03T18:47:03.1788198Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T18:47:03.1822118Z http.https://github.com/.extraheader 2024-04-03T18:47:03.1861404Z Entering 'third_party/mimalloc' 2024-04-03T18:47:03.1897678Z http.https://github.com/.extraheader 2024-04-03T18:47:03.1934955Z Entering 'third_party/nccl/nccl' 2024-04-03T18:47:03.1970812Z http.https://github.com/.extraheader 2024-04-03T18:47:03.2007899Z Entering 'third_party/neon2sse' 2024-04-03T18:47:03.2043567Z http.https://github.com/.extraheader 2024-04-03T18:47:03.2081332Z Entering 'third_party/nlohmann' 2024-04-03T18:47:03.2117134Z http.https://github.com/.extraheader 2024-04-03T18:47:03.2156614Z Entering 'third_party/onnx' 2024-04-03T18:47:03.2191966Z http.https://github.com/.extraheader 2024-04-03T18:47:03.2247242Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T18:47:03.2282824Z http.https://github.com/.extraheader 2024-04-03T18:47:03.2319642Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T18:47:03.2354765Z http.https://github.com/.extraheader 2024-04-03T18:47:03.2395446Z Entering 'third_party/onnx-tensorrt' 2024-04-03T18:47:03.2431153Z http.https://github.com/.extraheader 2024-04-03T18:47:03.2466373Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T18:47:03.2499864Z http.https://github.com/.extraheader 2024-04-03T18:47:03.2540325Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T18:47:03.2575440Z http.https://github.com/.extraheader 2024-04-03T18:47:03.2612265Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T18:47:03.2646354Z http.https://github.com/.extraheader 2024-04-03T18:47:03.2682978Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T18:47:03.2716830Z http.https://github.com/.extraheader 2024-04-03T18:47:03.2758328Z Entering 'third_party/pocketfft' 2024-04-03T18:47:03.2792503Z http.https://github.com/.extraheader 2024-04-03T18:47:03.2827830Z Entering 'third_party/protobuf' 2024-04-03T18:47:03.2861325Z http.https://github.com/.extraheader 2024-04-03T18:47:03.2900224Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T18:47:03.2935135Z http.https://github.com/.extraheader 2024-04-03T18:47:03.2970265Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T18:47:03.3004172Z http.https://github.com/.extraheader 2024-04-03T18:47:03.3042177Z Entering 'third_party/psimd' 2024-04-03T18:47:03.3077031Z http.https://github.com/.extraheader 2024-04-03T18:47:03.3112422Z Entering 'third_party/pthreadpool' 2024-04-03T18:47:03.3147157Z http.https://github.com/.extraheader 2024-04-03T18:47:03.3182601Z Entering 'third_party/pybind11' 2024-04-03T18:47:03.3217116Z http.https://github.com/.extraheader 2024-04-03T18:47:03.3254200Z Entering 'third_party/python-peachpy' 2024-04-03T18:47:03.3289058Z http.https://github.com/.extraheader 2024-04-03T18:47:03.3324458Z Entering 'third_party/sleef' 2024-04-03T18:47:03.3360200Z http.https://github.com/.extraheader 2024-04-03T18:47:03.3396094Z Entering 'third_party/tbb' 2024-04-03T18:47:03.3430602Z http.https://github.com/.extraheader 2024-04-03T18:47:03.3468041Z Entering 'third_party/tensorpipe' 2024-04-03T18:47:03.3502159Z http.https://github.com/.extraheader 2024-04-03T18:47:03.3537866Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T18:47:03.3571276Z http.https://github.com/.extraheader 2024-04-03T18:47:03.3606288Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T18:47:03.3639285Z http.https://github.com/.extraheader 2024-04-03T18:47:03.3675954Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T18:47:03.3709470Z http.https://github.com/.extraheader 2024-04-03T18:47:03.3744141Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T18:47:03.3778038Z http.https://github.com/.extraheader 2024-04-03T18:47:03.3812133Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T18:47:03.3846030Z http.https://github.com/.extraheader 2024-04-03T18:47:03.3884148Z Entering 'third_party/zstd' 2024-04-03T18:47:03.3919214Z http.https://github.com/.extraheader 2024-04-03T18:47:03.4257630Z Cleaning up orphan processes