2024-04-03T16:00:22.8599681Z Current runner version: '2.315.0' 2024-04-03T16:00:22.8606299Z Runner name: 'i-0ae24b2353acec2d3' 2024-04-03T16:00:22.8607109Z Runner group name: 'Default' 2024-04-03T16:00:22.8607946Z Machine name: 'ip-10-0-33-10' 2024-04-03T16:00:22.8612188Z ##[group]GITHUB_TOKEN Permissions 2024-04-03T16:00:22.8614303Z Actions: read 2024-04-03T16:00:22.8614805Z Checks: read 2024-04-03T16:00:22.8615335Z Contents: read 2024-04-03T16:00:22.8615785Z Deployments: read 2024-04-03T16:00:22.8616289Z Discussions: read 2024-04-03T16:00:22.8616807Z Issues: read 2024-04-03T16:00:22.8617241Z Metadata: read 2024-04-03T16:00:22.8617749Z Packages: read 2024-04-03T16:00:22.8618245Z Pages: read 2024-04-03T16:00:22.8618683Z PullRequests: read 2024-04-03T16:00:22.8619236Z RepositoryProjects: read 2024-04-03T16:00:22.8619815Z SecurityEvents: read 2024-04-03T16:00:22.8620297Z Statuses: read 2024-04-03T16:00:22.8620821Z ##[endgroup] 2024-04-03T16:00:22.8624246Z Secret source: None 2024-04-03T16:00:22.8625101Z Prepare workflow directory 2024-04-03T16:00:22.9776326Z Prepare all required actions 2024-04-03T16:00:22.9941348Z Getting action download info 2024-04-03T16:00:23.1632818Z Download action repository 'pytorch/test-infra@main' (SHA:f5543b9b2d7b7c7b8eccd6576b68fc0472b1fbde) 2024-04-03T16:00:23.4630955Z Download action repository 'pytorch/pytorch@main' (SHA:74b3a7920ef9b96315ef20adef858b00cc067af6) 2024-04-03T16:00:26.4818538Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-04-03T16:00:26.6264903Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-04-03T16:00:26.8716340Z Getting action download info 2024-04-03T16:00:26.9718332Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-04-03T16:00:27.1650939Z Getting action download info 2024-04-03T16:00:27.2633043Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-04-03T16:00:27.4055166Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/123154/merge (4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0) 2024-04-03T16:00:27.4057565Z ##[group] Inputs 2024-04-03T16:00:27.4058020Z build-environment: linux-jammy-py3.8-gcc11 2024-04-03T16:00:27.4061214Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "default", "shard": 2, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "default", "shard": 3, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "docs_test", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge", "unstable": "unstable"}, {"config": "jit_legacy", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "backwards_compat", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge"}]} 2024-04-03T16:00:27.4064877Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:00:27.4065891Z sync-tag: 2024-04-03T16:00:27.4066700Z timeout-minutes: 240 2024-04-03T16:00:27.4067071Z use-gha: 2024-04-03T16:00:27.4067387Z dashboard-tag: 2024-04-03T16:00:27.4067722Z s3-bucket: gha-artifacts 2024-04-03T16:00:27.4068112Z aws-role-to-assume: 2024-04-03T16:00:27.4068475Z ##[endgroup] 2024-04-03T16:00:27.4069244Z Complete job name: linux-jammy-py3.8-gcc11 / test (distributed, 2, 2, linux.2xlarge) 2024-04-03T16:00:27.4885079Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-04-03T16:00:27.4885684Z with: 2024-04-03T16:00:27.4886388Z github-secret: *** 2024-04-03T16:00:27.4887389Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-04-03T16:00:27.4888622Z activate-with-label: false 2024-04-03T16:00:27.4889003Z label: with-ssh 2024-04-03T16:00:27.4889351Z remove-existing-keys: true 2024-04-03T16:00:27.4889731Z env: 2024-04-03T16:00:27.4890029Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:00:27.4890438Z ##[endgroup] 2024-04-03T16:00:27.5850781Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-04-03T16:00:27.9082835Z Grabbing public ssh keys from https://github.com/caic99.keys 2024-04-03T16:00:27.9734660Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-04-03T16:00:27.9749891Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-04-03T16:00:27.9777330Z Login using: ssh ec2-user@ec2-3-83-22-9.compute-1.amazonaws.com 2024-04-03T16:00:27.9778741Z All testing is done inside the container, to start an interactive session run: 2024-04-03T16:00:27.9780234Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-04-03T16:00:28.0069890Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-04-03T16:00:28.0070566Z with: 2024-04-03T16:00:28.0070883Z submodules: recursive 2024-04-03T16:00:28.0071254Z fetch-depth: 0 2024-04-03T16:00:28.0071559Z env: 2024-04-03T16:00:28.0071859Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:00:28.0072236Z ##[endgroup] 2024-04-03T16:00:28.0288390Z ##[group]Run retry () { 2024-04-03T16:00:28.0288833Z retry () { 2024-04-03T16:00:28.0289362Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-04-03T16:00:28.0289959Z } 2024-04-03T16:00:28.0290296Z echo "${GITHUB_WORKSPACE}" 2024-04-03T16:00:28.0290754Z if [ -z "${NO_SUDO}" ]; then 2024-04-03T16:00:28.0291254Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-04-03T16:00:28.0291747Z else 2024-04-03T16:00:28.0292121Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-04-03T16:00:28.0292569Z fi 2024-04-03T16:00:28.0292962Z mkdir "${GITHUB_WORKSPACE}" 2024-04-03T16:00:28.0316893Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:00:28.0317449Z env: 2024-04-03T16:00:28.0317744Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:00:28.0318120Z NO_SUDO: 2024-04-03T16:00:28.0318430Z ##[endgroup] 2024-04-03T16:00:28.0411564Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:00:28.0617587Z ##[group]Run malfet/checkout@silent-checkout 2024-04-03T16:00:28.0618018Z with: 2024-04-03T16:00:28.0618316Z ref: d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:00:28.0618750Z fetch-depth: 0 2024-04-03T16:00:28.0619043Z submodules: recursive 2024-04-03T16:00:28.0619350Z quiet-checkout: true 2024-04-03T16:00:28.0619826Z repository: pytorch/pytorch 2024-04-03T16:00:28.0620336Z token: *** 2024-04-03T16:00:28.0620604Z ssh-strict: true 2024-04-03T16:00:28.0620894Z persist-credentials: true 2024-04-03T16:00:28.0621229Z clean: true 2024-04-03T16:00:28.0621539Z sparse-checkout-cone-mode: true 2024-04-03T16:00:28.0621903Z lfs: false 2024-04-03T16:00:28.0622181Z set-safe-directory: true 2024-04-03T16:00:28.0622492Z env: 2024-04-03T16:00:28.0622725Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:00:28.0623043Z ##[endgroup] 2024-04-03T16:00:28.1610582Z Syncing repository: pytorch/pytorch 2024-04-03T16:00:28.1613221Z ##[group]Getting Git version info 2024-04-03T16:00:28.1614515Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-04-03T16:00:28.1615995Z [command]/usr/bin/git version 2024-04-03T16:00:28.1616573Z git version 2.40.1 2024-04-03T16:00:28.1618714Z ##[endgroup] 2024-04-03T16:00:28.1633276Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/9b619bee-9ded-4784-bf31-6fb87993cca7' before making global git config changes 2024-04-03T16:00:28.1635500Z Adding repository directory to the temporary git global config as a safe directory 2024-04-03T16:00:28.1637619Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:00:28.1693528Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-04-03T16:00:28.1694722Z ##[group]Initializing the repository 2024-04-03T16:00:28.1695478Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:00:28.1773018Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-04-03T16:00:28.1773932Z hint: is subject to change. To configure the initial branch name to use in all 2024-04-03T16:00:28.1775255Z hint: of your new repositories, which will suppress this warning, call: 2024-04-03T16:00:28.1776219Z hint: 2024-04-03T16:00:28.1776926Z hint: git config --global init.defaultBranch 2024-04-03T16:00:28.1777725Z hint: 2024-04-03T16:00:28.1778594Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-04-03T16:00:28.1779989Z hint: 'development'. The just-created branch can be renamed via this command: 2024-04-03T16:00:28.1781249Z hint: 2024-04-03T16:00:28.1781821Z hint: git branch -m 2024-04-03T16:00:28.1783159Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-04-03T16:00:28.1785673Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-04-03T16:00:28.1812098Z ##[endgroup] 2024-04-03T16:00:28.1813038Z ##[group]Disabling automatic garbage collection 2024-04-03T16:00:28.1816357Z [command]/usr/bin/git config --local gc.auto 0 2024-04-03T16:00:28.1841893Z ##[endgroup] 2024-04-03T16:00:28.1842750Z ##[group]Setting up auth 2024-04-03T16:00:28.1848533Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-04-03T16:00:28.1876225Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-04-03T16:00:28.2100601Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-04-03T16:00:28.2133183Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-04-03T16:00:28.2363691Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-04-03T16:00:28.2405204Z ##[endgroup] 2024-04-03T16:00:28.2405743Z ##[group]Fetching the repository 2024-04-03T16:00:28.2411282Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-04-03T16:00:33.4496913Z remote: Enumerating objects: 1123757 2024-04-03T16:00:33.4497606Z remote: Enumerating objects: 1124401, done. 2024-04-03T16:00:33.4498754Z remote: Counting objects: 0% (1/644) 2024-04-03T16:00:33.4499428Z remote: Counting objects: 1% (7/644) 2024-04-03T16:00:33.4500164Z remote: Counting objects: 2% (13/644) 2024-04-03T16:00:33.4500925Z remote: Counting objects: 3% (20/644) 2024-04-03T16:00:33.4501662Z remote: Counting objects: 4% (26/644) 2024-04-03T16:00:33.4502396Z remote: Counting objects: 5% (33/644) 2024-04-03T16:00:33.4503103Z remote: Counting objects: 6% (39/644) 2024-04-03T16:00:33.4503818Z remote: Counting objects: 7% (46/644) 2024-04-03T16:00:33.4504629Z remote: Counting objects: 8% (52/644) 2024-04-03T16:00:33.4505461Z remote: Counting objects: 9% (58/644) 2024-04-03T16:00:33.4506254Z remote: Counting objects: 10% (65/644) 2024-04-03T16:00:33.4507027Z remote: Counting objects: 11% (71/644) 2024-04-03T16:00:33.4507819Z remote: Counting objects: 12% (78/644) 2024-04-03T16:00:33.4508542Z remote: Counting objects: 13% (84/644) 2024-04-03T16:00:33.4509235Z remote: Counting objects: 14% (91/644) 2024-04-03T16:00:33.4509907Z remote: Counting objects: 15% (97/644) 2024-04-03T16:00:33.4510616Z remote: Counting objects: 16% (104/644) 2024-04-03T16:00:33.4511342Z remote: Counting objects: 17% (110/644) 2024-04-03T16:00:33.4511877Z remote: Counting objects: 18% (116/644) 2024-04-03T16:00:33.4512344Z remote: Counting objects: 19% (123/644) 2024-04-03T16:00:33.4512824Z remote: Counting objects: 20% (129/644) 2024-04-03T16:00:33.4513308Z remote: Counting objects: 21% (136/644) 2024-04-03T16:00:33.4513793Z remote: Counting objects: 22% (142/644) 2024-04-03T16:00:33.4514259Z remote: Counting objects: 23% (149/644) 2024-04-03T16:00:33.4514733Z remote: Counting objects: 24% (155/644) 2024-04-03T16:00:33.4515210Z remote: Counting objects: 25% (161/644) 2024-04-03T16:00:33.4515674Z remote: Counting objects: 26% (168/644) 2024-04-03T16:00:33.4516160Z remote: Counting objects: 27% (174/644) 2024-04-03T16:00:33.4516918Z remote: Counting objects: 28% (181/644) 2024-04-03T16:00:33.4517568Z remote: Counting objects: 29% (187/644) 2024-04-03T16:00:33.4518056Z remote: Counting objects: 30% (194/644) 2024-04-03T16:00:33.4518539Z remote: Counting objects: 31% (200/644) 2024-04-03T16:00:33.4519007Z remote: Counting objects: 32% (207/644) 2024-04-03T16:00:33.4519492Z remote: Counting objects: 33% (213/644) 2024-04-03T16:00:33.4519976Z remote: Counting objects: 34% (219/644) 2024-04-03T16:00:33.4520498Z remote: Counting objects: 35% (226/644) 2024-04-03T16:00:33.4521052Z remote: Counting objects: 36% (232/644) 2024-04-03T16:00:33.4521551Z remote: Counting objects: 37% (239/644) 2024-04-03T16:00:33.4522272Z remote: Counting objects: 38% (245/644) 2024-04-03T16:00:33.4522744Z remote: Counting objects: 39% (252/644) 2024-04-03T16:00:33.4523308Z remote: Counting objects: 40% (258/644) 2024-04-03T16:00:33.4523865Z remote: Counting objects: 41% (265/644) 2024-04-03T16:00:33.4524339Z remote: Counting objects: 42% (271/644) 2024-04-03T16:00:33.4524821Z remote: Counting objects: 43% (277/644) 2024-04-03T16:00:33.4525304Z remote: Counting objects: 44% (284/644) 2024-04-03T16:00:33.4525775Z remote: Counting objects: 45% (290/644) 2024-04-03T16:00:33.4526259Z remote: Counting objects: 46% (297/644) 2024-04-03T16:00:33.4526742Z remote: Counting objects: 47% (303/644) 2024-04-03T16:00:33.4527224Z remote: Counting objects: 48% (310/644) 2024-04-03T16:00:33.4527691Z remote: Counting objects: 49% (316/644) 2024-04-03T16:00:33.4528175Z remote: Counting objects: 50% (322/644) 2024-04-03T16:00:33.4528656Z remote: Counting objects: 51% (329/644) 2024-04-03T16:00:33.4529126Z remote: Counting objects: 52% (335/644) 2024-04-03T16:00:33.4529609Z remote: Counting objects: 53% (342/644) 2024-04-03T16:00:33.4530095Z remote: Counting objects: 54% (348/644) 2024-04-03T16:00:33.4530569Z remote: Counting objects: 55% (355/644) 2024-04-03T16:00:33.4531052Z remote: Counting objects: 56% (361/644) 2024-04-03T16:00:33.4531535Z remote: Counting objects: 57% (368/644) 2024-04-03T16:00:33.4532142Z remote: Counting objects: 58% (374/644) 2024-04-03T16:00:33.4532625Z remote: Counting objects: 59% (380/644) 2024-04-03T16:00:33.4533103Z remote: Counting objects: 60% (387/644) 2024-04-03T16:00:33.4533569Z remote: Counting objects: 61% (393/644) 2024-04-03T16:00:33.4534054Z remote: Counting objects: 62% (400/644) 2024-04-03T16:00:33.4534603Z remote: Counting objects: 63% (406/644) 2024-04-03T16:00:33.4535088Z remote: Counting objects: 64% (413/644) 2024-04-03T16:00:33.4535558Z remote: Counting objects: 65% (419/644) 2024-04-03T16:00:33.4536040Z remote: Counting objects: 66% (426/644) 2024-04-03T16:00:33.4536532Z remote: Counting objects: 67% (432/644) 2024-04-03T16:00:33.4537356Z remote: Counting objects: 68% (438/644) 2024-04-03T16:00:33.4537844Z remote: Counting objects: 69% (445/644) 2024-04-03T16:00:33.4538335Z remote: Counting objects: 70% (451/644) 2024-04-03T16:00:33.4538805Z remote: Counting objects: 71% (458/644) 2024-04-03T16:00:33.4539291Z remote: Counting objects: 72% (464/644) 2024-04-03T16:00:33.4539775Z remote: Counting objects: 73% (471/644) 2024-04-03T16:00:33.4540245Z remote: Counting objects: 74% (477/644) 2024-04-03T16:00:33.4540726Z remote: Counting objects: 75% (483/644) 2024-04-03T16:00:33.4541212Z remote: Counting objects: 76% (490/644) 2024-04-03T16:00:33.4541694Z remote: Counting objects: 77% (496/644) 2024-04-03T16:00:33.4542164Z remote: Counting objects: 78% (503/644) 2024-04-03T16:00:33.4542787Z remote: Counting objects: 79% (509/644) 2024-04-03T16:00:33.4543339Z remote: Counting objects: 80% (516/644) 2024-04-03T16:00:33.4543893Z remote: Counting objects: 81% (522/644) 2024-04-03T16:00:33.4544489Z remote: Counting objects: 82% (529/644) 2024-04-03T16:00:33.4545045Z remote: Counting objects: 83% (535/644) 2024-04-03T16:00:33.4545516Z remote: Counting objects: 84% (541/644) 2024-04-03T16:00:33.4546113Z remote: Counting objects: 85% (548/644) 2024-04-03T16:00:33.4546692Z remote: Counting objects: 86% (554/644) 2024-04-03T16:00:33.4547158Z remote: Counting objects: 87% (561/644) 2024-04-03T16:00:33.4547643Z remote: Counting objects: 88% (567/644) 2024-04-03T16:00:33.4548125Z remote: Counting objects: 89% (574/644) 2024-04-03T16:00:33.4548591Z remote: Counting objects: 90% (580/644) 2024-04-03T16:00:33.4549068Z remote: Counting objects: 91% (587/644) 2024-04-03T16:00:33.4549549Z remote: Counting objects: 92% (593/644) 2024-04-03T16:00:33.4550124Z remote: Counting objects: 93% (599/644) 2024-04-03T16:00:33.4550616Z remote: Counting objects: 94% (606/644) 2024-04-03T16:00:33.4551108Z remote: Counting objects: 95% (612/644) 2024-04-03T16:00:33.4551579Z remote: Counting objects: 96% (619/644) 2024-04-03T16:00:33.4552065Z remote: Counting objects: 97% (625/644) 2024-04-03T16:00:33.4552544Z remote: Counting objects: 98% (632/644) 2024-04-03T16:00:33.4553025Z remote: Counting objects: 99% (638/644) 2024-04-03T16:00:33.4553495Z remote: Counting objects: 100% (644/644) 2024-04-03T16:00:33.4554011Z remote: Counting objects: 100% (644/644), done. 2024-04-03T16:00:33.4554559Z remote: Compressing objects: 0% (1/358) 2024-04-03T16:00:33.4603145Z remote: Compressing objects: 1% (4/358) 2024-04-03T16:00:33.4648863Z remote: Compressing objects: 2% (8/358) 2024-04-03T16:00:33.4852521Z remote: Compressing objects: 3% (11/358) 2024-04-03T16:00:33.4894470Z remote: Compressing objects: 4% (15/358) 2024-04-03T16:00:33.5230250Z remote: Compressing objects: 5% (18/358) 2024-04-03T16:00:33.5365229Z remote: Compressing objects: 6% (22/358) 2024-04-03T16:00:33.5439371Z remote: Compressing objects: 7% (26/358) 2024-04-03T16:00:33.5568241Z remote: Compressing objects: 8% (29/358) 2024-04-03T16:00:33.5589422Z remote: Compressing objects: 9% (33/358) 2024-04-03T16:00:33.5643666Z remote: Compressing objects: 10% (36/358) 2024-04-03T16:00:33.5688898Z remote: Compressing objects: 11% (40/358) 2024-04-03T16:00:33.5734678Z remote: Compressing objects: 12% (43/358) 2024-04-03T16:00:33.5775730Z remote: Compressing objects: 13% (47/358) 2024-04-03T16:00:33.5783160Z remote: Compressing objects: 14% (51/358) 2024-04-03T16:00:33.5791529Z remote: Compressing objects: 15% (54/358) 2024-04-03T16:00:33.5811253Z remote: Compressing objects: 16% (58/358) 2024-04-03T16:00:33.5832967Z remote: Compressing objects: 17% (61/358) 2024-04-03T16:00:33.5844909Z remote: Compressing objects: 18% (65/358) 2024-04-03T16:00:33.5849807Z remote: Compressing objects: 19% (69/358) 2024-04-03T16:00:33.5854992Z remote: Compressing objects: 20% (72/358) 2024-04-03T16:00:33.5857117Z remote: Compressing objects: 21% (76/358) 2024-04-03T16:00:33.5859462Z remote: Compressing objects: 22% (79/358) 2024-04-03T16:00:33.5862538Z remote: Compressing objects: 23% (83/358) 2024-04-03T16:00:33.5864481Z remote: Compressing objects: 24% (86/358) 2024-04-03T16:00:33.5874757Z remote: Compressing objects: 25% (90/358) 2024-04-03T16:00:33.5877276Z remote: Compressing objects: 26% (94/358) 2024-04-03T16:00:33.5884449Z remote: Compressing objects: 27% (97/358) 2024-04-03T16:00:33.5888700Z remote: Compressing objects: 28% (101/358) 2024-04-03T16:00:33.5895961Z remote: Compressing objects: 29% (104/358) 2024-04-03T16:00:33.5902315Z remote: Compressing objects: 30% (108/358) 2024-04-03T16:00:33.5905366Z remote: Compressing objects: 31% (111/358) 2024-04-03T16:00:33.5915989Z remote: Compressing objects: 32% (115/358) 2024-04-03T16:00:33.5921819Z remote: Compressing objects: 33% (119/358) 2024-04-03T16:00:33.5928135Z remote: Compressing objects: 34% (122/358) 2024-04-03T16:00:33.5934469Z remote: Compressing objects: 35% (126/358) 2024-04-03T16:00:33.5938734Z remote: Compressing objects: 36% (129/358) 2024-04-03T16:00:33.5941846Z remote: Compressing objects: 37% (133/358) 2024-04-03T16:00:33.5944330Z remote: Compressing objects: 38% (137/358) 2024-04-03T16:00:33.5947132Z remote: Compressing objects: 39% (140/358) 2024-04-03T16:00:33.5950766Z remote: Compressing objects: 40% (144/358) 2024-04-03T16:00:33.5957371Z remote: Compressing objects: 41% (147/358) 2024-04-03T16:00:33.5960375Z remote: Compressing objects: 42% (151/358) 2024-04-03T16:00:33.5965224Z remote: Compressing objects: 43% (154/358) 2024-04-03T16:00:33.5968043Z remote: Compressing objects: 44% (158/358) 2024-04-03T16:00:33.5970533Z remote: Compressing objects: 45% (162/358) 2024-04-03T16:00:33.5973946Z remote: Compressing objects: 46% (165/358) 2024-04-03T16:00:33.5976510Z remote: Compressing objects: 47% (169/358) 2024-04-03T16:00:33.5979075Z remote: Compressing objects: 48% (172/358) 2024-04-03T16:00:33.5981642Z remote: Compressing objects: 49% (176/358) 2024-04-03T16:00:33.5984290Z remote: Compressing objects: 50% (179/358) 2024-04-03T16:00:33.5984810Z remote: Compressing objects: 51% (183/358) 2024-04-03T16:00:33.5985327Z remote: Compressing objects: 52% (187/358) 2024-04-03T16:00:33.5987217Z remote: Compressing objects: 53% (190/358) 2024-04-03T16:00:33.5987735Z remote: Compressing objects: 54% (194/358) 2024-04-03T16:00:33.5988250Z remote: Compressing objects: 55% (197/358) 2024-04-03T16:00:33.5988792Z remote: Compressing objects: 56% (201/358) 2024-04-03T16:00:33.5989948Z remote: Compressing objects: 57% (205/358) 2024-04-03T16:00:33.5990551Z remote: Compressing objects: 58% (208/358) 2024-04-03T16:00:33.5991081Z remote: Compressing objects: 59% (212/358) 2024-04-03T16:00:33.5991712Z remote: Compressing objects: 60% (215/358) 2024-04-03T16:00:33.5992345Z remote: Compressing objects: 61% (219/358) 2024-04-03T16:00:33.5992864Z remote: Compressing objects: 62% (222/358) 2024-04-03T16:00:33.5993375Z remote: Compressing objects: 63% (226/358) 2024-04-03T16:00:33.5993874Z remote: Compressing objects: 64% (230/358) 2024-04-03T16:00:33.5994624Z remote: Compressing objects: 65% (233/358) 2024-04-03T16:00:33.5995135Z remote: Compressing objects: 66% (237/358) 2024-04-03T16:00:33.5995636Z remote: Compressing objects: 67% (240/358) 2024-04-03T16:00:33.5996162Z remote: Compressing objects: 68% (244/358) 2024-04-03T16:00:33.5997047Z remote: Compressing objects: 69% (248/358) 2024-04-03T16:00:33.5997603Z remote: Compressing objects: 70% (251/358) 2024-04-03T16:00:33.5999911Z remote: Compressing objects: 71% (255/358) 2024-04-03T16:00:33.6000437Z remote: Compressing objects: 72% (258/358) 2024-04-03T16:00:33.6002439Z remote: Compressing objects: 73% (262/358) 2024-04-03T16:00:33.6005098Z remote: Compressing objects: 74% (265/358) 2024-04-03T16:00:33.6007479Z remote: Compressing objects: 75% (269/358) 2024-04-03T16:00:33.6008004Z remote: Compressing objects: 76% (273/358) 2024-04-03T16:00:33.6010228Z remote: Compressing objects: 77% (276/358) 2024-04-03T16:00:33.6010744Z remote: Compressing objects: 78% (280/358) 2024-04-03T16:00:33.6012694Z remote: Compressing objects: 79% (283/358) 2024-04-03T16:00:33.6013228Z remote: Compressing objects: 80% (287/358) 2024-04-03T16:00:33.6013903Z remote: Compressing objects: 81% (290/358) 2024-04-03T16:00:33.6015342Z remote: Compressing objects: 82% (294/358) 2024-04-03T16:00:33.6015868Z remote: Compressing objects: 83% (298/358) 2024-04-03T16:00:33.6017741Z remote: Compressing objects: 84% (301/358) 2024-04-03T16:00:33.6020335Z remote: Compressing objects: 85% (305/358) 2024-04-03T16:00:33.6020860Z remote: Compressing objects: 86% (308/358) 2024-04-03T16:00:33.6021359Z remote: Compressing objects: 87% (312/358) 2024-04-03T16:00:33.6022971Z remote: Compressing objects: 88% (316/358) 2024-04-03T16:00:33.6023492Z remote: Compressing objects: 89% (319/358) 2024-04-03T16:00:33.6025681Z remote: Compressing objects: 90% (323/358) 2024-04-03T16:00:33.6026236Z remote: Compressing objects: 91% (326/358) 2024-04-03T16:00:33.6026759Z remote: Compressing objects: 92% (330/358) 2024-04-03T16:00:33.6027289Z remote: Compressing objects: 93% (333/358) 2024-04-03T16:00:33.6028276Z remote: Compressing objects: 94% (337/358) 2024-04-03T16:00:33.6028816Z remote: Compressing objects: 95% (341/358) 2024-04-03T16:00:33.6029332Z remote: Compressing objects: 96% (344/358) 2024-04-03T16:00:33.6030063Z remote: Compressing objects: 97% (348/358) 2024-04-03T16:00:33.6030806Z remote: Compressing objects: 98% (351/358) 2024-04-03T16:00:33.6031322Z remote: Compressing objects: 99% (355/358) 2024-04-03T16:00:33.6031837Z remote: Compressing objects: 100% (358/358) 2024-04-03T16:00:33.6032376Z remote: Compressing objects: 100% (358/358), done. 2024-04-03T16:00:54.8706947Z remote: Total 1124401 (delta 348), reused 520 (delta 286), pack-reused 1123757 2024-04-03T16:01:25.2921200Z [command]/usr/bin/git rev-parse --verify --quiet d253bd7befc64137de913dbb8727b511f1052411^{object} 2024-04-03T16:01:25.2943551Z d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:01:25.2948495Z ##[endgroup] 2024-04-03T16:01:25.2949073Z ##[group]Determining the checkout info 2024-04-03T16:01:25.2949823Z ##[endgroup] 2024-04-03T16:01:25.2950340Z ##[group]Checking out the ref 2024-04-03T16:01:25.2952072Z [command]/usr/bin/git checkout --quiet --force d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:01:26.6902202Z ##[endgroup] 2024-04-03T16:01:26.6902791Z ##[group]Setting up auth for fetching submodules 2024-04-03T16:01:26.6906538Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-04-03T16:01:26.6951296Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-04-03T16:01:26.6977741Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-04-03T16:01:26.7003022Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-04-03T16:01:26.7026077Z ##[endgroup] 2024-04-03T16:01:26.7026579Z ##[group]Fetching submodules 2024-04-03T16:01:26.7029803Z [command]/usr/bin/git submodule sync --recursive 2024-04-03T16:01:26.7283466Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-04-03T16:01:26.7530945Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-04-03T16:01:26.7533921Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-04-03T16:01:26.7536841Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-04-03T16:01:26.7539697Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-04-03T16:01:26.7542397Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2024-04-03T16:01:26.7546154Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-04-03T16:01:26.7550204Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-04-03T16:01:26.7552860Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-04-03T16:01:26.7556937Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-04-03T16:01:26.7560499Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2024-04-03T16:01:26.7565327Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-04-03T16:01:26.7569000Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-04-03T16:01:26.7573712Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-04-03T16:01:26.7578064Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-04-03T16:01:26.7582884Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-04-03T16:01:26.7587586Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-04-03T16:01:26.7592429Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-04-03T16:01:26.7597748Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:01:26.7602665Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-04-03T16:01:26.7608031Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-04-03T16:01:26.7612995Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-04-03T16:01:26.7618618Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2024-04-03T16:01:26.7623848Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-04-03T16:01:26.7629388Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-04-03T16:01:26.7635405Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-04-03T16:01:26.7641284Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-04-03T16:01:26.7647139Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2024-04-03T16:01:26.7653224Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-04-03T16:01:26.7659013Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-04-03T16:01:26.7665386Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2024-04-03T16:01:26.7673005Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-04-03T16:01:26.7679575Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-04-03T16:01:26.7686194Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-04-03T16:01:26.7693303Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-04-03T16:01:26.7699718Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-04-03T16:01:26.7706633Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-04-03T16:01:26.7713441Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-04-03T16:01:26.7720153Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2024-04-03T16:01:26.7727683Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-04-03T16:01:26.7734701Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2024-04-03T16:01:26.7757967Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-04-03T16:01:27.0073037Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-04-03T16:01:27.1795645Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-04-03T16:01:27.3326541Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-04-03T16:01:27.5438897Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2024-04-03T16:01:27.7872529Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-04-03T16:01:29.9532747Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-04-03T16:01:42.1897353Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-04-03T16:01:42.5843765Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-04-03T16:01:43.1650212Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2024-04-03T16:01:44.6270829Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-04-03T16:01:45.7628217Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-04-03T16:01:48.2667362Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-04-03T16:01:54.5104160Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-04-03T16:01:55.6939635Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-04-03T16:01:57.3760340Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-04-03T16:01:58.8257079Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-04-03T16:01:59.0129247Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-04-03T16:01:59.4516264Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-04-03T16:01:59.7691241Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-04-03T16:02:01.1276118Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-04-03T16:02:01.5472715Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2024-04-03T16:02:01.7314814Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-04-03T16:02:01.9824798Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-04-03T16:02:03.6831068Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-04-03T16:02:04.3906557Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-04-03T16:02:05.0226561Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2024-04-03T16:02:05.3979239Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-04-03T16:02:11.9545993Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-04-03T16:02:14.5208065Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2024-04-03T16:02:15.3743740Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-04-03T16:02:15.6522749Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-04-03T16:02:24.2552495Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-04-03T16:02:24.4838627Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-04-03T16:02:24.7422309Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-04-03T16:02:25.7638020Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-04-03T16:02:26.1226909Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-04-03T16:02:26.7703828Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2024-04-03T16:02:28.7209352Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-04-03T16:02:29.2057779Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2024-04-03T16:02:31.7722989Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-04-03T16:02:31.7819724Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-04-03T16:02:31.7894614Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-04-03T16:02:31.8096358Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-04-03T16:02:31.8293487Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2024-04-03T16:02:31.8629435Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-04-03T16:02:32.6973245Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-04-03T16:02:32.7168576Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-04-03T16:02:32.8022221Z Submodule path 'third_party/cpuinfo': checked out 'd6860c477c99f1fce9e28eb206891af3c0e1a1d7' 2024-04-03T16:02:32.8333739Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2024-04-03T16:02:32.8576909Z Submodule path 'third_party/cudnn_frontend': checked out '150798fe976556078f443fdb059a1ff0361f58a2' 2024-04-03T16:02:33.3083913Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-04-03T16:02:33.5356560Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-04-03T16:02:33.5972962Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-04-03T16:02:33.5987010Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:02:33.5989231Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:02:33.5991943Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:02:33.5994450Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:02:33.5997458Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:02:33.6019996Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-04-03T16:02:34.7191076Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-04-03T16:02:35.4229347Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-04-03T16:02:37.9873096Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-04-03T16:02:39.2483684Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-04-03T16:02:39.6361393Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-04-03T16:02:39.7237823Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-04-03T16:02:40.0926212Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-04-03T16:02:40.1496966Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-04-03T16:02:40.1597293Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-04-03T16:02:40.2591588Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-04-03T16:02:40.2909494Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-04-03T16:02:40.2987662Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-04-03T16:02:40.3342071Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-04-03T16:02:40.3555711Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-04-03T16:02:40.3955159Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-04-03T16:02:40.4062876Z Submodule path 'third_party/ideep': checked out '8a6cc4e09dc509f04f83c085e38786b1fb44e14d' 2024-04-03T16:02:40.4075913Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-04-03T16:02:40.4095130Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-04-03T16:02:52.6660360Z Submodule path 'third_party/ideep/mkl-dnn': checked out '86e6af5974177e513fd3fee58425e1063e7f1361' 2024-04-03T16:02:52.6751693Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2024-04-03T16:02:52.6891380Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-04-03T16:02:52.7786140Z Submodule path 'third_party/kineto': checked out '8466a8b111b36dc725e6855d52a0b133d925a8e0' 2024-04-03T16:02:52.7801618Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:02:52.7803733Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:02:52.7805766Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:02:52.7826675Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-04-03T16:02:53.6592742Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-04-03T16:02:55.0737556Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-04-03T16:02:56.4537465Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-04-03T16:02:56.4550958Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:02:56.4553240Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:02:56.4555705Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:02:56.4558259Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:02:56.4560898Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:02:56.4563810Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:02:56.4566429Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:02:56.4569317Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:02:56.4592260Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-04-03T16:02:57.6281868Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-04-03T16:02:58.0340104Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-04-03T16:02:59.4470257Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-04-03T16:02:59.7887144Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-04-03T16:03:00.2257121Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-04-03T16:03:01.5632608Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-04-03T16:03:09.1975712Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-04-03T16:03:09.6473502Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-04-03T16:03:09.6627237Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-04-03T16:03:09.6965271Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-04-03T16:03:09.7074251Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-04-03T16:03:09.7086180Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:03:09.7107069Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-04-03T16:03:10.1224562Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-04-03T16:03:10.1380376Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-04-03T16:03:10.1734893Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-04-03T16:03:10.2690332Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-04-03T16:03:10.2827425Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-04-03T16:03:10.3159274Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-04-03T16:03:10.3678846Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-04-03T16:03:10.3989412Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-04-03T16:03:10.4187770Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-04-03T16:03:10.4311603Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2024-04-03T16:03:10.5273081Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-04-03T16:03:10.8101956Z Submodule path 'third_party/onnx': checked out 'ccde5da81388ffa770ca98b64e07f803ad089414' 2024-04-03T16:03:10.8132005Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-04-03T16:03:10.8133621Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-04-03T16:03:10.8156803Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-04-03T16:03:11.2918998Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-04-03T16:03:12.5015743Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-04-03T16:03:12.5304843Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-04-03T16:03:12.5467475Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2024-04-03T16:03:12.5481100Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:03:12.5499790Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2024-04-03T16:03:15.2141334Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2024-04-03T16:03:15.2159277Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:03:15.2161387Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:03:15.2184307Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2024-04-03T16:03:15.8157566Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2024-04-03T16:03:16.8144241Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2024-04-03T16:03:16.8798442Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2024-04-03T16:03:16.8811388Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:03:16.8832799Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2024-04-03T16:03:17.1264749Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-04-03T16:03:17.1345451Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-04-03T16:03:17.3685383Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-04-03T16:03:17.3703534Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:03:17.3705363Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-04-03T16:03:17.3728351Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-04-03T16:03:17.8414758Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-04-03T16:03:19.1233474Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-04-03T16:03:19.1887391Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-04-03T16:03:19.1961121Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-04-03T16:03:19.2062997Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-04-03T16:03:19.2349381Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-04-03T16:03:19.2601563Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-04-03T16:03:19.2963054Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-04-03T16:03:19.3929958Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2024-04-03T16:03:19.4155812Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-04-03T16:03:19.4169048Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:03:19.4171070Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:03:19.4173506Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:03:19.4176019Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:03:19.4197622Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-04-03T16:03:20.7724112Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-04-03T16:03:21.1791213Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-04-03T16:03:22.5305996Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-04-03T16:03:23.5763079Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-04-03T16:03:23.5889552Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-04-03T16:03:23.6507321Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-04-03T16:03:23.6752943Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-04-03T16:03:23.6764257Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:03:23.6784301Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-04-03T16:03:23.9438131Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-04-03T16:03:24.0787988Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2024-04-03T16:03:24.0818688Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-04-03T16:03:24.1076212Z Entering 'android/libs/fbjni' 2024-04-03T16:03:24.1108907Z Entering 'third_party/FP16' 2024-04-03T16:03:24.1141842Z Entering 'third_party/FXdiv' 2024-04-03T16:03:24.1174798Z Entering 'third_party/NNPACK' 2024-04-03T16:03:24.1209990Z Entering 'third_party/QNNPACK' 2024-04-03T16:03:24.1242545Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:03:24.1275737Z Entering 'third_party/XNNPACK' 2024-04-03T16:03:24.1324659Z Entering 'third_party/benchmark' 2024-04-03T16:03:24.1358273Z Entering 'third_party/cpuinfo' 2024-04-03T16:03:24.1392186Z Entering 'third_party/cub' 2024-04-03T16:03:24.1423971Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:03:24.1457370Z Entering 'third_party/cutlass' 2024-04-03T16:03:24.1497532Z Entering 'third_party/eigen' 2024-04-03T16:03:24.1532402Z Entering 'third_party/fbgemm' 2024-04-03T16:03:24.1566895Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:03:24.1598242Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:03:24.1629091Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:03:24.1667649Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:03:24.1699443Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:03:24.1732663Z Entering 'third_party/flatbuffers' 2024-04-03T16:03:24.1767677Z Entering 'third_party/fmt' 2024-04-03T16:03:24.1800437Z Entering 'third_party/foxi' 2024-04-03T16:03:24.1832954Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:03:24.1865007Z Entering 'third_party/gloo' 2024-04-03T16:03:24.1897986Z Entering 'third_party/googletest' 2024-04-03T16:03:24.1931538Z Entering 'third_party/ideep' 2024-04-03T16:03:24.1963870Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:03:24.2002798Z Entering 'third_party/ios-cmake' 2024-04-03T16:03:24.2036563Z Entering 'third_party/ittapi' 2024-04-03T16:03:24.2068926Z Entering 'third_party/kineto' 2024-04-03T16:03:24.2101685Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:03:24.2134472Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:03:24.2168254Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:03:24.2201135Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:03:24.2234475Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:03:24.2265897Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:03:24.2299312Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:03:24.2331788Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:03:24.2364346Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:03:24.2397832Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:03:24.2431805Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:03:24.2463471Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:03:24.2497924Z Entering 'third_party/mimalloc' 2024-04-03T16:03:24.2532480Z Entering 'third_party/nccl/nccl' 2024-04-03T16:03:24.2567268Z Entering 'third_party/neon2sse' 2024-04-03T16:03:24.2600349Z Entering 'third_party/nlohmann' 2024-04-03T16:03:24.2635911Z Entering 'third_party/onnx' 2024-04-03T16:03:24.2684743Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:03:24.2718223Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:03:24.2752388Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:03:24.2785042Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:03:24.2821432Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:03:24.2855099Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:03:24.2887738Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:03:24.2923201Z Entering 'third_party/pocketfft' 2024-04-03T16:03:24.2956317Z Entering 'third_party/protobuf' 2024-04-03T16:03:24.2992128Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:03:24.3024436Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:03:24.3058199Z Entering 'third_party/psimd' 2024-04-03T16:03:24.3091031Z Entering 'third_party/pthreadpool' 2024-04-03T16:03:24.3124853Z Entering 'third_party/pybind11' 2024-04-03T16:03:24.3158205Z Entering 'third_party/python-peachpy' 2024-04-03T16:03:24.3191223Z Entering 'third_party/sleef' 2024-04-03T16:03:24.3223680Z Entering 'third_party/tbb' 2024-04-03T16:03:24.3259974Z Entering 'third_party/tensorpipe' 2024-04-03T16:03:24.3293137Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:03:24.3325152Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:03:24.3357583Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:03:24.3388541Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:03:24.3419802Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:03:24.3454115Z Entering 'third_party/zstd' 2024-04-03T16:03:24.3498222Z ##[endgroup] 2024-04-03T16:03:24.3500134Z ##[group]Persisting credentials for submodules 2024-04-03T16:03:24.3505444Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-04-03T16:03:24.3758335Z Entering 'android/libs/fbjni' 2024-04-03T16:03:24.3798476Z Entering 'third_party/FP16' 2024-04-03T16:03:24.3838406Z Entering 'third_party/FXdiv' 2024-04-03T16:03:24.3879567Z Entering 'third_party/NNPACK' 2024-04-03T16:03:24.3921162Z Entering 'third_party/QNNPACK' 2024-04-03T16:03:24.3962562Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:03:24.4003938Z Entering 'third_party/XNNPACK' 2024-04-03T16:03:24.4059390Z Entering 'third_party/benchmark' 2024-04-03T16:03:24.4100508Z Entering 'third_party/cpuinfo' 2024-04-03T16:03:24.4141719Z Entering 'third_party/cub' 2024-04-03T16:03:24.4182268Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:03:24.4222282Z Entering 'third_party/cutlass' 2024-04-03T16:03:24.4269858Z Entering 'third_party/eigen' 2024-04-03T16:03:24.4312416Z Entering 'third_party/fbgemm' 2024-04-03T16:03:24.4353439Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:03:24.4392689Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:03:24.4433965Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:03:24.4479566Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:03:24.4519352Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:03:24.4559180Z Entering 'third_party/flatbuffers' 2024-04-03T16:03:24.4602169Z Entering 'third_party/fmt' 2024-04-03T16:03:24.4643155Z Entering 'third_party/foxi' 2024-04-03T16:03:24.4684205Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:03:24.4724913Z Entering 'third_party/gloo' 2024-04-03T16:03:24.4765203Z Entering 'third_party/googletest' 2024-04-03T16:03:24.4807247Z Entering 'third_party/ideep' 2024-04-03T16:03:24.4846430Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:03:24.4892180Z Entering 'third_party/ios-cmake' 2024-04-03T16:03:24.4932692Z Entering 'third_party/ittapi' 2024-04-03T16:03:24.4972625Z Entering 'third_party/kineto' 2024-04-03T16:03:24.5013025Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:03:24.5053005Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:03:24.5093625Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:03:24.5132831Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:03:24.5172799Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:03:24.5211373Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:03:24.5252827Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:03:24.5292609Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:03:24.5332895Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:03:24.5373888Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:03:24.5415665Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:03:24.5455672Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:03:24.5496253Z Entering 'third_party/mimalloc' 2024-04-03T16:03:24.5537154Z Entering 'third_party/nccl/nccl' 2024-04-03T16:03:24.5577468Z Entering 'third_party/neon2sse' 2024-04-03T16:03:24.5618198Z Entering 'third_party/nlohmann' 2024-04-03T16:03:24.5659552Z Entering 'third_party/onnx' 2024-04-03T16:03:24.5715862Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:03:24.5756448Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:03:24.5798982Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:03:24.5838928Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:03:24.5882737Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:03:24.5922545Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:03:24.5961319Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:03:24.6003829Z Entering 'third_party/pocketfft' 2024-04-03T16:03:24.6044965Z Entering 'third_party/protobuf' 2024-04-03T16:03:24.6087374Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:03:24.6126514Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:03:24.6167483Z Entering 'third_party/psimd' 2024-04-03T16:03:24.6209286Z Entering 'third_party/pthreadpool' 2024-04-03T16:03:24.6250062Z Entering 'third_party/pybind11' 2024-04-03T16:03:24.6290982Z Entering 'third_party/python-peachpy' 2024-04-03T16:03:24.6331593Z Entering 'third_party/sleef' 2024-04-03T16:03:24.6374153Z Entering 'third_party/tbb' 2024-04-03T16:03:24.6416356Z Entering 'third_party/tensorpipe' 2024-04-03T16:03:24.6456599Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:03:24.6497456Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:03:24.6537763Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:03:24.6578440Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:03:24.6616989Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:03:24.6658672Z Entering 'third_party/zstd' 2024-04-03T16:03:24.6713726Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-04-03T16:03:24.6962791Z Entering 'android/libs/fbjni' 2024-04-03T16:03:24.7000132Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-04-03T16:03:24.7014572Z Entering 'third_party/FP16' 2024-04-03T16:03:24.7052170Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-04-03T16:03:24.7065224Z Entering 'third_party/FXdiv' 2024-04-03T16:03:24.7103532Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-04-03T16:03:24.7118409Z Entering 'third_party/NNPACK' 2024-04-03T16:03:24.7157268Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-04-03T16:03:24.7172382Z Entering 'third_party/QNNPACK' 2024-04-03T16:03:24.7209136Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2024-04-03T16:03:24.7223659Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:03:24.7262488Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-04-03T16:03:24.7276806Z Entering 'third_party/XNNPACK' 2024-04-03T16:03:24.7315151Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-04-03T16:03:24.7345915Z Entering 'third_party/benchmark' 2024-04-03T16:03:24.7383960Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:03:24.7399227Z Entering 'third_party/cpuinfo' 2024-04-03T16:03:24.7437020Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-04-03T16:03:24.7452389Z Entering 'third_party/cub' 2024-04-03T16:03:24.7490901Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2024-04-03T16:03:24.7505667Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:03:24.7544591Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-04-03T16:03:24.7559120Z Entering 'third_party/cutlass' 2024-04-03T16:03:24.7598137Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-04-03T16:03:24.7620042Z Entering 'third_party/eigen' 2024-04-03T16:03:24.7658484Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-04-03T16:03:24.7675615Z Entering 'third_party/fbgemm' 2024-04-03T16:03:24.7713555Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-04-03T16:03:24.7728353Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:03:24.7767167Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-04-03T16:03:24.7780842Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:03:24.7819024Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-04-03T16:03:24.7833077Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:03:24.7870208Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-04-03T16:03:24.7890332Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:03:24.7927284Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:03:24.7941146Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:03:24.7979700Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-04-03T16:03:24.7994536Z Entering 'third_party/flatbuffers' 2024-04-03T16:03:24.8032542Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-04-03T16:03:24.8049791Z Entering 'third_party/fmt' 2024-04-03T16:03:24.8088463Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-04-03T16:03:24.8103302Z Entering 'third_party/foxi' 2024-04-03T16:03:24.8139793Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-04-03T16:03:24.8155158Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:03:24.8192910Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-04-03T16:03:24.8207680Z Entering 'third_party/gloo' 2024-04-03T16:03:24.8246033Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-04-03T16:03:24.8260807Z Entering 'third_party/googletest' 2024-04-03T16:03:24.8298897Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:03:24.8313116Z Entering 'third_party/ideep' 2024-04-03T16:03:24.8351181Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-04-03T16:03:24.8364599Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:03:24.8402361Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-04-03T16:03:24.8423616Z Entering 'third_party/ios-cmake' 2024-04-03T16:03:24.8461238Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2024-04-03T16:03:24.8475287Z Entering 'third_party/ittapi' 2024-04-03T16:03:24.8514607Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-04-03T16:03:24.8529850Z Entering 'third_party/kineto' 2024-04-03T16:03:24.8568239Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-04-03T16:03:24.8582830Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:03:24.8620946Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-04-03T16:03:24.8635391Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:03:24.8675583Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-04-03T16:03:24.8690714Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:03:24.8728554Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-04-03T16:03:24.8742469Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:03:24.8781877Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-04-03T16:03:24.8795725Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:03:24.8834850Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-04-03T16:03:24.8847859Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:03:24.8887121Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-04-03T16:03:24.8902408Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:03:24.8940220Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-04-03T16:03:24.8955695Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:03:24.8994005Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:03:24.9008524Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:03:24.9047040Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-04-03T16:03:24.9062517Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:03:24.9100246Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-04-03T16:03:24.9116229Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:03:24.9153863Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-04-03T16:03:24.9168628Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:03:24.9207046Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-04-03T16:03:24.9222589Z Entering 'third_party/mimalloc' 2024-04-03T16:03:24.9260207Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-04-03T16:03:24.9277093Z Entering 'third_party/nccl/nccl' 2024-04-03T16:03:24.9314538Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-04-03T16:03:24.9329106Z Entering 'third_party/neon2sse' 2024-04-03T16:03:24.9366819Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2024-04-03T16:03:24.9381238Z Entering 'third_party/nlohmann' 2024-04-03T16:03:24.9418500Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-04-03T16:03:24.9434816Z Entering 'third_party/onnx' 2024-04-03T16:03:24.9472988Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-04-03T16:03:24.9502846Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:03:24.9540089Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:03:24.9554462Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:03:24.9591135Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:03:24.9606636Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:03:24.9645222Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2024-04-03T16:03:24.9658352Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:03:24.9695976Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2024-04-03T16:03:24.9714779Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:03:24.9753496Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:03:24.9766931Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:03:24.9804448Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:03:24.9817559Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:03:24.9855659Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-04-03T16:03:24.9873426Z Entering 'third_party/pocketfft' 2024-04-03T16:03:24.9911353Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-04-03T16:03:24.9924438Z Entering 'third_party/protobuf' 2024-04-03T16:03:24.9962733Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-04-03T16:03:24.9980464Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:03:25.0016875Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:03:25.0031110Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:03:25.0067813Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:03:25.0083709Z Entering 'third_party/psimd' 2024-04-03T16:03:25.0120998Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-04-03T16:03:25.0135433Z Entering 'third_party/pthreadpool' 2024-04-03T16:03:25.0172540Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-04-03T16:03:25.0187002Z Entering 'third_party/pybind11' 2024-04-03T16:03:25.0225102Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:03:25.0239763Z Entering 'third_party/python-peachpy' 2024-04-03T16:03:25.0276998Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-04-03T16:03:25.0291049Z Entering 'third_party/sleef' 2024-04-03T16:03:25.0328791Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-04-03T16:03:25.0343067Z Entering 'third_party/tbb' 2024-04-03T16:03:25.0381550Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2024-04-03T16:03:25.0397953Z Entering 'third_party/tensorpipe' 2024-04-03T16:03:25.0435558Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-04-03T16:03:25.0449279Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:03:25.0487267Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:03:25.0500630Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:03:25.0537612Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-04-03T16:03:25.0550713Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:03:25.0589012Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-04-03T16:03:25.0603292Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:03:25.0641228Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:03:25.0653478Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:03:25.0691153Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-04-03T16:03:25.0707563Z Entering 'third_party/zstd' 2024-04-03T16:03:25.0746086Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2024-04-03T16:03:25.1489952Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-04-03T16:03:25.1744403Z Entering 'android/libs/fbjni' 2024-04-03T16:03:25.1777837Z Entering 'third_party/FP16' 2024-04-03T16:03:25.1811259Z Entering 'third_party/FXdiv' 2024-04-03T16:03:25.1844412Z Entering 'third_party/NNPACK' 2024-04-03T16:03:25.1879292Z Entering 'third_party/QNNPACK' 2024-04-03T16:03:25.1912132Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:03:25.1944780Z Entering 'third_party/XNNPACK' 2024-04-03T16:03:25.1993932Z Entering 'third_party/benchmark' 2024-04-03T16:03:25.2028561Z Entering 'third_party/cpuinfo' 2024-04-03T16:03:25.2061694Z Entering 'third_party/cub' 2024-04-03T16:03:25.2095472Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:03:25.2129039Z Entering 'third_party/cutlass' 2024-04-03T16:03:25.2170484Z Entering 'third_party/eigen' 2024-04-03T16:03:25.2205242Z Entering 'third_party/fbgemm' 2024-04-03T16:03:25.2240477Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:03:25.2273024Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:03:25.2305530Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:03:25.2343609Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:03:25.2376427Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:03:25.2410000Z Entering 'third_party/flatbuffers' 2024-04-03T16:03:25.2445895Z Entering 'third_party/fmt' 2024-04-03T16:03:25.2479835Z Entering 'third_party/foxi' 2024-04-03T16:03:25.2512440Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:03:25.2545457Z Entering 'third_party/gloo' 2024-04-03T16:03:25.2578730Z Entering 'third_party/googletest' 2024-04-03T16:03:25.2612402Z Entering 'third_party/ideep' 2024-04-03T16:03:25.2645178Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:03:25.2684454Z Entering 'third_party/ios-cmake' 2024-04-03T16:03:25.2717490Z Entering 'third_party/ittapi' 2024-04-03T16:03:25.2750704Z Entering 'third_party/kineto' 2024-04-03T16:03:25.2784120Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:03:25.2816742Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:03:25.2849816Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:03:25.2882174Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:03:25.2915829Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:03:25.2947165Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:03:25.2980088Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:03:25.3013694Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:03:25.3046279Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:03:25.3079062Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:03:25.3112797Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:03:25.3145009Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:03:25.3178917Z Entering 'third_party/mimalloc' 2024-04-03T16:03:25.3212666Z Entering 'third_party/nccl/nccl' 2024-04-03T16:03:25.3245606Z Entering 'third_party/neon2sse' 2024-04-03T16:03:25.3278911Z Entering 'third_party/nlohmann' 2024-04-03T16:03:25.3314085Z Entering 'third_party/onnx' 2024-04-03T16:03:25.3362873Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:03:25.3395146Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:03:25.3429832Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:03:25.3462089Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:03:25.3499110Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:03:25.3532020Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:03:25.3564494Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:03:25.3600134Z Entering 'third_party/pocketfft' 2024-04-03T16:03:25.3633463Z Entering 'third_party/protobuf' 2024-04-03T16:03:25.3668746Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:03:25.3701811Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:03:25.3735576Z Entering 'third_party/psimd' 2024-04-03T16:03:25.3767573Z Entering 'third_party/pthreadpool' 2024-04-03T16:03:25.3800865Z Entering 'third_party/pybind11' 2024-04-03T16:03:25.3833665Z Entering 'third_party/python-peachpy' 2024-04-03T16:03:25.3866284Z Entering 'third_party/sleef' 2024-04-03T16:03:25.3898984Z Entering 'third_party/tbb' 2024-04-03T16:03:25.3934257Z Entering 'third_party/tensorpipe' 2024-04-03T16:03:25.3967876Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:03:25.3999970Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:03:25.4032040Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:03:25.4062917Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:03:25.4094423Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:03:25.4129150Z Entering 'third_party/zstd' 2024-04-03T16:03:25.4173742Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-04-03T16:03:25.4419680Z Entering 'android/libs/fbjni' 2024-04-03T16:03:25.4454720Z Entering 'third_party/FP16' 2024-04-03T16:03:25.4487625Z Entering 'third_party/FXdiv' 2024-04-03T16:03:25.4522562Z Entering 'third_party/NNPACK' 2024-04-03T16:03:25.4555680Z Entering 'third_party/QNNPACK' 2024-04-03T16:03:25.4587779Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:03:25.4621250Z Entering 'third_party/XNNPACK' 2024-04-03T16:03:25.4670391Z Entering 'third_party/benchmark' 2024-04-03T16:03:25.4703551Z Entering 'third_party/cpuinfo' 2024-04-03T16:03:25.4736124Z Entering 'third_party/cub' 2024-04-03T16:03:25.4769308Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:03:25.4802724Z Entering 'third_party/cutlass' 2024-04-03T16:03:25.4841679Z Entering 'third_party/eigen' 2024-04-03T16:03:25.4876835Z Entering 'third_party/fbgemm' 2024-04-03T16:03:25.4910259Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:03:25.4942121Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:03:25.4974128Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:03:25.5012834Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:03:25.5044826Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:03:25.5079510Z Entering 'third_party/flatbuffers' 2024-04-03T16:03:25.5114771Z Entering 'third_party/fmt' 2024-04-03T16:03:25.5147830Z Entering 'third_party/foxi' 2024-04-03T16:03:25.5181119Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:03:25.5212991Z Entering 'third_party/gloo' 2024-04-03T16:03:25.5246451Z Entering 'third_party/googletest' 2024-04-03T16:03:25.5279661Z Entering 'third_party/ideep' 2024-04-03T16:03:25.5312999Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:03:25.5351224Z Entering 'third_party/ios-cmake' 2024-04-03T16:03:25.5384283Z Entering 'third_party/ittapi' 2024-04-03T16:03:25.5416845Z Entering 'third_party/kineto' 2024-04-03T16:03:25.5450435Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:03:25.5482545Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:03:25.5515993Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:03:25.5548056Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:03:25.5579687Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:03:25.5611498Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:03:25.5644466Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:03:25.5677003Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:03:25.5708985Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:03:25.5742397Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:03:25.5775123Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:03:25.5807156Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:03:25.5840448Z Entering 'third_party/mimalloc' 2024-04-03T16:03:25.5874634Z Entering 'third_party/nccl/nccl' 2024-04-03T16:03:25.5907280Z Entering 'third_party/neon2sse' 2024-04-03T16:03:25.5939151Z Entering 'third_party/nlohmann' 2024-04-03T16:03:25.5973957Z Entering 'third_party/onnx' 2024-04-03T16:03:25.6020933Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:03:25.6052647Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:03:25.6088992Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:03:25.6121943Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:03:25.6156990Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:03:25.6188331Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:03:25.6220352Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:03:25.6255259Z Entering 'third_party/pocketfft' 2024-04-03T16:03:25.6288359Z Entering 'third_party/protobuf' 2024-04-03T16:03:25.6324703Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:03:25.6355666Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:03:25.6388605Z Entering 'third_party/psimd' 2024-04-03T16:03:25.6422541Z Entering 'third_party/pthreadpool' 2024-04-03T16:03:25.6454737Z Entering 'third_party/pybind11' 2024-04-03T16:03:25.6487638Z Entering 'third_party/python-peachpy' 2024-04-03T16:03:25.6521940Z Entering 'third_party/sleef' 2024-04-03T16:03:25.6554929Z Entering 'third_party/tbb' 2024-04-03T16:03:25.6590492Z Entering 'third_party/tensorpipe' 2024-04-03T16:03:25.6622816Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:03:25.6655018Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:03:25.6686489Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:03:25.6718684Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:03:25.6749765Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:03:25.6784480Z Entering 'third_party/zstd' 2024-04-03T16:03:25.6830690Z ##[endgroup] 2024-04-03T16:03:25.6864444Z [command]/usr/bin/git log -1 --format='%H' 2024-04-03T16:03:25.6888927Z 'd253bd7befc64137de913dbb8727b511f1052411' 2024-04-03T16:03:25.7051184Z Prepare all required actions 2024-04-03T16:03:25.7051707Z Getting action download info 2024-04-03T16:03:25.8800088Z ##[group]Run ./.github/actions/setup-linux 2024-04-03T16:03:25.8800638Z env: 2024-04-03T16:03:25.8801191Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:25.8801565Z ##[endgroup] 2024-04-03T16:03:25.8865503Z ##[group]Run set -euo pipefail 2024-04-03T16:03:25.8865974Z set -euo pipefail 2024-04-03T16:03:25.8866385Z function get_ec2_metadata() { 2024-04-03T16:03:25.8866956Z  # Pulled from instance metadata endpoint for EC2 2024-04-03T16:03:25.8867901Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-04-03T16:03:25.8868732Z  category=$1 2024-04-03T16:03:25.8869293Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-04-03T16:03:25.8870136Z  runner_name_str=i-0ae24b2353acec2d3 2024-04-03T16:03:25.8870735Z  if [[ $runner_name_str != *"gcp"* ]]; then 2024-04-03T16:03:25.8871417Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-04-03T16:03:25.8872034Z  else 2024-04-03T16:03:25.8872584Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-04-03T16:03:25.8873212Z  fi 2024-04-03T16:03:25.8873528Z } 2024-04-03T16:03:25.8873912Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-04-03T16:03:25.8874531Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-04-03T16:03:25.8875214Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-04-03T16:03:25.8875827Z echo "system info $(uname -a)" 2024-04-03T16:03:25.8883835Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:25.8884370Z env: 2024-04-03T16:03:25.8884662Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:25.8885037Z ##[endgroup] 2024-04-03T16:03:25.8992194Z ami-id: ami-0ce0c36d7a00b20e2 2024-04-03T16:03:25.9043288Z instance-id: i-0ae24b2353acec2d3 2024-04-03T16:03:25.9092239Z instance-type: c5.2xlarge 2024-04-03T16:03:25.9098982Z system info Linux ip-10-0-33-10.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-04-03T16:03:25.9122276Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-04-03T16:03:25.9123252Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-04-03T16:03:25.9130913Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:25.9131447Z env: 2024-04-03T16:03:25.9131749Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:25.9132122Z ##[endgroup] 2024-04-03T16:03:25.9198311Z ##[group]Run if systemctl is-active --quiet docker; then 2024-04-03T16:03:25.9198903Z if systemctl is-active --quiet docker; then 2024-04-03T16:03:25.9199424Z  echo "Docker daemon is running..."; 2024-04-03T16:03:25.9199878Z else 2024-04-03T16:03:25.9200500Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-04-03T16:03:25.9201065Z fi 2024-04-03T16:03:25.9207948Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:25.9208433Z env: 2024-04-03T16:03:25.9208687Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:25.9209000Z ##[endgroup] 2024-04-03T16:03:25.9311936Z Docker daemon is running... 2024-04-03T16:03:25.9357060Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-04-03T16:03:25.9357615Z with: 2024-04-03T16:03:25.9357860Z shell: bash 2024-04-03T16:03:25.9358245Z timeout_minutes: 5 2024-04-03T16:03:25.9358554Z max_attempts: 3 2024-04-03T16:03:25.9358854Z retry_wait_seconds: 30 2024-04-03T16:03:25.9360498Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-04-03T16:03:25.9361991Z polling_interval_seconds: 1 2024-04-03T16:03:25.9362360Z warning_on_retry: true 2024-04-03T16:03:25.9362701Z continue_on_error: false 2024-04-03T16:03:25.9363010Z env: 2024-04-03T16:03:25.9363269Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:25.9363616Z AWS_RETRY_MODE: standard 2024-04-03T16:03:25.9363942Z AWS_MAX_ATTEMPTS: 5 2024-04-03T16:03:25.9364275Z AWS_DEFAULT_REGION: us-east-1 2024-04-03T16:03:25.9364638Z ##[endgroup] 2024-04-03T16:03:27.1861356Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-04-03T16:03:27.1862757Z Configure a credential helper to remove this warning. See 2024-04-03T16:03:27.1863669Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-04-03T16:03:27.1864234Z 2024-04-03T16:03:27.1864360Z Login Succeeded 2024-04-03T16:03:27.9900104Z Command completed after 1 attempt(s). 2024-04-03T16:03:27.9945559Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-04-03T16:03:27.9946327Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-04-03T16:03:27.9947008Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-04-03T16:03:27.9954950Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:27.9955490Z env: 2024-04-03T16:03:27.9955795Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:27.9956155Z ##[endgroup] 2024-04-03T16:03:28.0027393Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T16:03:28.0028170Z # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T16:03:28.0028754Z # shellcheck disable=SC2046 2024-04-03T16:03:28.0029199Z docker stop $(docker ps -q) || true 2024-04-03T16:03:28.0029853Z # Prune all of the docker images 2024-04-03T16:03:28.0030278Z docker system prune -af 2024-04-03T16:03:28.0037313Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:28.0037805Z env: 2024-04-03T16:03:28.0038062Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:28.0038373Z ##[endgroup] 2024-04-03T16:03:28.0339219Z "docker stop" requires at least 1 argument. 2024-04-03T16:03:28.0339903Z See 'docker stop --help'. 2024-04-03T16:03:28.0340154Z 2024-04-03T16:03:28.0340402Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-04-03T16:03:28.0340794Z 2024-04-03T16:03:28.0340952Z Stop one or more running containers 2024-04-03T16:03:28.0503221Z Total reclaimed space: 0B 2024-04-03T16:03:28.0535987Z ##[group]Run set +e 2024-04-03T16:03:28.0536323Z set +e 2024-04-03T16:03:28.0536601Z set -x 2024-04-03T16:03:28.0536876Z  2024-04-03T16:03:28.0537186Z PT_DOMAIN=download.pytorch.org 2024-04-03T16:03:28.0537964Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-04-03T16:03:28.0539073Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-04-03T16:03:28.0539834Z # one is returned at random 2024-04-03T16:03:28.0540365Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-04-03T16:03:28.0540855Z  2024-04-03T16:03:28.0541150Z if [ -z "${RESOLVED_IP}" ]; then 2024-04-03T16:03:28.0541758Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-04-03T16:03:28.0542501Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-04-03T16:03:28.0543043Z  2024-04-03T16:03:28.0543349Z  if [ -z "${RESOLVED_IP}" ]; then 2024-04-03T16:03:28.0543987Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-04-03T16:03:28.0544484Z  exit 1 2024-04-03T16:03:28.0544994Z  fi 2024-04-03T16:03:28.0545248Z fi 2024-04-03T16:03:28.0545507Z  2024-04-03T16:03:28.0545846Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-04-03T16:03:28.0546437Z  # Clean up any old records first 2024-04-03T16:03:28.0546939Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-04-03T16:03:28.0547377Z fi 2024-04-03T16:03:28.0547620Z  2024-04-03T16:03:28.0548038Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-04-03T16:03:28.0548576Z cat /etc/hosts 2024-04-03T16:03:28.0556236Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:28.0556710Z env: 2024-04-03T16:03:28.0556968Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:28.0557292Z ##[endgroup] 2024-04-03T16:03:28.0579340Z + PT_DOMAIN=download.pytorch.org 2024-04-03T16:03:28.0584299Z ++ dig -4 +short download.pytorch.org 2024-04-03T16:03:28.0585019Z ++ tail -n1 2024-04-03T16:03:28.0857840Z + RESOLVED_IP=18.160.41.63 2024-04-03T16:03:28.0858335Z + '[' -z 18.160.41.63 ']' 2024-04-03T16:03:28.0858754Z + grep -r download.pytorch.org /etc/hosts 2024-04-03T16:03:28.0867219Z + echo '18.160.41.63 download.pytorch.org' 2024-04-03T16:03:28.0867725Z + sudo tee -a /etc/hosts 2024-04-03T16:03:28.0957303Z 18.160.41.63 download.pytorch.org 2024-04-03T16:03:28.0969795Z + cat /etc/hosts 2024-04-03T16:03:28.0975355Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-04-03T16:03:28.0987100Z ::1 localhost6 localhost6.localdomain6 2024-04-03T16:03:28.0987606Z 18.160.41.63 download.pytorch.org 2024-04-03T16:03:28.1214874Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-04-03T16:03:28.1215471Z with: 2024-04-03T16:03:28.1216345Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:28.1217389Z docker-build-dir: .ci/docker 2024-04-03T16:03:28.1217751Z working-directory: . 2024-04-03T16:03:28.1218193Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:28.1218715Z force-push: false 2024-04-03T16:03:28.1218991Z env: 2024-04-03T16:03:28.1219228Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:28.1219555Z ##[endgroup] 2024-04-03T16:03:28.1239101Z ##[group]Run set -ex 2024-04-03T16:03:28.1239652Z set -ex 2024-04-03T16:03:28.1239931Z  2024-04-03T16:03:28.1240560Z # If the docker build directory or the build script doesn't exist, the action will 2024-04-03T16:03:28.1241558Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-04-03T16:03:28.1242351Z # job could then download the pre-built image as usual 2024-04-03T16:03:28.1243092Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-04-03T16:03:28.1243767Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-04-03T16:03:28.1244377Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:03:28.1244930Z  2024-04-03T16:03:28.1245473Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-04-03T16:03:28.1246082Z  exit 0 2024-04-03T16:03:28.1246361Z else 2024-04-03T16:03:28.1246707Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-04-03T16:03:28.1247128Z fi 2024-04-03T16:03:28.1247381Z  2024-04-03T16:03:28.1247831Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-04-03T16:03:28.1248692Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-04-03T16:03:28.1249453Z  # use it as it is, but first let's extract the tag 2024-04-03T16:03:28.1250148Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-04-03T16:03:28.1250995Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:03:28.1251669Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:03:28.1252220Z else 2024-04-03T16:03:28.1252640Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-04-03T16:03:28.1253278Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:03:28.1254182Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:03:28.1254941Z fi 2024-04-03T16:03:28.1262268Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:28.1262756Z env: 2024-04-03T16:03:28.1262997Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:28.1263332Z REPO_NAME: pytorch 2024-04-03T16:03:28.1264252Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:28.1265244Z DOCKER_BUILD_DIR: .ci/docker 2024-04-03T16:03:28.1265724Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:28.1266231Z ##[endgroup] 2024-04-03T16:03:28.1287664Z + [[ ! -d .ci/docker ]] 2024-04-03T16:03:28.1288061Z + [[ ! -f .ci/docker/build.sh ]] 2024-04-03T16:03:28.1288431Z + echo skip=false 2024-04-03T16:03:28.1290025Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-04-03T16:03:28.1294543Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:28.1296247Z ++ awk -F '[:,]' '{print $2}' 2024-04-03T16:03:28.1303818Z + DOCKER_TAG=16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:28.1304530Z + echo docker-tag=16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:28.1305930Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:28.1333005Z ##[group]Run set +e 2024-04-03T16:03:28.1333362Z set +e 2024-04-03T16:03:28.1333628Z set -x 2024-04-03T16:03:28.1333900Z  2024-04-03T16:03:28.1334142Z login() { 2024-04-03T16:03:28.1334785Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-04-03T16:03:28.1335499Z } 2024-04-03T16:03:28.1335750Z  2024-04-03T16:03:28.1336208Z retry () { 2024-04-03T16:03:28.1336600Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-04-03T16:03:28.1337034Z } 2024-04-03T16:03:28.1337274Z  2024-04-03T16:03:28.1337570Z retry login "${DOCKER_REGISTRY}" 2024-04-03T16:03:28.1337968Z  2024-04-03T16:03:28.1338415Z # Check if image already exists, if it does then skip building it 2024-04-03T16:03:28.1339110Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-04-03T16:03:28.1339601Z  exit 0 2024-04-03T16:03:28.1339863Z fi 2024-04-03T16:03:28.1340110Z  2024-04-03T16:03:28.1340590Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-04-03T16:03:28.1341404Z # be empty. The default action would be to continue rebuild the image 2024-04-03T16:03:28.1342132Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-04-03T16:03:28.1342792Z  # if we're on the base branch then use the parent commit 2024-04-03T16:03:28.1343352Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-04-03T16:03:28.1343765Z else 2024-04-03T16:03:28.1344207Z  # otherwise we're on a PR, so use the most recent base commit 2024-04-03T16:03:28.1344885Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-04-03T16:03:28.1345486Z fi 2024-04-03T16:03:28.1345746Z  2024-04-03T16:03:28.1346045Z if [[ -z "${MERGE_BASE}" ]]; then 2024-04-03T16:03:28.1346519Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-04-03T16:03:28.1346965Z  2024-04-03T16:03:28.1347614Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-04-03T16:03:28.1348364Z  exit 0 2024-04-03T16:03:28.1348639Z fi 2024-04-03T16:03:28.1348887Z  2024-04-03T16:03:28.1349284Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-04-03T16:03:28.1350438Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-04-03T16:03:28.1351245Z  exit 1 2024-04-03T16:03:28.1351511Z fi 2024-04-03T16:03:28.1351766Z  2024-04-03T16:03:28.1352246Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-04-03T16:03:28.1353192Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-04-03T16:03:28.1354045Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-04-03T16:03:28.1355014Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-04-03T16:03:28.1356119Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-04-03T16:03:28.1356752Z fi 2024-04-03T16:03:28.1357006Z  2024-04-03T16:03:28.1357332Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-04-03T16:03:28.1364784Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:28.1365269Z env: 2024-04-03T16:03:28.1365529Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:28.1365874Z DOCKER_BUILD_DIR: .ci/docker 2024-04-03T16:03:28.1366312Z BASE_REVISION: 1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:03:28.1367379Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:28.1368418Z DOCKER_TAG: 16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:28.1369004Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:28.1369496Z ##[endgroup] 2024-04-03T16:03:28.1389401Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:28.1390240Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:28.1391673Z + aws ecr get-login-password --region us-east-1 2024-04-03T16:03:28.1392863Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:28.5192114Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-04-03T16:03:28.5193270Z Configure a credential helper to remove this warning. See 2024-04-03T16:03:28.5194647Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-04-03T16:03:28.5195223Z 2024-04-03T16:03:28.5195331Z Login Succeeded 2024-04-03T16:03:28.5205976Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:28.7270276Z { 2024-04-03T16:03:28.7270653Z "schemaVersion": 2, 2024-04-03T16:03:28.7271422Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-04-03T16:03:28.7272304Z "config": { 2024-04-03T16:03:28.7273075Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-04-03T16:03:28.7274088Z "size": 40203, 2024-04-03T16:03:28.7275158Z "digest": "sha256:edab37a6fecd099844f4888cdb58ab57de8cc1ffb0964933664a10a6b9fa73a8" 2024-04-03T16:03:28.7276077Z }, 2024-04-03T16:03:28.7276442Z "layers": [ 2024-04-03T16:03:28.7276909Z { 2024-04-03T16:03:28.7277618Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7278722Z "size": 30451302, 2024-04-03T16:03:28.7279556Z "digest": "sha256:23828d760c7b04df02891af556c40ca44c2dd79d6837ea6f18fac24f4108448c" 2024-04-03T16:03:28.7280625Z }, 2024-04-03T16:03:28.7280989Z { 2024-04-03T16:03:28.7281670Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7282569Z "size": 1828, 2024-04-03T16:03:28.7283380Z "digest": "sha256:4c71b2109c4fce41644c534918307f4ec9dd5d74b2d1491181d8b896339253e9" 2024-04-03T16:03:28.7284357Z }, 2024-04-03T16:03:28.7284717Z { 2024-04-03T16:03:28.7285377Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7286304Z "size": 313213453, 2024-04-03T16:03:28.7287194Z "digest": "sha256:42a766fc09097cc12b6898fbc22c4b5afc61dc6e8cdca26fda1af54cd5808163" 2024-04-03T16:03:28.7288132Z }, 2024-04-03T16:03:28.7288465Z { 2024-04-03T16:03:28.7289113Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7290014Z "size": 803, 2024-04-03T16:03:28.7290922Z "digest": "sha256:0e2f2458798f790b54c7f9276d0e1a9a69ec59f7271c51c3fcae9227ad87e7eb" 2024-04-03T16:03:28.7292066Z }, 2024-04-03T16:03:28.7292378Z { 2024-04-03T16:03:28.7293177Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7294026Z "size": 106, 2024-04-03T16:03:28.7295025Z "digest": "sha256:52475bb51e22a833df2d915ac317fb98bfd19fee4ca169046b28174a752de665" 2024-04-03T16:03:28.7295938Z }, 2024-04-03T16:03:28.7296162Z { 2024-04-03T16:03:28.7296594Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7297590Z "size": 545, 2024-04-03T16:03:28.7298472Z "digest": "sha256:928ebba6ddfb9f4b945e7b6b29221dfe7b1831becab9dc384c852b9449db7229" 2024-04-03T16:03:28.7299598Z }, 2024-04-03T16:03:28.7300028Z { 2024-04-03T16:03:28.7300838Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7301539Z "size": 1240, 2024-04-03T16:03:28.7301867Z + exit 0 2024-04-03T16:03:28.7302374Z "digest": "sha256:71bfb06f00796d9766e8078ce0d22c3504c748cdda403b229e08b9f66c90bce7" 2024-04-03T16:03:28.7302995Z }, 2024-04-03T16:03:28.7303209Z { 2024-04-03T16:03:28.7303635Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7304189Z "size": 483, 2024-04-03T16:03:28.7304703Z "digest": "sha256:56090b14b977c9226e4e1d467688ba76f54434a7f44807fcfed50926a75aa080" 2024-04-03T16:03:28.7305306Z }, 2024-04-03T16:03:28.7305534Z { 2024-04-03T16:03:28.7305942Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7306492Z "size": 110313371, 2024-04-03T16:03:28.7307049Z "digest": "sha256:22343a243e5681e355fd8661325b3eedb736d5da7e950ccea5d37c9ee3bb3a52" 2024-04-03T16:03:28.7307647Z }, 2024-04-03T16:03:28.7307871Z { 2024-04-03T16:03:28.7308290Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7308823Z "size": 3706, 2024-04-03T16:03:28.7309344Z "digest": "sha256:c29bb23a799916209b3b5f78924aad9644913961489623775658d5e3786c1e03" 2024-04-03T16:03:28.7310130Z }, 2024-04-03T16:03:28.7310345Z { 2024-04-03T16:03:28.7310768Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7311319Z "size": 1904, 2024-04-03T16:03:28.7311857Z "digest": "sha256:84aa8c2bf621b344af8103f37e465c7c28753ecee0bb9418eeefb32f112a07b0" 2024-04-03T16:03:28.7312479Z }, 2024-04-03T16:03:28.7312704Z { 2024-04-03T16:03:28.7313111Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7313656Z "size": 700, 2024-04-03T16:03:28.7314184Z "digest": "sha256:869f7e29fe434c4e7c345a34b7dda28235a1e275755824a3b2faf073a4f64cdf" 2024-04-03T16:03:28.7314786Z }, 2024-04-03T16:03:28.7315009Z { 2024-04-03T16:03:28.7315430Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7315969Z "size": 2590646918, 2024-04-03T16:03:28.7316521Z "digest": "sha256:a35b44e43b45a2801de628c066941679429d266b5d0bcd155ade98198da98461" 2024-04-03T16:03:28.7317254Z }, 2024-04-03T16:03:28.7317467Z { 2024-04-03T16:03:28.7317893Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7318443Z "size": 32, 2024-04-03T16:03:28.7318968Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:28.7319592Z }, 2024-04-03T16:03:28.7319819Z { 2024-04-03T16:03:28.7320224Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7320857Z "size": 380, 2024-04-03T16:03:28.7321405Z "digest": "sha256:f7cdcec5e6209d6102c3247b2bddf556339da28d039352be239d287af12c3da7" 2024-04-03T16:03:28.7322017Z }, 2024-04-03T16:03:28.7322252Z { 2024-04-03T16:03:28.7322673Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7323208Z "size": 220395, 2024-04-03T16:03:28.7323761Z "digest": "sha256:76fd1df065af26b8c5cbc6b570fa6b05472bc0639b5a8546ac1a70fddba60ad1" 2024-04-03T16:03:28.7324391Z }, 2024-04-03T16:03:28.7324599Z { 2024-04-03T16:03:28.7325021Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7325564Z "size": 230, 2024-04-03T16:03:28.7326068Z "digest": "sha256:83315c4c7ef0a0b8556648bfcb8072517770229c65bd5374b453d5741fcd9e4f" 2024-04-03T16:03:28.7326668Z }, 2024-04-03T16:03:28.7326893Z { 2024-04-03T16:03:28.7327302Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7327851Z "size": 2691615, 2024-04-03T16:03:28.7328372Z "digest": "sha256:6460360cb1298811ed09c68381447bbab44626003d5320a65264e700538b0b0d" 2024-04-03T16:03:28.7328951Z }, 2024-04-03T16:03:28.7329174Z { 2024-04-03T16:03:28.7329686Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7330226Z "size": 1994, 2024-04-03T16:03:28.7330771Z "digest": "sha256:f5187f0f8da243ed7751f5eefa2cec05f1393803726eeee84e330057805ca678" 2024-04-03T16:03:28.7331394Z }, 2024-04-03T16:03:28.7331605Z { 2024-04-03T16:03:28.7332033Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7332580Z "size": 105, 2024-04-03T16:03:28.7333095Z "digest": "sha256:b1fe483228e92afe41514cbd9402f338907bd8f46304587c793f7d1606bb1a39" 2024-04-03T16:03:28.7333706Z }, 2024-04-03T16:03:28.7333928Z { 2024-04-03T16:03:28.7334334Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7334884Z "size": 563, 2024-04-03T16:03:28.7335414Z "digest": "sha256:f148109b25c63db922ba0015afc51f9b12bd576ad392435463f6c2ee325f24de" 2024-04-03T16:03:28.7336015Z }, 2024-04-03T16:03:28.7336240Z { 2024-04-03T16:03:28.7336660Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7337191Z "size": 32, 2024-04-03T16:03:28.7337729Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:28.7338351Z }, 2024-04-03T16:03:28.7338562Z { 2024-04-03T16:03:28.7338987Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7339536Z "size": 104, 2024-04-03T16:03:28.7340045Z "digest": "sha256:6e94329e7d9bf7082fc26cec2348784557d985302de815ad85bf204ae9fe67e3" 2024-04-03T16:03:28.7340656Z }, 2024-04-03T16:03:28.7340878Z { 2024-04-03T16:03:28.7341284Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7341831Z "size": 503, 2024-04-03T16:03:28.7342359Z "digest": "sha256:e418f79cac9b43921f1702331a3d93926e2e664608005326054dabcdae1ad4d4" 2024-04-03T16:03:28.7342956Z }, 2024-04-03T16:03:28.7343166Z { 2024-04-03T16:03:28.7343587Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7344136Z "size": 101109813, 2024-04-03T16:03:28.7344662Z "digest": "sha256:77938992fe7d6f9a688194fb8672562ebc57c33a75795b9e880e215974705c68" 2024-04-03T16:03:28.7346947Z }, 2024-04-03T16:03:28.7347314Z { 2024-04-03T16:03:28.7348036Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7349094Z "size": 109, 2024-04-03T16:03:28.7350129Z "digest": "sha256:dd295d40e381bdd36f41fe90b77efba1a050acf7f813197058bb3b925e774bd1" 2024-04-03T16:03:28.7351178Z }, 2024-04-03T16:03:28.7393179Z { 2024-04-03T16:03:28.7393972Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7395078Z "size": 557, 2024-04-03T16:03:28.7396042Z "digest": "sha256:6c196b06c1db85ebaab271581b67c41bf5f6beeab678f989ac6366ff5ee37649" 2024-04-03T16:03:28.7397255Z }, 2024-04-03T16:03:28.7397679Z { 2024-04-03T16:03:28.7398445Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7399363Z "size": 1139068, 2024-04-03T16:03:28.7400459Z "digest": "sha256:d2a4f8c972bba57f9ea2255f6631e9054cea5efdbf91f82a56ab7b897fa5c1d1" 2024-04-03T16:03:28.7401071Z }, 2024-04-03T16:03:28.7401288Z { 2024-04-03T16:03:28.7401700Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7402616Z "size": 103, 2024-04-03T16:03:28.7403508Z "digest": "sha256:e17fa7c51474c7c233f32368ce4a8a33e8ac0992e26c2d76479767026b44ceb3" 2024-04-03T16:03:28.7404719Z }, 2024-04-03T16:03:28.7405120Z { 2024-04-03T16:03:28.7405637Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7406188Z "size": 1489, 2024-04-03T16:03:28.7406722Z "digest": "sha256:cff02b807961d4ba1cd0afa55af83f4c5eecd17857db2049d7b8163477197a13" 2024-04-03T16:03:28.7407347Z }, 2024-04-03T16:03:28.7407578Z { 2024-04-03T16:03:28.7407985Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7408533Z "size": 461458425, 2024-04-03T16:03:28.7409231Z "digest": "sha256:e96da33903b7fc6a3bd99919e3008604c7b7912425f0821837508056cc42ac40" 2024-04-03T16:03:28.7409832Z }, 2024-04-03T16:03:28.7410056Z { 2024-04-03T16:03:28.7410478Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7411019Z "size": 160, 2024-04-03T16:03:28.7411552Z "digest": "sha256:42bd346d5d1c5a410024ee9ecf597a86fc80f47363b6170c7008a5bfde7a1935" 2024-04-03T16:03:28.7412167Z }, 2024-04-03T16:03:28.7412378Z { 2024-04-03T16:03:28.7412796Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7413343Z "size": 2210, 2024-04-03T16:03:28.7413867Z "digest": "sha256:f688bbce9791905b8e853b07768c0b16b689aff2fb5adf40b76836a55e856da8" 2024-04-03T16:03:28.7414483Z }, 2024-04-03T16:03:28.7414709Z { 2024-04-03T16:03:28.7415117Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7415664Z "size": 151, 2024-04-03T16:03:28.7416207Z "digest": "sha256:8fbc77b9694b8d91b1f39276723dece503921f6578be17b0b2dfaf43e04f316b" 2024-04-03T16:03:28.7416808Z }, 2024-04-03T16:03:28.7417033Z { 2024-04-03T16:03:28.7417452Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7417982Z "size": 648, 2024-04-03T16:03:28.7418534Z "digest": "sha256:e2a3bc47943f6d4122b5c4adbbfff57e0d9d734ba8fce407089307ecd51be02b" 2024-04-03T16:03:28.7419158Z }, 2024-04-03T16:03:28.7419369Z { 2024-04-03T16:03:28.7419788Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7420333Z "size": 32, 2024-04-03T16:03:28.7420856Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:28.7421477Z }, 2024-04-03T16:03:28.7421700Z { 2024-04-03T16:03:28.7422106Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7422651Z "size": 163, 2024-04-03T16:03:28.7423187Z "digest": "sha256:8237da8eed865d197e234faec5be6043bd60f07d7409b664bb649ed2cab44fef" 2024-04-03T16:03:28.7423804Z }, 2024-04-03T16:03:28.7424023Z { 2024-04-03T16:03:28.7424442Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7424974Z "size": 109, 2024-04-03T16:03:28.7425522Z "digest": "sha256:71bb79f7fa9caaf8e330a2d1f846acae32a8c4be0e87e1a7e1aab1c01153ddcd" 2024-04-03T16:03:28.7426271Z }, 2024-04-03T16:03:28.7426484Z { 2024-04-03T16:03:28.7426905Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7427455Z "size": 102, 2024-04-03T16:03:28.7427966Z "digest": "sha256:9688271757d9d394c509517c6cb13ec1cfcd741bd1ad31529b69aff129a3f553" 2024-04-03T16:03:28.7428567Z }, 2024-04-03T16:03:28.7428791Z { 2024-04-03T16:03:28.7429195Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7429928Z "size": 421, 2024-04-03T16:03:28.7430483Z "digest": "sha256:8fba31fce28cff8c2ff69af296d90a4bf77fa61d79ed329e345544d12a70f9f0" 2024-04-03T16:03:28.7431105Z }, 2024-04-03T16:03:28.7431328Z { 2024-04-03T16:03:28.7431758Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7432289Z "size": 32, 2024-04-03T16:03:28.7432820Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:28.7433446Z }, 2024-04-03T16:03:28.7433657Z { 2024-04-03T16:03:28.7434073Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7434621Z "size": 111, 2024-04-03T16:03:28.7435140Z "digest": "sha256:d2f8ea8e2cd5a274189517f6e9212602e45c45b6ad24d3e6efca36b641c3c711" 2024-04-03T16:03:28.7435754Z }, 2024-04-03T16:03:28.7435977Z { 2024-04-03T16:03:28.7436384Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7436928Z "size": 474, 2024-04-03T16:03:28.7437457Z "digest": "sha256:a337c4be3a83b6eb74f2d656fc330bf670263c5a4d580727840df8478a761897" 2024-04-03T16:03:28.7438059Z }, 2024-04-03T16:03:28.7438282Z { 2024-04-03T16:03:28.7438801Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7439338Z "size": 32, 2024-04-03T16:03:28.7439871Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:28.7440613Z }, 2024-04-03T16:03:28.7440827Z { 2024-04-03T16:03:28.7441238Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7441785Z "size": 112, 2024-04-03T16:03:28.7442302Z "digest": "sha256:e2be7086a133e82cb0b446c958db2bb6279d7c81594e240dd90bd96ca3f8d337" 2024-04-03T16:03:28.7442919Z }, 2024-04-03T16:03:28.7443141Z { 2024-04-03T16:03:28.7443540Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7444082Z "size": 564, 2024-04-03T16:03:28.7444623Z "digest": "sha256:e8b6b86420ea9bb0eb2bf0e269ae6323b5c83f3b9236e97dfb5dadfda69a8160" 2024-04-03T16:03:28.7445237Z }, 2024-04-03T16:03:28.7445460Z { 2024-04-03T16:03:28.7445874Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7446416Z "size": 43166471, 2024-04-03T16:03:28.7446940Z "digest": "sha256:4673681701f97c1326775e836dfad15804d325f29fa24d2144d0d32deb5bd456" 2024-04-03T16:03:28.7447534Z }, 2024-04-03T16:03:28.7447745Z { 2024-04-03T16:03:28.7448164Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7448708Z "size": 106, 2024-04-03T16:03:28.7449227Z "digest": "sha256:402352c3f6525cefd02a2ffb8975241b7f1ea3fef56fed20d1fb3ee2b562fa2b" 2024-04-03T16:03:28.7449847Z }, 2024-04-03T16:03:28.7450068Z { 2024-04-03T16:03:28.7450470Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7451049Z "size": 295, 2024-04-03T16:03:28.7451573Z "digest": "sha256:d6271b2f16b74d1b43a4b796fb74885f0065b4045389a65007b5e5ca6517fa87" 2024-04-03T16:03:28.7452164Z }, 2024-04-03T16:03:28.7452390Z { 2024-04-03T16:03:28.7452811Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7453353Z "size": 32, 2024-04-03T16:03:28.7453883Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:28.7454506Z }, 2024-04-03T16:03:28.7454715Z { 2024-04-03T16:03:28.7455128Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7455789Z "size": 106, 2024-04-03T16:03:28.7456310Z "digest": "sha256:0a60eb3db52708e5fe6dc16a141ef3fec47c615c96b1311298120b3e2fa9ead6" 2024-04-03T16:03:28.7456930Z }, 2024-04-03T16:03:28.7457163Z { 2024-04-03T16:03:28.7457565Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7458112Z "size": 425, 2024-04-03T16:03:28.7458634Z "digest": "sha256:5b9f4e04399989649963f72ecbcd3257433a55da6db030f52b4d1d1f3bffc1b4" 2024-04-03T16:03:28.7459228Z }, 2024-04-03T16:03:28.7459450Z { 2024-04-03T16:03:28.7459868Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7460403Z "size": 19308750, 2024-04-03T16:03:28.7460952Z "digest": "sha256:3316bc233a5488f32dcd31964f191c1844dade9b6dc226b8da9db9057c5f9f63" 2024-04-03T16:03:28.7461563Z }, 2024-04-03T16:03:28.7461771Z { 2024-04-03T16:03:28.7462182Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7462731Z "size": 108, 2024-04-03T16:03:28.7463269Z "digest": "sha256:ee5d0a9a6a1eedcd98bf22747c17bdea3ca3e6e8979be9f6f0d6dca8655d3d39" 2024-04-03T16:03:28.7463906Z }, 2024-04-03T16:03:28.7464126Z { 2024-04-03T16:03:28.7464526Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7465069Z "size": 438, 2024-04-03T16:03:28.7465593Z "digest": "sha256:7396a8060fceee68a7f0d96ab5f17e2466f05430b1c4fab23241b780248b0e2e" 2024-04-03T16:03:28.7466192Z }, 2024-04-03T16:03:28.7466413Z { 2024-04-03T16:03:28.7466833Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7467364Z "size": 700, 2024-04-03T16:03:28.7467957Z "digest": "sha256:869f7e29fe434c4e7c345a34b7dda28235a1e275755824a3b2faf073a4f64cdf" 2024-04-03T16:03:28.7468568Z }, 2024-04-03T16:03:28.7468777Z { 2024-04-03T16:03:28.7469190Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7469975Z "size": 141, 2024-04-03T16:03:28.7470584Z "digest": "sha256:8f2884fd6deca001da0a08315864318acd90ef8f9bbd2f79b821b2e121ac04b4" 2024-04-03T16:03:28.7471205Z }, 2024-04-03T16:03:28.7471432Z { 2024-04-03T16:03:28.7471838Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7472381Z "size": 136, 2024-04-03T16:03:28.7472926Z "digest": "sha256:9b6932bd4bc842c9bafcdad19b3a9a796c16cbdad4d20caaf47002b8b74f148a" 2024-04-03T16:03:28.7473544Z }, 2024-04-03T16:03:28.7473767Z { 2024-04-03T16:03:28.7474182Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7474711Z "size": 32, 2024-04-03T16:03:28.7475241Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:28.7475867Z }, 2024-04-03T16:03:28.7476073Z { 2024-04-03T16:03:28.7476486Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7477030Z "size": 189, 2024-04-03T16:03:28.7477543Z "digest": "sha256:7066d7f0b725d30bf85e388a4e99fb73655e6ba43b948cf1be6aed407249e357" 2024-04-03T16:03:28.7478157Z }, 2024-04-03T16:03:28.7478377Z { 2024-04-03T16:03:28.7478779Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7479320Z "size": 1193, 2024-04-03T16:03:28.7479841Z "digest": "sha256:8a03304f5dc79086302ddbe9a18669358304a92ed8ade623494e72eeca414bd9" 2024-04-03T16:03:28.7480532Z }, 2024-04-03T16:03:28.7480746Z { 2024-04-03T16:03:28.7481169Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7481708Z "size": 700, 2024-04-03T16:03:28.7482232Z "digest": "sha256:869f7e29fe434c4e7c345a34b7dda28235a1e275755824a3b2faf073a4f64cdf" 2024-04-03T16:03:28.7482835Z }, 2024-04-03T16:03:28.7483049Z { 2024-04-03T16:03:28.7483462Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7484006Z "size": 137, 2024-04-03T16:03:28.7484517Z "digest": "sha256:63f01b7de7c9c0753c106e467eeac8c023357ae6dc402ca14015ef5d99de3da5" 2024-04-03T16:03:28.7485254Z }, 2024-04-03T16:03:28.7485484Z { 2024-04-03T16:03:28.7485886Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7486435Z "size": 1645357786, 2024-04-03T16:03:28.7486982Z "digest": "sha256:9c7bc24d85e16389f450276d031b09c515961f9f71baf9838a32ba99c64571ab" 2024-04-03T16:03:28.7487588Z }, 2024-04-03T16:03:28.7487800Z { 2024-04-03T16:03:28.7488215Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7488757Z "size": 154, 2024-04-03T16:03:28.7489266Z "digest": "sha256:0829c9fd377b7a48911dd53ddc38bf4a29c0d94c592184923f4195efa47a487f" 2024-04-03T16:03:28.7489874Z }, 2024-04-03T16:03:28.7490094Z { 2024-04-03T16:03:28.7490500Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7491042Z "size": 628, 2024-04-03T16:03:28.7491600Z "digest": "sha256:c0d204447609e55f6d096e704a166a553d3d0b9f6a2f2b1369389ea7856db263" 2024-04-03T16:03:28.7492188Z }, 2024-04-03T16:03:28.7492410Z { 2024-04-03T16:03:28.7492827Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7493360Z "size": 700, 2024-04-03T16:03:28.7493881Z "digest": "sha256:869f7e29fe434c4e7c345a34b7dda28235a1e275755824a3b2faf073a4f64cdf" 2024-04-03T16:03:28.7494484Z }, 2024-04-03T16:03:28.7494694Z { 2024-04-03T16:03:28.7495187Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7495751Z "size": 142, 2024-04-03T16:03:28.7496253Z "digest": "sha256:779e99646108784b34aa56e42f18349cdd830012c4bd3233e0e4d929ce272f35" 2024-04-03T16:03:28.7496921Z }, 2024-04-03T16:03:28.7497144Z { 2024-04-03T16:03:28.7497654Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7498204Z "size": 32, 2024-04-03T16:03:28.7498736Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:28.7499357Z }, 2024-04-03T16:03:28.7499568Z { 2024-04-03T16:03:28.7499995Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7500541Z "size": 158, 2024-04-03T16:03:28.7501071Z "digest": "sha256:80f4cc22eacdc87b398bbd2d5c658a78fa998805084868fddebfbfb46b1e0507" 2024-04-03T16:03:28.7501699Z }, 2024-04-03T16:03:28.7501919Z { 2024-04-03T16:03:28.7502320Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7502862Z "size": 1562, 2024-04-03T16:03:28.7503415Z "digest": "sha256:e0aa2cd02b55bb1e67ef7e0bfba2f8bf629e7b8f1b8a1cd7a63bc562805e4450" 2024-04-03T16:03:28.7504037Z }, 2024-04-03T16:03:28.7504258Z { 2024-04-03T16:03:28.7504670Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7505202Z "size": 32, 2024-04-03T16:03:28.7505774Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:28.7506532Z }, 2024-04-03T16:03:28.7506746Z { 2024-04-03T16:03:28.7507159Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7507709Z "size": 128, 2024-04-03T16:03:28.7508224Z "digest": "sha256:f064f2e8f1a86f5462546997b860cdff57ddcd2a21a4b3c86b0ab413eb173e22" 2024-04-03T16:03:28.7508833Z }, 2024-04-03T16:03:28.7509054Z { 2024-04-03T16:03:28.7509454Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7510139Z "size": 380, 2024-04-03T16:03:28.7510655Z "digest": "sha256:7874003851e3610104b2deaca398908082b276deac1baed8640e611c039721c7" 2024-04-03T16:03:28.7511243Z }, 2024-04-03T16:03:28.7511469Z { 2024-04-03T16:03:28.7511889Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7512418Z "size": 32, 2024-04-03T16:03:28.7512952Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:28.7513566Z }, 2024-04-03T16:03:28.7513773Z { 2024-04-03T16:03:28.7514186Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7514836Z "size": 104, 2024-04-03T16:03:28.7515354Z "digest": "sha256:57eb0af0150176410de409eebc484512635f93bd6d90e515dfe0705a6b59941e" 2024-04-03T16:03:28.7515963Z }, 2024-04-03T16:03:28.7516189Z { 2024-04-03T16:03:28.7516594Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7517142Z "size": 1840, 2024-04-03T16:03:28.7517668Z "digest": "sha256:111b525fbe5a364938574520a5fa49d6a3e3f53fcaf3cf5e415d93c9a1655b4d" 2024-04-03T16:03:28.7518262Z }, 2024-04-03T16:03:28.7518480Z { 2024-04-03T16:03:28.7518890Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7519419Z "size": 7529636, 2024-04-03T16:03:28.7519962Z "digest": "sha256:06eb833ae33bb5b9ac945f6d4588015261b4f9d6ce63809be582319c4d2ad0c6" 2024-04-03T16:03:28.7520650Z }, 2024-04-03T16:03:28.7520862Z { 2024-04-03T16:03:28.7521276Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7521819Z "size": 164, 2024-04-03T16:03:28.7522341Z "digest": "sha256:589a8ceece91b6ecad93fc270011293465962dde63d4e4380bc49901c962f636" 2024-04-03T16:03:28.7522955Z }, 2024-04-03T16:03:28.7523175Z { 2024-04-03T16:03:28.7523573Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7524118Z "size": 7938, 2024-04-03T16:03:28.7524642Z "digest": "sha256:a4814dd3f69b9429489be0ce78788fd65239f710ca52ddc34027145ea26a7545" 2024-04-03T16:03:28.7525230Z }, 2024-04-03T16:03:28.7525451Z { 2024-04-03T16:03:28.7525866Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7526397Z "size": 8068, 2024-04-03T16:03:28.7527102Z "digest": "sha256:1f18671b7fca523a587f03c066f4a857df8fceb3485bb8133be4f8ba46b113f4" 2024-04-03T16:03:28.7527728Z }, 2024-04-03T16:03:28.7527938Z { 2024-04-03T16:03:28.7528355Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7528906Z "size": 302, 2024-04-03T16:03:28.7529421Z "digest": "sha256:4002c8a87e7ae430b8882d8bb58e0c96616d7e574f1e4bffa35c05b4894e98ef" 2024-04-03T16:03:28.7530036Z }, 2024-04-03T16:03:28.7530258Z { 2024-04-03T16:03:28.7530660Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7531205Z "size": 32, 2024-04-03T16:03:28.7531735Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:28.7532340Z }, 2024-04-03T16:03:28.7532562Z { 2024-04-03T16:03:28.7532974Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7533506Z "size": 108, 2024-04-03T16:03:28.7534033Z "digest": "sha256:cc9527ae45fc240b262a1e61a4955e451b4680ae30c5bcb08d5bb3ea1b93c11e" 2024-04-03T16:03:28.7534646Z }, 2024-04-03T16:03:28.7534862Z { 2024-04-03T16:03:28.7535275Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:28.7535819Z "size": 54145478, 2024-04-03T16:03:28.7536364Z "digest": "sha256:b848d28edf10dd6dc82ce8b6a32dd20d6ff3c8662ba5932bc0d2d7e7bea131e6" 2024-04-03T16:03:28.7536996Z } 2024-04-03T16:03:28.7537219Z ] 2024-04-03T16:03:28.7537450Z } 2024-04-03T16:03:28.7648781Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-04-03T16:03:28.7649406Z tag=${ECR_DOCKER_IMAGE##*/} 2024-04-03T16:03:28.7649943Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-04-03T16:03:28.7657805Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:28.7658289Z env: 2024-04-03T16:03:28.7658546Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:28.7659465Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:28.7660441Z ##[endgroup] 2024-04-03T16:03:28.7682738Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-jammy-py3.8-gcc11-16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:28.7733925Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-04-03T16:03:28.7734570Z with: 2024-04-03T16:03:28.7735417Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:28.7736533Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:28.7737024Z env: 2024-04-03T16:03:28.7737278Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:28.7737600Z ##[endgroup] 2024-04-03T16:03:28.7754689Z ##[group]Run set -x 2024-04-03T16:03:28.7755013Z set -x 2024-04-03T16:03:28.7755289Z set +e 2024-04-03T16:03:28.7755555Z  2024-04-03T16:03:28.7755797Z login() { 2024-04-03T16:03:28.7756445Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-04-03T16:03:28.7757155Z } 2024-04-03T16:03:28.7757392Z  2024-04-03T16:03:28.7757692Z retry () { 2024-04-03T16:03:28.7758045Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-04-03T16:03:28.7758493Z } 2024-04-03T16:03:28.7758830Z  2024-04-03T16:03:28.7759105Z retry login "${DOCKER_REGISTRY}" 2024-04-03T16:03:28.7759497Z  2024-04-03T16:03:28.7759768Z set -e 2024-04-03T16:03:28.7760206Z # ignore output since only exit code is used for conditional 2024-04-03T16:03:28.7760984Z # only pull docker image if it's not available locally 2024-04-03T16:03:28.7761739Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-04-03T16:03:28.7762421Z  retry docker pull "${DOCKER_IMAGE}" 2024-04-03T16:03:28.7762827Z fi 2024-04-03T16:03:28.7769882Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:28.7770362Z env: 2024-04-03T16:03:28.7770611Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:28.7771526Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:28.7772623Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:28.7773132Z ##[endgroup] 2024-04-03T16:03:28.7792556Z + set +e 2024-04-03T16:03:28.7793159Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:28.7793803Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:28.7795294Z + aws ecr get-login-password --region us-east-1 2024-04-03T16:03:28.7796388Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:29.1631477Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-04-03T16:03:29.1632555Z Configure a credential helper to remove this warning. See 2024-04-03T16:03:29.1633407Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-04-03T16:03:29.1633946Z 2024-04-03T16:03:29.1634054Z Login Succeeded 2024-04-03T16:03:29.1642328Z + set -e 2024-04-03T16:03:29.1643617Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:29.1792876Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:29.1795819Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:29.3926423Z 16884143090f148aa56809ceba26eeb699b5dc49: Pulling from pytorch/pytorch-linux-jammy-py3.8-gcc11 2024-04-03T16:03:29.3946505Z 23828d760c7b: Pulling fs layer 2024-04-03T16:03:29.3950928Z 4c71b2109c4f: Pulling fs layer 2024-04-03T16:03:29.3951611Z 42a766fc0909: Pulling fs layer 2024-04-03T16:03:29.3952318Z 0e2f2458798f: Pulling fs layer 2024-04-03T16:03:29.3954719Z 52475bb51e22: Pulling fs layer 2024-04-03T16:03:29.3955655Z 928ebba6ddfb: Pulling fs layer 2024-04-03T16:03:29.3956304Z 71bfb06f0079: Pulling fs layer 2024-04-03T16:03:29.3956952Z 56090b14b977: Pulling fs layer 2024-04-03T16:03:29.3957795Z 22343a243e56: Pulling fs layer 2024-04-03T16:03:29.3958449Z c29bb23a7999: Pulling fs layer 2024-04-03T16:03:29.3959309Z 84aa8c2bf621: Pulling fs layer 2024-04-03T16:03:29.3959973Z 869f7e29fe43: Pulling fs layer 2024-04-03T16:03:29.3960725Z a35b44e43b45: Pulling fs layer 2024-04-03T16:03:29.3961415Z 4f4fb700ef54: Pulling fs layer 2024-04-03T16:03:29.3962099Z f7cdcec5e620: Pulling fs layer 2024-04-03T16:03:29.3962754Z 76fd1df065af: Pulling fs layer 2024-04-03T16:03:29.3963375Z 56090b14b977: Waiting 2024-04-03T16:03:29.3963856Z 928ebba6ddfb: Waiting 2024-04-03T16:03:29.3964149Z 0e2f2458798f: Waiting 2024-04-03T16:03:29.3964441Z 22343a243e56: Waiting 2024-04-03T16:03:29.3964737Z 71bfb06f0079: Waiting 2024-04-03T16:03:29.3965017Z 52475bb51e22: Waiting 2024-04-03T16:03:29.3965317Z f7cdcec5e620: Waiting 2024-04-03T16:03:29.3965652Z 83315c4c7ef0: Pulling fs layer 2024-04-03T16:03:29.3965981Z c29bb23a7999: Waiting 2024-04-03T16:03:29.3966278Z a35b44e43b45: Waiting 2024-04-03T16:03:29.3966582Z 84aa8c2bf621: Waiting 2024-04-03T16:03:29.3966863Z 4f4fb700ef54: Waiting 2024-04-03T16:03:29.3967157Z 869f7e29fe43: Waiting 2024-04-03T16:03:29.3967451Z 76fd1df065af: Waiting 2024-04-03T16:03:29.3967747Z 6460360cb129: Pulling fs layer 2024-04-03T16:03:29.3968111Z f5187f0f8da2: Pulling fs layer 2024-04-03T16:03:29.3968474Z b1fe483228e9: Pulling fs layer 2024-04-03T16:03:29.3968819Z f148109b25c6: Pulling fs layer 2024-04-03T16:03:29.3969178Z 6e94329e7d9b: Pulling fs layer 2024-04-03T16:03:29.3969518Z 83315c4c7ef0: Waiting 2024-04-03T16:03:29.3969819Z e418f79cac9b: Pulling fs layer 2024-04-03T16:03:29.3970162Z 6460360cb129: Waiting 2024-04-03T16:03:29.3970458Z b1fe483228e9: Waiting 2024-04-03T16:03:29.3970756Z 77938992fe7d: Pulling fs layer 2024-04-03T16:03:29.3971115Z dd295d40e381: Pulling fs layer 2024-04-03T16:03:29.3971458Z f148109b25c6: Waiting 2024-04-03T16:03:29.3971784Z 6e94329e7d9b: Waiting 2024-04-03T16:03:29.3972081Z e418f79cac9b: Waiting 2024-04-03T16:03:29.3972375Z 77938992fe7d: Waiting 2024-04-03T16:03:29.3972678Z 6c196b06c1db: Pulling fs layer 2024-04-03T16:03:29.3973045Z d2a4f8c972bb: Pulling fs layer 2024-04-03T16:03:29.3973408Z e17fa7c51474: Pulling fs layer 2024-04-03T16:03:29.3973736Z dd295d40e381: Waiting 2024-04-03T16:03:29.3974049Z cff02b807961: Pulling fs layer 2024-04-03T16:03:29.3974393Z d2a4f8c972bb: Waiting 2024-04-03T16:03:29.3974693Z e96da33903b7: Pulling fs layer 2024-04-03T16:03:29.3975033Z 6c196b06c1db: Waiting 2024-04-03T16:03:29.3975348Z 42bd346d5d1c: Pulling fs layer 2024-04-03T16:03:29.3975680Z cff02b807961: Waiting 2024-04-03T16:03:29.3975992Z f688bbce9791: Pulling fs layer 2024-04-03T16:03:29.3976338Z e17fa7c51474: Waiting 2024-04-03T16:03:29.3976621Z e96da33903b7: Waiting 2024-04-03T16:03:29.3976936Z 8fbc77b9694b: Pulling fs layer 2024-04-03T16:03:29.3977300Z e2a3bc47943f: Pulling fs layer 2024-04-03T16:03:29.3977638Z 42bd346d5d1c: Waiting 2024-04-03T16:03:29.3977950Z 8237da8eed86: Pulling fs layer 2024-04-03T16:03:29.3978312Z 71bb79f7fa9c: Pulling fs layer 2024-04-03T16:03:29.3978746Z 9688271757d9: Pulling fs layer 2024-04-03T16:03:29.3979091Z 8fbc77b9694b: Waiting 2024-04-03T16:03:29.3979407Z 8fba31fce28c: Pulling fs layer 2024-04-03T16:03:29.3979739Z f688bbce9791: Waiting 2024-04-03T16:03:29.3980056Z d2f8ea8e2cd5: Pulling fs layer 2024-04-03T16:03:29.3980405Z e2a3bc47943f: Waiting 2024-04-03T16:03:29.3980706Z a337c4be3a83: Pulling fs layer 2024-04-03T16:03:29.3981050Z 8237da8eed86: Waiting 2024-04-03T16:03:29.3981360Z e2be7086a133: Pulling fs layer 2024-04-03T16:03:29.3981820Z e8b6b86420ea: Pulling fs layer 2024-04-03T16:03:29.3982164Z 71bb79f7fa9c: Waiting 2024-04-03T16:03:29.3982610Z 4673681701f9: Pulling fs layer 2024-04-03T16:03:29.3982956Z 402352c3f652: Pulling fs layer 2024-04-03T16:03:29.3983353Z d6271b2f16b7: Pulling fs layer 2024-04-03T16:03:29.3983701Z 9688271757d9: Waiting 2024-04-03T16:03:29.3984023Z 8fba31fce28c: Waiting 2024-04-03T16:03:29.3984658Z 0a60eb3db527: Pulling fs layer 2024-04-03T16:03:29.3985027Z 5b9f4e043999: Pulling fs layer 2024-04-03T16:03:29.3985441Z d2f8ea8e2cd5: Waiting 2024-04-03T16:03:29.3985760Z 3316bc233a54: Pulling fs layer 2024-04-03T16:03:29.3986099Z 4673681701f9: Waiting 2024-04-03T16:03:29.3986383Z e8b6b86420ea: Waiting 2024-04-03T16:03:29.3986778Z ee5d0a9a6a1e: Pulling fs layer 2024-04-03T16:03:29.3987131Z a337c4be3a83: Waiting 2024-04-03T16:03:29.3987435Z 7396a8060fce: Pulling fs layer 2024-04-03T16:03:29.3987782Z e2be7086a133: Waiting 2024-04-03T16:03:29.3988132Z 402352c3f652: Waiting 2024-04-03T16:03:29.3988434Z 8f2884fd6dec: Pulling fs layer 2024-04-03T16:03:29.3988841Z d6271b2f16b7: Waiting 2024-04-03T16:03:29.3989219Z 9b6932bd4bc8: Pulling fs layer 2024-04-03T16:03:29.3989551Z 0a60eb3db527: Waiting 2024-04-03T16:03:29.3990065Z 5b9f4e043999: Waiting 2024-04-03T16:03:29.3990362Z 3316bc233a54: Waiting 2024-04-03T16:03:29.3990705Z 7066d7f0b725: Pulling fs layer 2024-04-03T16:03:29.3991065Z ee5d0a9a6a1e: Waiting 2024-04-03T16:03:29.3991364Z 7396a8060fce: Waiting 2024-04-03T16:03:29.3991646Z 8f2884fd6dec: Waiting 2024-04-03T16:03:29.3992094Z 8a03304f5dc7: Pulling fs layer 2024-04-03T16:03:29.3992458Z 63f01b7de7c9: Pulling fs layer 2024-04-03T16:03:29.3992787Z 7066d7f0b725: Waiting 2024-04-03T16:03:29.3993083Z 8a03304f5dc7: Waiting 2024-04-03T16:03:29.3993382Z 9c7bc24d85e1: Pulling fs layer 2024-04-03T16:03:29.3993744Z 0829c9fd377b: Pulling fs layer 2024-04-03T16:03:29.3994104Z c0d204447609: Pulling fs layer 2024-04-03T16:03:29.3994446Z 779e99646108: Pulling fs layer 2024-04-03T16:03:29.3994784Z 63f01b7de7c9: Waiting 2024-04-03T16:03:29.3995085Z 9c7bc24d85e1: Waiting 2024-04-03T16:03:29.3995388Z 80f4cc22eacd: Pulling fs layer 2024-04-03T16:03:29.3995754Z e0aa2cd02b55: Pulling fs layer 2024-04-03T16:03:29.3996115Z f064f2e8f1a8: Pulling fs layer 2024-04-03T16:03:29.3996447Z 80f4cc22eacd: Waiting 2024-04-03T16:03:29.3996760Z 7874003851e3: Pulling fs layer 2024-04-03T16:03:29.3997098Z e0aa2cd02b55: Waiting 2024-04-03T16:03:29.3997406Z 57eb0af01501: Pulling fs layer 2024-04-03T16:03:29.3997828Z 111b525fbe5a: Pulling fs layer 2024-04-03T16:03:29.3998232Z 7874003851e3: Waiting 2024-04-03T16:03:29.3998529Z 06eb833ae33b: Pulling fs layer 2024-04-03T16:03:29.3998892Z 589a8ceece91: Pulling fs layer 2024-04-03T16:03:29.3999250Z a4814dd3f69b: Pulling fs layer 2024-04-03T16:03:29.3999649Z 1f18671b7fca: Pulling fs layer 2024-04-03T16:03:29.3999991Z 57eb0af01501: Waiting 2024-04-03T16:03:29.4000404Z 4002c8a87e7a: Pulling fs layer 2024-04-03T16:03:29.4000737Z 111b525fbe5a: Waiting 2024-04-03T16:03:29.4001032Z 06eb833ae33b: Waiting 2024-04-03T16:03:29.4001345Z cc9527ae45fc: Pulling fs layer 2024-04-03T16:03:29.4001681Z 589a8ceece91: Waiting 2024-04-03T16:03:29.4001996Z b848d28edf10: Pulling fs layer 2024-04-03T16:03:29.4002341Z a4814dd3f69b: Waiting 2024-04-03T16:03:29.4002623Z 4002c8a87e7a: Waiting 2024-04-03T16:03:29.4002915Z cc9527ae45fc: Waiting 2024-04-03T16:03:29.4003209Z 1f18671b7fca: Waiting 2024-04-03T16:03:29.4003499Z b848d28edf10: Waiting 2024-04-03T16:03:29.4003791Z 0829c9fd377b: Waiting 2024-04-03T16:03:29.4892926Z 4c71b2109c4f: Verifying Checksum 2024-04-03T16:03:29.5545071Z 4c71b2109c4f: Download complete 2024-04-03T16:03:29.5545763Z 0e2f2458798f: Download complete 2024-04-03T16:03:29.6364733Z 52475bb51e22: Verifying Checksum 2024-04-03T16:03:29.6365447Z 52475bb51e22: Download complete 2024-04-03T16:03:29.7027794Z 928ebba6ddfb: Verifying Checksum 2024-04-03T16:03:29.7028489Z 928ebba6ddfb: Download complete 2024-04-03T16:03:29.7797395Z 71bfb06f0079: Verifying Checksum 2024-04-03T16:03:29.7797827Z 71bfb06f0079: Download complete 2024-04-03T16:03:29.8040133Z 23828d760c7b: Download complete 2024-04-03T16:03:29.8731132Z 56090b14b977: Verifying Checksum 2024-04-03T16:03:29.8731621Z 56090b14b977: Download complete 2024-04-03T16:03:29.9621352Z c29bb23a7999: Verifying Checksum 2024-04-03T16:03:29.9621806Z c29bb23a7999: Download complete 2024-04-03T16:03:30.1554272Z 869f7e29fe43: Verifying Checksum 2024-04-03T16:03:30.1555271Z 869f7e29fe43: Download complete 2024-04-03T16:03:30.5184525Z 23828d760c7b: Pull complete 2024-04-03T16:03:30.6075217Z 4c71b2109c4f: Pull complete 2024-04-03T16:03:30.9714175Z 22343a243e56: Verifying Checksum 2024-04-03T16:03:30.9714922Z 22343a243e56: Download complete 2024-04-03T16:03:30.9795683Z 4f4fb700ef54: Verifying Checksum 2024-04-03T16:03:30.9796164Z 4f4fb700ef54: Download complete 2024-04-03T16:03:31.0886284Z f7cdcec5e620: Download complete 2024-04-03T16:03:31.1601645Z 76fd1df065af: Verifying Checksum 2024-04-03T16:03:31.1602429Z 76fd1df065af: Download complete 2024-04-03T16:03:31.2264242Z 83315c4c7ef0: Verifying Checksum 2024-04-03T16:03:31.2265272Z 83315c4c7ef0: Download complete 2024-04-03T16:03:31.3212187Z 6460360cb129: Verifying Checksum 2024-04-03T16:03:31.3212967Z 6460360cb129: Download complete 2024-04-03T16:03:31.4210251Z f5187f0f8da2: Verifying Checksum 2024-04-03T16:03:31.4210803Z f5187f0f8da2: Download complete 2024-04-03T16:03:31.4942648Z b1fe483228e9: Download complete 2024-04-03T16:03:31.5766586Z f148109b25c6: Verifying Checksum 2024-04-03T16:03:31.5767185Z f148109b25c6: Download complete 2024-04-03T16:03:31.6663242Z 6e94329e7d9b: Verifying Checksum 2024-04-03T16:03:31.6663800Z 6e94329e7d9b: Download complete 2024-04-03T16:03:31.7430983Z e418f79cac9b: Verifying Checksum 2024-04-03T16:03:31.7461222Z e418f79cac9b: Download complete 2024-04-03T16:03:32.5904447Z 42a766fc0909: Verifying Checksum 2024-04-03T16:03:32.5905330Z 42a766fc0909: Download complete 2024-04-03T16:03:32.6717280Z dd295d40e381: Verifying Checksum 2024-04-03T16:03:32.6718164Z dd295d40e381: Download complete 2024-04-03T16:03:32.7663095Z 6c196b06c1db: Verifying Checksum 2024-04-03T16:03:32.7663871Z 6c196b06c1db: Download complete 2024-04-03T16:03:32.8205618Z 77938992fe7d: Verifying Checksum 2024-04-03T16:03:32.8206203Z 77938992fe7d: Download complete 2024-04-03T16:03:32.8542909Z d2a4f8c972bb: Verifying Checksum 2024-04-03T16:03:32.8543730Z d2a4f8c972bb: Download complete 2024-04-03T16:03:32.8937494Z e17fa7c51474: Verifying Checksum 2024-04-03T16:03:32.8938638Z e17fa7c51474: Download complete 2024-04-03T16:03:32.9458928Z cff02b807961: Verifying Checksum 2024-04-03T16:03:32.9459695Z cff02b807961: Download complete 2024-04-03T16:03:33.0316848Z 42bd346d5d1c: Verifying Checksum 2024-04-03T16:03:33.0317630Z 42bd346d5d1c: Download complete 2024-04-03T16:03:33.1177985Z f688bbce9791: Verifying Checksum 2024-04-03T16:03:33.1178568Z f688bbce9791: Download complete 2024-04-03T16:03:33.2141574Z 8fbc77b9694b: Verifying Checksum 2024-04-03T16:03:33.2142342Z 8fbc77b9694b: Download complete 2024-04-03T16:03:33.3031326Z e2a3bc47943f: Verifying Checksum 2024-04-03T16:03:33.3031811Z e2a3bc47943f: Download complete 2024-04-03T16:03:33.3917413Z 8237da8eed86: Verifying Checksum 2024-04-03T16:03:33.3917895Z 8237da8eed86: Download complete 2024-04-03T16:03:33.4641900Z 71bb79f7fa9c: Download complete 2024-04-03T16:03:33.5372822Z 9688271757d9: Verifying Checksum 2024-04-03T16:03:33.5373515Z 9688271757d9: Download complete 2024-04-03T16:03:33.6260694Z 8fba31fce28c: Verifying Checksum 2024-04-03T16:03:33.6261434Z 8fba31fce28c: Download complete 2024-04-03T16:03:33.7433764Z d2f8ea8e2cd5: Verifying Checksum 2024-04-03T16:03:33.7434634Z d2f8ea8e2cd5: Download complete 2024-04-03T16:03:33.8349831Z a337c4be3a83: Verifying Checksum 2024-04-03T16:03:33.9169615Z e2be7086a133: Verifying Checksum 2024-04-03T16:03:33.9170103Z e2be7086a133: Download complete 2024-04-03T16:03:34.0149519Z e8b6b86420ea: Verifying Checksum 2024-04-03T16:03:34.0150454Z e8b6b86420ea: Download complete 2024-04-03T16:03:34.5243891Z 4673681701f9: Verifying Checksum 2024-04-03T16:03:34.5244680Z 4673681701f9: Download complete 2024-04-03T16:03:34.6023915Z 402352c3f652: Verifying Checksum 2024-04-03T16:03:34.6024622Z 402352c3f652: Download complete 2024-04-03T16:03:34.6868939Z d6271b2f16b7: Verifying Checksum 2024-04-03T16:03:34.6869797Z d6271b2f16b7: Download complete 2024-04-03T16:03:34.7735692Z 0a60eb3db527: Verifying Checksum 2024-04-03T16:03:34.7736555Z 0a60eb3db527: Download complete 2024-04-03T16:03:34.8579496Z 5b9f4e043999: Download complete 2024-04-03T16:03:35.1139807Z 3316bc233a54: Verifying Checksum 2024-04-03T16:03:35.1141111Z 3316bc233a54: Download complete 2024-04-03T16:03:35.1981544Z ee5d0a9a6a1e: Download complete 2024-04-03T16:03:35.2822551Z 7396a8060fce: Verifying Checksum 2024-04-03T16:03:35.2823295Z 7396a8060fce: Download complete 2024-04-03T16:03:35.3723194Z 8f2884fd6dec: Verifying Checksum 2024-04-03T16:03:35.3723913Z 8f2884fd6dec: Download complete 2024-04-03T16:03:35.4418401Z 9b6932bd4bc8: Download complete 2024-04-03T16:03:35.5200946Z 7066d7f0b725: Verifying Checksum 2024-04-03T16:03:35.5201587Z 7066d7f0b725: Download complete 2024-04-03T16:03:35.5962335Z 8a03304f5dc7: Download complete 2024-04-03T16:03:35.6921615Z 63f01b7de7c9: Download complete 2024-04-03T16:03:37.5778795Z e96da33903b7: Verifying Checksum 2024-04-03T16:03:37.5779590Z e96da33903b7: Download complete 2024-04-03T16:03:37.6524203Z 0829c9fd377b: Download complete 2024-04-03T16:03:37.7292289Z c0d204447609: Verifying Checksum 2024-04-03T16:03:37.7293154Z c0d204447609: Download complete 2024-04-03T16:03:37.8077714Z 779e99646108: Verifying Checksum 2024-04-03T16:03:37.8078201Z 779e99646108: Download complete 2024-04-03T16:03:37.9052821Z 80f4cc22eacd: Verifying Checksum 2024-04-03T16:03:37.9053287Z 80f4cc22eacd: Download complete 2024-04-03T16:03:37.9845158Z e0aa2cd02b55: Download complete 2024-04-03T16:03:38.0613688Z f064f2e8f1a8: Download complete 2024-04-03T16:03:38.1365129Z 7874003851e3: Verifying Checksum 2024-04-03T16:03:38.1365719Z 7874003851e3: Download complete 2024-04-03T16:03:38.2205493Z 57eb0af01501: Verifying Checksum 2024-04-03T16:03:38.2206039Z 57eb0af01501: Download complete 2024-04-03T16:03:38.2983537Z 111b525fbe5a: Download complete 2024-04-03T16:03:38.4338325Z 06eb833ae33b: Verifying Checksum 2024-04-03T16:03:38.4338821Z 06eb833ae33b: Download complete 2024-04-03T16:03:38.5239361Z 589a8ceece91: Verifying Checksum 2024-04-03T16:03:38.5239872Z 589a8ceece91: Download complete 2024-04-03T16:03:38.5925222Z a4814dd3f69b: Download complete 2024-04-03T16:03:38.6741428Z 1f18671b7fca: Download complete 2024-04-03T16:03:38.7523213Z 4002c8a87e7a: Download complete 2024-04-03T16:03:38.8340666Z cc9527ae45fc: Download complete 2024-04-03T16:03:38.8471608Z 42a766fc0909: Pull complete 2024-04-03T16:03:39.0781125Z 0e2f2458798f: Pull complete 2024-04-03T16:03:39.2914812Z 52475bb51e22: Pull complete 2024-04-03T16:03:39.4381079Z b848d28edf10: Verifying Checksum 2024-04-03T16:03:39.4381571Z b848d28edf10: Download complete 2024-04-03T16:03:39.4534132Z 928ebba6ddfb: Pull complete 2024-04-03T16:03:39.5941582Z 71bfb06f0079: Pull complete 2024-04-03T16:03:39.7216275Z 56090b14b977: Pull complete 2024-04-03T16:03:55.5239598Z 22343a243e56: Pull complete 2024-04-03T16:03:55.9682065Z c29bb23a7999: Pull complete 2024-04-03T16:03:56.4202287Z 84aa8c2bf621: Pull complete 2024-04-03T16:03:56.8983436Z 869f7e29fe43: Pull complete 2024-04-03T16:04:01.6038810Z 9c7bc24d85e1: Verifying Checksum 2024-04-03T16:04:01.6039310Z 9c7bc24d85e1: Download complete 2024-04-03T16:04:06.1717323Z a35b44e43b45: Download complete 2024-04-03T16:04:52.8219240Z a35b44e43b45: Pull complete 2024-04-03T16:04:52.9166206Z 4f4fb700ef54: Pull complete 2024-04-03T16:04:53.0115114Z f7cdcec5e620: Pull complete 2024-04-03T16:04:53.1164234Z 76fd1df065af: Pull complete 2024-04-03T16:04:53.2164981Z 83315c4c7ef0: Pull complete 2024-04-03T16:04:53.3180360Z 6460360cb129: Pull complete 2024-04-03T16:04:53.3990973Z f5187f0f8da2: Pull complete 2024-04-03T16:04:53.4856406Z b1fe483228e9: Pull complete 2024-04-03T16:04:53.5846054Z f148109b25c6: Pull complete 2024-04-03T16:04:53.8041595Z 6e94329e7d9b: Pull complete 2024-04-03T16:04:53.9084355Z e418f79cac9b: Pull complete 2024-04-03T16:04:55.9743735Z 77938992fe7d: Pull complete 2024-04-03T16:04:56.2075364Z dd295d40e381: Pull complete 2024-04-03T16:04:56.3489045Z 6c196b06c1db: Pull complete 2024-04-03T16:04:56.4746155Z d2a4f8c972bb: Pull complete 2024-04-03T16:04:56.5572439Z e17fa7c51474: Pull complete 2024-04-03T16:04:56.6405575Z cff02b807961: Pull complete 2024-04-03T16:05:02.5525007Z e96da33903b7: Pull complete 2024-04-03T16:05:02.7331039Z 42bd346d5d1c: Pull complete 2024-04-03T16:05:02.9529339Z f688bbce9791: Pull complete 2024-04-03T16:05:03.1526806Z 8fbc77b9694b: Pull complete 2024-04-03T16:05:03.2977115Z e2a3bc47943f: Pull complete 2024-04-03T16:05:03.7182517Z 8237da8eed86: Pull complete 2024-04-03T16:05:03.9421308Z 71bb79f7fa9c: Pull complete 2024-04-03T16:05:04.1099248Z 9688271757d9: Pull complete 2024-04-03T16:05:04.3250794Z 8fba31fce28c: Pull complete 2024-04-03T16:05:04.5408704Z d2f8ea8e2cd5: Pull complete 2024-04-03T16:05:04.6402230Z a337c4be3a83: Pull complete 2024-04-03T16:05:04.8432876Z e2be7086a133: Pull complete 2024-04-03T16:05:04.9256544Z e8b6b86420ea: Pull complete 2024-04-03T16:05:06.0312174Z 4673681701f9: Pull complete 2024-04-03T16:05:06.1932032Z 402352c3f652: Pull complete 2024-04-03T16:05:06.3310511Z d6271b2f16b7: Pull complete 2024-04-03T16:05:06.6371667Z 0a60eb3db527: Pull complete 2024-04-03T16:05:06.8673816Z 5b9f4e043999: Pull complete 2024-04-03T16:05:07.2982935Z 3316bc233a54: Pull complete 2024-04-03T16:05:07.4980423Z ee5d0a9a6a1e: Pull complete 2024-04-03T16:05:07.7170485Z 7396a8060fce: Pull complete 2024-04-03T16:05:08.0796324Z 8f2884fd6dec: Pull complete 2024-04-03T16:05:08.1846011Z 9b6932bd4bc8: Pull complete 2024-04-03T16:05:08.3527409Z 7066d7f0b725: Pull complete 2024-04-03T16:05:08.4436626Z 8a03304f5dc7: Pull complete 2024-04-03T16:05:08.6301476Z 63f01b7de7c9: Pull complete 2024-04-03T16:05:35.8424656Z 9c7bc24d85e1: Pull complete 2024-04-03T16:05:36.2906734Z 0829c9fd377b: Pull complete 2024-04-03T16:05:36.5694196Z c0d204447609: Pull complete 2024-04-03T16:05:37.0044724Z 779e99646108: Pull complete 2024-04-03T16:05:37.4486371Z 80f4cc22eacd: Pull complete 2024-04-03T16:05:37.6514484Z e0aa2cd02b55: Pull complete 2024-04-03T16:05:38.0976867Z f064f2e8f1a8: Pull complete 2024-04-03T16:05:38.3169420Z 7874003851e3: Pull complete 2024-04-03T16:05:38.7618171Z 57eb0af01501: Pull complete 2024-04-03T16:05:38.9801119Z 111b525fbe5a: Pull complete 2024-04-03T16:05:39.3304011Z 06eb833ae33b: Pull complete 2024-04-03T16:05:39.5054510Z 589a8ceece91: Pull complete 2024-04-03T16:05:39.6960589Z a4814dd3f69b: Pull complete 2024-04-03T16:05:39.9082693Z 1f18671b7fca: Pull complete 2024-04-03T16:05:40.1256875Z 4002c8a87e7a: Pull complete 2024-04-03T16:05:40.5589394Z cc9527ae45fc: Pull complete 2024-04-03T16:05:42.1494992Z b848d28edf10: Pull complete 2024-04-03T16:05:42.2612924Z Digest: sha256:66934e8a2aa7cad8e06a1fbc5a6f0e239ee3d124c7929e4418f76d11beb80c7b 2024-04-03T16:05:42.3019337Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:05:42.3223879Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:05:42.3284750Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-04-03T16:05:42.3285549Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-04-03T16:05:42.3286252Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-04-03T16:05:42.3286928Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:05:42.3297056Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:05:42.3297532Z env: 2024-04-03T16:05:42.3297798Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:42.3298130Z ##[endgroup] 2024-04-03T16:05:45.8102645Z Defaulting to user installation because normal site-packages is not writeable 2024-04-03T16:05:46.2551081Z Collecting psutil==5.9.1 2024-04-03T16:05:46.2697768Z Downloading psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-04-03T16:05:46.3430538Z Collecting nvidia-ml-py==11.525.84 2024-04-03T16:05:46.3474760Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-04-03T16:05:46.3829896Z Installing collected packages: psutil, nvidia-ml-py 2024-04-03T16:05:46.5498201Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-04-03T16:05:46.7657582Z Prepare all required actions 2024-04-03T16:05:46.7658638Z Getting action download info 2024-04-03T16:05:46.8794523Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-04-03T16:05:47.0895094Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-04-03T16:05:47.2115606Z ##[group]Run ./.github/actions/download-build-artifacts 2024-04-03T16:05:47.2116078Z with: 2024-04-03T16:05:47.2116364Z name: linux-jammy-py3.8-gcc11 2024-04-03T16:05:47.2116751Z s3-bucket: gha-artifacts 2024-04-03T16:05:47.2117057Z env: 2024-04-03T16:05:47.2117313Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:47.2117638Z ##[endgroup] 2024-04-03T16:05:47.2261164Z ##[group]Run seemethere/download-artifact-s3@v4 2024-04-03T16:05:47.2261595Z with: 2024-04-03T16:05:47.2261873Z name: linux-jammy-py3.8-gcc11 2024-04-03T16:05:47.2262253Z s3-bucket: gha-artifacts 2024-04-03T16:05:47.2262626Z region: us-east-1 2024-04-03T16:05:47.2262893Z env: 2024-04-03T16:05:47.2263155Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:47.2263481Z ##[endgroup] 2024-04-03T16:05:47.6927286Z (node:24826) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-04-03T16:05:47.6928023Z 2024-04-03T16:05:47.6928274Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-04-03T16:05:47.6929380Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-04-03T16:05:47.6930988Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-04-03T16:05:47.7763704Z Found 1 objects with prefix pytorch/pytorch/8536950919/linux-jammy-py3.8-gcc11/ 2024-04-03T16:05:47.7764901Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-04-03T16:05:51.3873262Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-04-03T16:05:51.3879474Z Artifact download has finished successfully 2024-04-03T16:05:51.4045460Z ##[group]Run unzip -o artifacts.zip 2024-04-03T16:05:51.4045904Z unzip -o artifacts.zip 2024-04-03T16:05:51.4053353Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:05:51.4053848Z env: 2024-04-03T16:05:51.4054112Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:51.4054427Z ##[endgroup] 2024-04-03T16:05:51.4402260Z Archive: artifacts.zip 2024-04-03T16:05:51.4403299Z creating: dist/ 2024-04-03T16:05:52.3361429Z inflating: dist/torch-2.4.0a0+gitd253bd7-cp38-cp38-linux_x86_64.whl 2024-04-03T16:05:52.3362097Z creating: build/custom_test_artifacts/ 2024-04-03T16:05:52.3362681Z creating: build/custom_test_artifacts/custom-op-build/ 2024-04-03T16:05:52.3363378Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-04-03T16:05:52.3364221Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-04-03T16:05:52.3365237Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-04-03T16:05:52.3366179Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-04-03T16:05:52.3367154Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-04-03T16:05:52.3368284Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-04-03T16:05:52.3369283Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-04-03T16:05:52.3370410Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-04-03T16:05:52.3371559Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-04-03T16:05:52.3372586Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-04-03T16:05:52.3373596Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-04-03T16:05:52.3374999Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-04-03T16:05:52.3376324Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-04-03T16:05:52.3377509Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-04-03T16:05:52.3378781Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-04-03T16:05:52.3379987Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-04-03T16:05:52.3381205Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-04-03T16:05:52.3382226Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-04-03T16:05:52.3383156Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-04-03T16:05:52.3384116Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-04-03T16:05:52.3385198Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-04-03T16:05:52.3386381Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-04-03T16:05:52.3387594Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-04-03T16:05:52.3388666Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-04-03T16:05:52.3389946Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-04-03T16:05:52.3391053Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-04-03T16:05:52.3392177Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-04-03T16:05:52.3393297Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-04-03T16:05:52.3394385Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-04-03T16:05:52.3401344Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-04-03T16:05:52.3523197Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-04-03T16:05:52.3524241Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-04-03T16:05:52.3525367Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-04-03T16:05:52.3526640Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-04-03T16:05:52.3527866Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-04-03T16:05:52.3529002Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-04-03T16:05:52.3530171Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-04-03T16:05:52.3531351Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-04-03T16:05:52.3532635Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-04-03T16:05:52.3533814Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-04-03T16:05:52.3534963Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-04-03T16:05:52.3545972Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-04-03T16:05:52.3622030Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-04-03T16:05:52.3623447Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-03T16:05:52.3624567Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-04-03T16:05:52.3625581Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-04-03T16:05:52.3626525Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-04-03T16:05:52.3627445Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-04-03T16:05:52.3628339Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-04-03T16:05:52.3629133Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-04-03T16:05:52.3630170Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-04-03T16:05:52.3732098Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-04-03T16:05:52.3791642Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-04-03T16:05:52.3792405Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-04-03T16:05:52.3793090Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-04-03T16:05:52.3793922Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-04-03T16:05:52.3795197Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-04-03T16:05:52.3796149Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-04-03T16:05:52.3797082Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-04-03T16:05:52.3798084Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-04-03T16:05:52.3799077Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-04-03T16:05:52.3800207Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-04-03T16:05:52.3801461Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-04-03T16:05:52.3802488Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-04-03T16:05:52.3803505Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-04-03T16:05:52.3804685Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-04-03T16:05:52.3805881Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-04-03T16:05:52.3807053Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-04-03T16:05:52.3808223Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-04-03T16:05:52.3809412Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-04-03T16:05:52.3810635Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-04-03T16:05:52.3811634Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-04-03T16:05:52.3812555Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-04-03T16:05:52.3813516Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-04-03T16:05:52.3814612Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-04-03T16:05:52.3815840Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-04-03T16:05:52.3817013Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-04-03T16:05:52.3818265Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-04-03T16:05:52.3819504Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-04-03T16:05:52.3820734Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-04-03T16:05:52.3821884Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-04-03T16:05:52.3823021Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-04-03T16:05:52.3824154Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-04-03T16:05:52.3830812Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-04-03T16:05:52.3890127Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-04-03T16:05:52.3891355Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-03T16:05:52.3892471Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-04-03T16:05:52.3893452Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-04-03T16:05:52.3894376Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-04-03T16:05:52.3895289Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-04-03T16:05:52.3896153Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-04-03T16:05:52.3896937Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-04-03T16:05:52.3897743Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-04-03T16:05:52.3943021Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-04-03T16:05:52.3943782Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-04-03T16:05:52.3944558Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-04-03T16:05:52.3945458Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-04-03T16:05:52.3946735Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-04-03T16:05:52.3947751Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-04-03T16:05:52.3948784Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-04-03T16:05:52.3950053Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-04-03T16:05:52.3951109Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-04-03T16:05:52.3952321Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-04-03T16:05:52.3953538Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-04-03T16:05:52.3954639Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-04-03T16:05:52.3955708Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-04-03T16:05:52.3956973Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-04-03T16:05:52.3958239Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-04-03T16:05:52.3959471Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-04-03T16:05:52.3960825Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-04-03T16:05:52.3962259Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-04-03T16:05:52.3963626Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-04-03T16:05:52.3964695Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-04-03T16:05:52.3965712Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-04-03T16:05:52.3966756Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-04-03T16:05:52.3967936Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-04-03T16:05:52.3969234Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-04-03T16:05:52.3970502Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-04-03T16:05:52.3971702Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-04-03T16:05:52.3972929Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-04-03T16:05:52.3974171Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-04-03T16:05:52.3975406Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-04-03T16:05:52.3976638Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-04-03T16:05:52.3977838Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-04-03T16:05:52.3979145Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-04-03T16:05:52.4083306Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-04-03T16:05:52.4084527Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-04-03T16:05:52.4085758Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-04-03T16:05:52.4087131Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-04-03T16:05:52.4088516Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-04-03T16:05:52.4089806Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-04-03T16:05:52.4091083Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-04-03T16:05:52.4092378Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-04-03T16:05:52.4093682Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-04-03T16:05:52.4094964Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-04-03T16:05:52.4096240Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-04-03T16:05:52.4106135Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-04-03T16:05:52.4156754Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-04-03T16:05:52.4158114Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-03T16:05:52.4159294Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-04-03T16:05:52.4160574Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-04-03T16:05:52.4161679Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-04-03T16:05:52.4162648Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-04-03T16:05:52.4163594Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-04-03T16:05:52.4164445Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-04-03T16:05:52.4165308Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-04-03T16:05:52.4260518Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-04-03T16:05:52.4301688Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-04-03T16:05:52.4302337Z creating: build/lib/ 2024-04-03T16:05:52.4302672Z inflating: build/lib/libclog.a 2024-04-03T16:05:52.4311842Z inflating: build/lib/libpthreadpool.a 2024-04-03T16:05:52.4318838Z inflating: build/lib/libittnotify.a 2024-04-03T16:05:52.4383129Z inflating: build/lib/libgtest.a 2024-04-03T16:05:52.4390879Z inflating: build/lib/libcpuinfo.a 2024-04-03T16:05:52.4398124Z inflating: build/lib/libcpuinfo_internals.a 2024-04-03T16:05:52.4398981Z inflating: build/lib/libfoxi_loader.a 2024-04-03T16:05:52.4419354Z inflating: build/lib/libfmt.a 2024-04-03T16:05:52.4500297Z inflating: build/lib/libprotobuf-lite.a 2024-04-03T16:05:52.4501085Z inflating: build/lib/libtorch_global_deps.so 2024-04-03T16:05:52.4572197Z inflating: build/lib/libbenchmark.a 2024-04-03T16:05:52.4598798Z inflating: build/lib/libtensorpipe_uv.a 2024-04-03T16:05:52.4600883Z inflating: build/lib/libnnpack_reference_layers.a 2024-04-03T16:05:52.4710826Z inflating: build/lib/libgloo.a 2024-04-03T16:05:52.4711654Z inflating: build/lib/libgtest_main.a 2024-04-03T16:05:52.4728553Z inflating: build/lib/libgmock.a 2024-04-03T16:05:52.4729119Z inflating: build/lib/libbenchmark_main.a 2024-04-03T16:05:52.4790996Z inflating: build/lib/libasmjit.a 2024-04-03T16:05:52.4803818Z inflating: build/lib/libqnnpack.a 2024-04-03T16:05:52.4822246Z inflating: build/lib/libpytorch_qnnpack.a 2024-04-03T16:05:52.5098965Z inflating: build/lib/libkineto.a 2024-04-03T16:05:52.5529939Z inflating: build/lib/libprotobuf.a 2024-04-03T16:05:52.5530791Z inflating: build/lib/libgmock_main.a 2024-04-03T16:05:52.6541924Z inflating: build/lib/libfbgemm.a 2024-04-03T16:05:52.7075107Z inflating: build/lib/libtensorpipe.a 2024-04-03T16:05:52.7163363Z inflating: build/lib/libc10.so 2024-04-03T16:05:52.7180717Z inflating: build/lib/libnnpack.a 2024-04-03T16:05:52.7664241Z inflating: build/lib/libprotoc.a 2024-04-03T16:05:52.7842297Z inflating: build/lib/libXNNPACK.a 2024-04-03T16:05:53.6562994Z inflating: build/lib/libdnnl.a 2024-04-03T16:05:53.6598679Z inflating: build/lib/libcaffe2_protos.a 2024-04-03T16:05:53.6637060Z inflating: build/lib/libonnx_proto.a 2024-04-03T16:05:53.7289415Z inflating: build/lib/libonnx.a 2024-04-03T16:05:55.9223394Z inflating: build/lib/libtorch_cpu.so 2024-04-03T16:05:55.9224309Z inflating: build/lib/libtorch.so 2024-04-03T16:05:55.9227591Z inflating: build/lib/libshm.so 2024-04-03T16:05:55.9246741Z inflating: build/lib/libjitbackend_test.so 2024-04-03T16:05:55.9271088Z inflating: build/lib/libbackend_with_compiler.so 2024-04-03T16:05:55.9275346Z inflating: build/lib/libunbox_lib.a 2024-04-03T16:05:55.9331231Z inflating: build/lib/libtorchbind_test.so 2024-04-03T16:05:56.1076779Z inflating: build/lib/libtorch_python.so 2024-04-03T16:05:56.1109827Z inflating: build/lib/libnnapi_backend.so 2024-04-03T16:05:56.1110278Z creating: build/bin/ 2024-04-03T16:05:56.1110623Z creating: build/bin/CMakeFiles/ 2024-04-03T16:05:56.1111407Z inflating: build/bin/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-03T16:05:56.1112078Z extracting: build/bin/CMakeFiles/progress.marks 2024-04-03T16:05:56.1179162Z inflating: build/bin/pow_test 2024-04-03T16:05:56.1179655Z inflating: build/bin/Makefile 2024-04-03T16:05:56.1282065Z inflating: build/bin/kernel_lambda_test 2024-04-03T16:05:56.1282564Z inflating: build/bin/cmake_install.cmake 2024-04-03T16:05:56.1336126Z inflating: build/bin/stride_properties_test 2024-04-03T16:05:56.1336626Z inflating: build/bin/CTestTestfile.cmake 2024-04-03T16:05:56.1387972Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-04-03T16:05:56.1438271Z inflating: build/bin/c10_StreamGuard_test 2024-04-03T16:05:56.1489884Z inflating: build/bin/c10_Device_test 2024-04-03T16:05:56.1542760Z inflating: build/bin/c10_DeviceGuard_test 2024-04-03T16:05:56.1595141Z inflating: build/bin/c10_SymInt_test 2024-04-03T16:05:56.1645681Z inflating: build/bin/c10_ConstexprCrc_test 2024-04-03T16:05:56.1701312Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-04-03T16:05:56.1758459Z inflating: build/bin/c10_SizesAndStrides_test 2024-04-03T16:05:56.1818938Z inflating: build/bin/c10_DispatchKeySet_test 2024-04-03T16:05:56.1873958Z inflating: build/bin/c10_Scalar_test 2024-04-03T16:05:56.1928459Z inflating: build/bin/c10_Bitset_test 2024-04-03T16:05:56.1985239Z inflating: build/bin/c10_InlineStreamGuard_test 2024-04-03T16:05:56.2036402Z inflating: build/bin/c10_DeadlockDetection_test 2024-04-03T16:05:56.2108106Z inflating: build/bin/c10_cow_test 2024-04-03T16:05:56.2160104Z inflating: build/bin/c10_Half_test 2024-04-03T16:05:56.2217095Z inflating: build/bin/c10_LeftRight_test 2024-04-03T16:05:56.2268415Z inflating: build/bin/c10_Synchronized_test 2024-04-03T16:05:56.2321565Z inflating: build/bin/c10_TypeIndex_test 2024-04-03T16:05:56.2378300Z inflating: build/bin/c10_Metaprogramming_test 2024-04-03T16:05:56.2434999Z inflating: build/bin/c10_ThreadLocal_test 2024-04-03T16:05:56.2485300Z inflating: build/bin/c10_TypeTraits_test 2024-04-03T16:05:56.2537669Z inflating: build/bin/c10_TypeList_test 2024-04-03T16:05:56.2590707Z inflating: build/bin/c10_accumulate_test 2024-04-03T16:05:56.2642277Z inflating: build/bin/c10_flags_test 2024-04-03T16:05:56.2694428Z inflating: build/bin/c10_irange_test 2024-04-03T16:05:56.2748161Z inflating: build/bin/c10_exception_test 2024-04-03T16:05:56.2799798Z inflating: build/bin/c10_generic_math_test 2024-04-03T16:05:56.2857745Z inflating: build/bin/c10_complex_math_test 2024-04-03T16:05:56.2914064Z inflating: build/bin/c10_bfloat16_test 2024-04-03T16:05:56.2970867Z inflating: build/bin/c10_logging_test 2024-04-03T16:05:56.3022646Z inflating: build/bin/c10_bit_cast_test 2024-04-03T16:05:56.3079199Z inflating: build/bin/c10_complex_test 2024-04-03T16:05:56.3133928Z inflating: build/bin/c10_registry_test 2024-04-03T16:05:56.3197486Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-04-03T16:05:56.3273104Z inflating: build/bin/c10_optional_test 2024-04-03T16:05:56.3325003Z inflating: build/bin/c10_tempfile_test 2024-04-03T16:05:56.3378246Z inflating: build/bin/c10_string_util_test 2024-04-03T16:05:56.3426429Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-04-03T16:05:56.3479202Z inflating: build/bin/c10_ssize_test 2024-04-03T16:05:56.3629835Z inflating: build/bin/c10_small_vector_test 2024-04-03T16:05:56.3791718Z inflating: build/bin/c10_intrusive_ptr_test 2024-04-03T16:05:56.3851692Z inflating: build/bin/c10_string_view_test 2024-04-03T16:05:56.3909414Z inflating: build/bin/c10_typeid_test 2024-04-03T16:05:56.4236479Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-04-03T16:05:56.4594636Z inflating: build/bin/vec_test_all_types_AVX2 2024-04-03T16:05:56.4939652Z inflating: build/bin/vec_test_all_types_AVX512 2024-04-03T16:05:56.5390979Z inflating: build/bin/protoc-3.13.0.0 2024-04-03T16:05:56.5842109Z inflating: build/bin/protoc 2024-04-03T16:05:56.5896260Z inflating: build/bin/Dimname_test 2024-04-03T16:05:56.5950776Z inflating: build/bin/HashStoreTest 2024-04-03T16:05:56.6008223Z inflating: build/bin/TCPStoreTest 2024-04-03T16:05:56.6062697Z inflating: build/bin/FileStoreTest 2024-04-03T16:05:56.6114897Z inflating: build/bin/operator_name_test 2024-04-03T16:05:56.6117464Z inflating: build/bin/example_allreduce 2024-04-03T16:05:56.6173768Z inflating: build/bin/type_ptr_test 2024-04-03T16:05:56.6241103Z inflating: build/bin/ProcessGroupGlooTest 2024-04-03T16:05:56.6300533Z inflating: build/bin/atest 2024-04-03T16:05:56.6375021Z inflating: build/bin/Dict_test 2024-04-03T16:05:56.6441279Z inflating: build/bin/MaybeOwned_test 2024-04-03T16:05:56.6501418Z inflating: build/bin/apply_utils_test 2024-04-03T16:05:56.6557603Z inflating: build/bin/broadcast_test 2024-04-03T16:05:56.6615729Z inflating: build/bin/NamedTensor_test 2024-04-03T16:05:56.6680288Z inflating: build/bin/basic 2024-04-03T16:05:56.6732563Z inflating: build/bin/cpu_allocator_test 2024-04-03T16:05:56.6783856Z inflating: build/bin/dispatch_key_set_test 2024-04-03T16:05:56.6837866Z inflating: build/bin/cpu_profiling_allocator_test 2024-04-03T16:05:56.6896454Z inflating: build/bin/static_runtime_bench 2024-04-03T16:05:56.6947442Z inflating: build/bin/lazy_tensor_test 2024-04-03T16:05:56.7007063Z inflating: build/bin/cpu_generator_test 2024-04-03T16:05:56.7058582Z inflating: build/bin/dlconvertor_test 2024-04-03T16:05:56.7115368Z inflating: build/bin/half_test 2024-04-03T16:05:56.7170205Z inflating: build/bin/memory_format_test 2024-04-03T16:05:56.7275964Z inflating: build/bin/List_test 2024-04-03T16:05:56.7335024Z inflating: build/bin/extension_backend_test 2024-04-03T16:05:56.7389454Z inflating: build/bin/math_kernel_test 2024-04-03T16:05:56.7443951Z inflating: build/bin/memory_overlapping_test 2024-04-03T16:05:56.7496542Z inflating: build/bin/StorageUtils_test 2024-04-03T16:05:56.7617784Z inflating: build/bin/kernel_function_legacy_test 2024-04-03T16:05:56.7676121Z inflating: build/bin/scalar_tensor_test 2024-04-03T16:05:56.7728824Z inflating: build/bin/reportMemoryUsage_test 2024-04-03T16:05:56.7782220Z inflating: build/bin/packedtensoraccessor_test 2024-04-03T16:05:56.7834622Z inflating: build/bin/operators_test 2024-04-03T16:05:56.7891929Z inflating: build/bin/native_test 2024-04-03T16:05:56.7950808Z inflating: build/bin/quantized_test 2024-04-03T16:05:56.7953251Z inflating: build/bin/thread_init_test 2024-04-03T16:05:56.8049723Z inflating: build/bin/kernel_function_test 2024-04-03T16:05:56.8108918Z inflating: build/bin/scalar_test 2024-04-03T16:05:56.8359443Z inflating: build/bin/static_runtime_test 2024-04-03T16:05:56.8410781Z inflating: build/bin/reduce_ops_test 2024-04-03T16:05:56.8537588Z inflating: build/bin/kernel_lambda_legacy_test 2024-04-03T16:05:56.8538766Z inflating: build/bin/verify_api_visibility 2024-04-03T16:05:56.8592546Z inflating: build/bin/undefined_tensor_test 2024-04-03T16:05:56.8642790Z inflating: build/bin/op_allowlist_test 2024-04-03T16:05:56.8738525Z inflating: build/bin/ivalue_test 2024-04-03T16:05:56.8793029Z inflating: build/bin/mobile_memory_cleanup 2024-04-03T16:05:56.8845660Z inflating: build/bin/CppSignature_test 2024-04-03T16:05:56.8907163Z inflating: build/bin/kernel_stackbased_test 2024-04-03T16:05:56.8959892Z inflating: build/bin/weakref_test 2024-04-03T16:05:56.9025125Z inflating: build/bin/inline_container_test 2024-04-03T16:05:56.9077839Z inflating: build/bin/wrapdim_test 2024-04-03T16:05:56.9138483Z inflating: build/bin/IListRef_test 2024-04-03T16:05:56.9209450Z inflating: build/bin/legacy_vmap_test 2024-04-03T16:05:56.9261999Z inflating: build/bin/xla_tensor_test 2024-04-03T16:05:56.9341182Z inflating: build/bin/tensor_iterator_test 2024-04-03T16:05:56.9401994Z inflating: build/bin/type_test 2024-04-03T16:05:56.9495515Z inflating: build/bin/cpu_rng_test 2024-04-03T16:05:56.9562706Z inflating: build/bin/KernelFunction_test 2024-04-03T16:05:56.9617947Z inflating: build/bin/test_parallel 2024-04-03T16:05:56.9714069Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-04-03T16:05:57.0021835Z inflating: build/bin/op_registration_test 2024-04-03T16:05:57.0025739Z inflating: build/bin/torch_shm_manager 2024-04-03T16:05:57.0082238Z inflating: build/bin/backend_fallback_test 2024-04-03T16:05:57.0095104Z inflating: build/bin/tutorial_tensorexpr 2024-04-03T16:05:57.0150988Z inflating: build/bin/test_dist_autograd 2024-04-03T16:05:57.0153252Z inflating: build/bin/parallel_benchmark 2024-04-03T16:05:57.0223072Z inflating: build/bin/test_cpp_rpc 2024-04-03T16:05:57.0231803Z inflating: build/bin/aot_model_compiler_test 2024-04-03T16:05:57.0299247Z inflating: build/bin/test_mobile_nnc 2024-04-03T16:05:57.0353966Z inflating: build/bin/test_edge_op_registration 2024-04-03T16:05:57.1139734Z inflating: build/bin/test_tensorexpr 2024-04-03T16:05:57.1477353Z inflating: build/bin/test_lazy 2024-04-03T16:05:57.2596864Z inflating: build/bin/test_api 2024-04-03T16:05:57.3157001Z inflating: build/bin/test_jit 2024-04-03T16:05:57.3157510Z creating: .additional_ci_files/ 2024-04-03T16:05:57.3192216Z inflating: .additional_ci_files/test-times.json 2024-04-03T16:05:57.3335693Z inflating: .additional_ci_files/test-class-times.json 2024-04-03T16:05:57.3364836Z ##[group]Run rm artifacts.zip 2024-04-03T16:05:57.3365220Z rm artifacts.zip 2024-04-03T16:05:57.3373059Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:05:57.3373551Z env: 2024-04-03T16:05:57.3373797Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:57.3374122Z ##[endgroup] 2024-04-03T16:05:57.3817011Z ##[group]Run df -H 2024-04-03T16:05:57.3817322Z df -H 2024-04-03T16:05:57.3824694Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:05:57.3825193Z env: 2024-04-03T16:05:57.3825460Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:57.3825796Z ##[endgroup] 2024-04-03T16:05:57.4080898Z Filesystem Size Used Avail Use% Mounted on 2024-04-03T16:05:57.4081536Z devtmpfs 8.2G 0 8.2G 0% /dev 2024-04-03T16:05:57.4082306Z tmpfs 8.2G 0 8.2G 0% /dev/shm 2024-04-03T16:05:57.4083096Z tmpfs 8.2G 467k 8.2G 1% /run 2024-04-03T16:05:57.4083729Z tmpfs 8.2G 0 8.2G 0% /sys/fs/cgroup 2024-04-03T16:05:57.4084190Z /dev/nvme0n1p1 162G 23G 139G 15% / 2024-04-03T16:05:57.4084622Z tmpfs 1.7G 0 1.7G 0% /run/user/0 2024-04-03T16:05:57.4189305Z Prepare all required actions 2024-04-03T16:05:57.4190021Z Getting action download info 2024-04-03T16:05:57.5386655Z ##[group]Run ./.github/actions/download-td-artifacts 2024-04-03T16:05:57.5387107Z with: 2024-04-03T16:05:57.5387347Z env: 2024-04-03T16:05:57.5387605Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:57.5387920Z ##[endgroup] 2024-04-03T16:05:57.5422060Z ##[group]Run seemethere/download-artifact-s3@v4 2024-04-03T16:05:57.5422520Z with: 2024-04-03T16:05:57.5422770Z name: td_results 2024-04-03T16:05:57.5423060Z s3-bucket: gha-artifacts 2024-04-03T16:05:57.5423401Z region: us-east-1 2024-04-03T16:05:57.5423678Z env: 2024-04-03T16:05:57.5423917Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:57.5424243Z ##[endgroup] 2024-04-03T16:05:58.0119674Z (node:24866) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-04-03T16:05:58.0120390Z 2024-04-03T16:05:58.0120802Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-04-03T16:05:58.0121524Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-04-03T16:05:58.0122362Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-04-03T16:05:58.1140794Z Found 1 objects with prefix pytorch/pytorch/8536950919/td_results/ 2024-04-03T16:05:58.1141844Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-04-03T16:05:58.1707870Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-04-03T16:05:58.1713612Z Artifact download has finished successfully 2024-04-03T16:05:58.1863389Z ##[group]Run mkdir -p .additional_ci_files 2024-04-03T16:05:58.1863839Z mkdir -p .additional_ci_files 2024-04-03T16:05:58.1864388Z mv td_results.json .additional_ci_files/td_results.json 2024-04-03T16:05:58.1874801Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:05:58.1875629Z env: 2024-04-03T16:05:58.1876063Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:58.1876603Z ##[endgroup] 2024-04-03T16:05:58.2054322Z ##[group]Run .github/scripts/parse_ref.py 2024-04-03T16:05:58.2055144Z .github/scripts/parse_ref.py 2024-04-03T16:05:58.2062525Z shell: /usr/bin/bash -e {0} 2024-04-03T16:05:58.2062859Z env: 2024-04-03T16:05:58.2063114Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:58.2063425Z ##[endgroup] 2024-04-03T16:05:58.2316016Z Prepare all required actions 2024-04-03T16:05:58.2357022Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-04-03T16:05:58.2357464Z with: 2024-04-03T16:05:58.2358243Z github-token: *** 2024-04-03T16:05:58.2358517Z env: 2024-04-03T16:05:58.2358774Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:58.2359106Z ##[endgroup] 2024-04-03T16:05:58.2376191Z ##[group]Run set -eux 2024-04-03T16:05:58.2376511Z set -eux 2024-04-03T16:05:58.2377091Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-04-03T16:05:58.2384780Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:05:58.2385256Z env: 2024-04-03T16:05:58.2385511Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:58.2386015Z GITHUB_TOKEN: *** 2024-04-03T16:05:58.2386305Z ##[endgroup] 2024-04-03T16:05:58.2406669Z + python3 .github/scripts/get_workflow_job_id.py 8536950919 i-0ae24b2353acec2d3 2024-04-03T16:06:40.8604871Z RuntimeError(('Is github alright?', "Recieved status code '502' when attempting to retrieve https://api.github.com/repos/pytorch/pytorch/actions/runs/8536950919/jobs?per_page=100:\n", 'Bad Gateway\n\nheaders=Server: GitHub.com\nDate: Wed, 03 Apr 2024 16:06:40 GMT\nContent-Type: application/json\nContent-Length: 32\nETag: "660d79a2-20"\nVary: Accept-Encoding, Accept, X-Requested-With\nX-GitHub-Request-Id: E2AA:5AAC:D915ED:16BF58C:660D7E86\nconnection: close\n\n')) 2024-04-03T16:06:40.8607745Z workflow-8536950919 2024-04-03T16:06:40.8851150Z Prepare all required actions 2024-04-03T16:06:40.8851578Z Getting action download info 2024-04-03T16:06:41.0067568Z ##[group]Run ./.github/actions/filter-test-configs 2024-04-03T16:06:41.0068026Z with: 2024-04-03T16:06:41.0068481Z github-token: *** 2024-04-03T16:06:41.0071770Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "default", "shard": 2, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "default", "shard": 3, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "docs_test", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge", "unstable": "unstable"}, {"config": "jit_legacy", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "backwards_compat", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge"}]} 2024-04-03T16:06:41.0074834Z env: 2024-04-03T16:06:41.0075080Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:06:41.0075411Z ##[endgroup] 2024-04-03T16:06:41.0117947Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-04-03T16:06:41.0118503Z with: 2024-04-03T16:06:41.0118750Z shell: bash 2024-04-03T16:06:41.0119011Z timeout_minutes: 10 2024-04-03T16:06:41.0119315Z max_attempts: 5 2024-04-03T16:06:41.0119605Z retry_wait_seconds: 30 2024-04-03T16:06:41.0120468Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-04-03T16:06:41.0121457Z polling_interval_seconds: 1 2024-04-03T16:06:41.0121813Z warning_on_retry: true 2024-04-03T16:06:41.0122140Z continue_on_error: false 2024-04-03T16:06:41.0122441Z env: 2024-04-03T16:06:41.0122689Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:06:41.0123190Z GITHUB_TOKEN: *** 2024-04-03T16:06:41.0123464Z ##[endgroup] 2024-04-03T16:06:41.0659567Z + python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-04-03T16:06:41.2736681Z Defaulting to user installation because normal site-packages is not writeable 2024-04-03T16:06:41.3877987Z Collecting requests==2.26.0 2024-04-03T16:06:41.4037527Z Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB) 2024-04-03T16:06:41.5292662Z Collecting pyyaml==6.0.1 2024-04-03T16:06:41.5357348Z Downloading PyYAML-6.0.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (670 kB) 2024-04-03T16:06:41.6085679Z Collecting certifi>=2017.4.17 2024-04-03T16:06:41.6122858Z Downloading certifi-2024.2.2-py3-none-any.whl (163 kB) 2024-04-03T16:06:41.8642971Z Collecting charset-normalizer~=2.0.0; python_version >= "3" 2024-04-03T16:06:41.8679687Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-04-03T16:06:41.9038094Z Collecting idna<4,>=2.5; python_version >= "3" 2024-04-03T16:06:41.9080907Z Downloading idna-3.6-py3-none-any.whl (61 kB) 2024-04-03T16:06:42.0011967Z Collecting urllib3<1.27,>=1.21.1 2024-04-03T16:06:42.0048815Z Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 2024-04-03T16:06:42.0854961Z Installing collected packages: certifi, charset-normalizer, idna, urllib3, requests, pyyaml 2024-04-03T16:06:42.1166871Z WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2024-04-03T16:06:42.1168096Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-04-03T16:06:42.3322578Z Successfully installed certifi-2024.2.2 charset-normalizer-2.0.12 idna-3.6 pyyaml-6.0.1 requests-2.26.0 urllib3-1.26.18 2024-04-03T16:06:43.0667668Z Command completed after 1 attempt(s). 2024-04-03T16:06:43.0709265Z ##[group]Run set -x 2024-04-03T16:06:43.0709903Z set -x 2024-04-03T16:06:43.0710192Z  2024-04-03T16:06:43.0710728Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-04-03T16:06:43.0711390Z # in runner workspace 2024-04-03T16:06:43.0712030Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-04-03T16:06:43.0719908Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:06:43.0720483Z env: 2024-04-03T16:06:43.0720733Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:06:43.0721058Z ##[endgroup] 2024-04-03T16:06:43.0742589Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-04-03T16:06:43.0960785Z ##[group]Run set -x 2024-04-03T16:06:43.0961131Z set -x 2024-04-03T16:06:43.0961393Z  2024-04-03T16:06:43.0961935Z # TODO: This is a very hacky way to get the job name. GitHub runner has the info 2024-04-03T16:06:43.0962848Z # but doesn't expose it in anyway. The job name is part of the job message the 2024-04-03T16:06:43.0963749Z # runner receives, so it's there and printed out to the diag log. Below is the 2024-04-03T16:06:43.0964656Z # code responsible for printing it. Need to check with GitHub to see if they can 2024-04-03T16:06:43.0965428Z # expose this variable as part of GitHub context. 2024-04-03T16:06:43.0966271Z # https://github.com/actions/runner/blob/main/src/Runner.Worker/JobExtension.cs#L345 2024-04-03T16:06:43.0967155Z pushd "/home/ec2-user/actions-runner/_work/pytorch/../../_diag" 2024-04-03T16:06:43.0967704Z pwd 2024-04-03T16:06:43.0967963Z  2024-04-03T16:06:43.0968490Z LOG_FILE=$(grep -l -r "4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0" *.log | tail -n 1) 2024-04-03T16:06:43.0969318Z if [ -n "${LOG_FILE}" ]; then 2024-04-03T16:06:43.0970000Z  # For some reasons, awk {print $2} on Linux and Windows (bash) work correctly while it 2024-04-03T16:06:43.0970739Z  # needs to be awk {print $3} on MacOS 2024-04-03T16:06:43.0971180Z  case ${RUNNER_OS} in 2024-04-03T16:06:43.0971537Z  macOS) 2024-04-03T16:06:43.0972186Z  JOB_NAME=$(grep -r "\"jobDisplayName\"" "${LOG_FILE}" | awk -F '[:]' '{print $3}' | sed 's/"//g' | xargs) 2024-04-03T16:06:43.0972897Z  ;; 2024-04-03T16:06:43.0973174Z  *) 2024-04-03T16:06:43.0973795Z  JOB_NAME=$(grep -r "\"jobDisplayName\"" "${LOG_FILE}" | awk -F '[:]' '{print $2}' | sed 's/"//g' | xargs) 2024-04-03T16:06:43.0974498Z  ;; 2024-04-03T16:06:43.0974773Z  esac 2024-04-03T16:06:43.0975167Z  echo "job-name=${JOB_NAME}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:06:43.0975633Z fi 2024-04-03T16:06:43.0975888Z  2024-04-03T16:06:43.0976137Z popd 2024-04-03T16:06:43.0983156Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:06:43.0983628Z env: 2024-04-03T16:06:43.0983883Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:06:43.0984204Z ##[endgroup] 2024-04-03T16:06:43.1004681Z + pushd /home/ec2-user/actions-runner/_work/pytorch/../../_diag 2024-04-03T16:06:43.1005313Z + pwd 2024-04-03T16:06:43.1006177Z ~/actions-runner/_diag ~/actions-runner/_work/pytorch/pytorch 2024-04-03T16:06:43.1006950Z /home/ec2-user/actions-runner/_diag 2024-04-03T16:06:43.1011356Z ++ grep -l -r 4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 Runner_20240403-154017-utc.log Runner_20240403-154020-utc.log Worker_20240403-155523-utc.log Worker_20240403-160020-utc.log 2024-04-03T16:06:43.1013409Z ++ tail -n 1 2024-04-03T16:06:43.1078738Z + LOG_FILE=Worker_20240403-160020-utc.log 2024-04-03T16:06:43.1079289Z + '[' -n Worker_20240403-160020-utc.log ']' 2024-04-03T16:06:43.1079898Z + case ${RUNNER_OS} in 2024-04-03T16:06:43.1084060Z ++ grep -r '"jobDisplayName"' Worker_20240403-160020-utc.log 2024-04-03T16:06:43.1084989Z ++ awk -F '[:]' '{print $2}' 2024-04-03T16:06:43.1085577Z ++ sed 's/"//g' 2024-04-03T16:06:43.1086360Z ++ xargs 2024-04-03T16:06:43.1145730Z + JOB_NAME='linux-jammy-py3.8-gcc11 / test (distributed, 2, 2, linux.2xlarge),' 2024-04-03T16:06:43.1146698Z + echo 'job-name=linux-jammy-py3.8-gcc11 / test (distributed, 2, 2, linux.2xlarge),' 2024-04-03T16:06:43.1147626Z + popd 2024-04-03T16:06:43.1147965Z ~/actions-runner/_work/pytorch/pytorch 2024-04-03T16:06:43.1173616Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-04-03T16:06:43.1174353Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-04-03T16:06:43.1174805Z echo "Job name: ${JOB_NAME}" 2024-04-03T16:06:43.1175172Z  2024-04-03T16:06:43.1175706Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-04-03T16:06:43.1176367Z # in runner workspace 2024-04-03T16:06:43.1176931Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-04-03T16:06:43.1177556Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-04-03T16:06:43.1177995Z  --job-name "${JOB_NAME}" \ 2024-04-03T16:06:43.1181104Z  --test-matrix "{"include": [{"config": "default", "shard": 1, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "default", "shard": 2, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "default", "shard": 3, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "docs_test", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge", "unstable": "unstable"}, {"config": "jit_legacy", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "backwards_compat", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge"}]}" \ 2024-04-03T16:06:43.1184218Z  --pr-number "${PR_NUMBER}" \ 2024-04-03T16:06:43.1184742Z  --tag "${TAG}" \ 2024-04-03T16:06:43.1185104Z  --event-name "${EVENT_NAME}" \ 2024-04-03T16:06:43.1185538Z  --schedule "${SCHEDULE}" \ 2024-04-03T16:06:43.1185951Z  --branch "${HEAD_BRANCH}" 2024-04-03T16:06:43.1193294Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:06:43.1193787Z env: 2024-04-03T16:06:43.1194049Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:06:43.1194576Z GITHUB_TOKEN: *** 2024-04-03T16:06:43.1195095Z JOB_NAME: linux-jammy-py3.8-gcc11 / test (distributed, 2, 2, linux.2xlarge), 2024-04-03T16:06:43.1195702Z PR_NUMBER: 123154 2024-04-03T16:06:43.1195992Z TAG: 2024-04-03T16:06:43.1196239Z EVENT_NAME: pull_request 2024-04-03T16:06:43.1196566Z SCHEDULE: 2024-04-03T16:06:43.1196819Z HEAD_BRANCH: 2024-04-03T16:06:43.1197091Z ##[endgroup] 2024-04-03T16:06:43.1216181Z Workflow: pull 2024-04-03T16:06:43.1216792Z Job name: linux-jammy-py3.8-gcc11 / test (distributed, 2, 2, linux.2xlarge), 2024-04-03T16:06:43.3419383Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-04-03T16:06:43.4780893Z fatal: unknown commit main 2024-04-03T16:06:43.4802057Z /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/filter_test_configs.py:458: UserWarning: failed to get commit messages: Command '['git', 'cherry', '-v', 'main']' returned non-zero exit status 128. 2024-04-03T16:06:43.4803688Z warnings.warn(f"failed to get commit messages: {e}") 2024-04-03T16:06:43.4922019Z ##[group]Run echo "Filtered matrix:" 2024-04-03T16:06:43.4922458Z echo "Filtered matrix:" 2024-04-03T16:06:43.4925497Z echo "{"include": [{"config": "default", "shard": 1, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "default", "shard": 2, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "default", "shard": 3, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "docs_test", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge", "unstable": "unstable"}, {"config": "jit_legacy", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "backwards_compat", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge"}]}" 2024-04-03T16:06:43.4928528Z  2024-04-03T16:06:43.4928897Z echo 2024-04-03T16:06:43.4929253Z echo "Is the current job unstable? False" 2024-04-03T16:06:43.4929697Z  2024-04-03T16:06:43.4929931Z echo 2024-04-03T16:06:43.4930265Z echo "Is keep-going label set? False" 2024-04-03T16:06:43.4930689Z  2024-04-03T16:06:43.4930923Z echo 2024-04-03T16:06:43.4931217Z echo "Renabled issues? " 2024-04-03T16:06:43.4938540Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:06:43.4939024Z env: 2024-04-03T16:06:43.4939286Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:06:43.4939599Z ##[endgroup] 2024-04-03T16:06:43.4959356Z Filtered matrix: 2024-04-03T16:06:43.4964338Z {include: [{config: default, shard: 1, num_shards: 3, runner: linux.2xlarge}, {config: default, shard: 2, num_shards: 3, runner: linux.2xlarge}, {config: default, shard: 3, num_shards: 3, runner: linux.2xlarge}, {config: docs_test, shard: 1, num_shards: 1, runner: linux.2xlarge, unstable: unstable}, {config: jit_legacy, shard: 1, num_shards: 1, runner: linux.2xlarge}, {config: backwards_compat, shard: 1, num_shards: 1, runner: linux.2xlarge}, {config: distributed, shard: 1, num_shards: 2, runner: linux.2xlarge}, {config: distributed, shard: 2, num_shards: 2, runner: linux.2xlarge}]} 2024-04-03T16:06:43.4967157Z 2024-04-03T16:06:43.4967299Z Is the current job unstable? False 2024-04-03T16:06:43.4967581Z 2024-04-03T16:06:43.4967805Z Is keep-going label set? False 2024-04-03T16:06:43.4968048Z 2024-04-03T16:06:43.4968166Z Renabled issues? 2024-04-03T16:06:43.5020513Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-04-03T16:06:43.5021348Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-04-03T16:06:43.5028597Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:06:43.5029079Z env: 2024-04-03T16:06:43.5029321Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:06:43.5029854Z JOB_TIMEOUT: 240 2024-04-03T16:06:43.5030145Z ##[endgroup] 2024-04-03T16:06:43.5103308Z ##[group]Run set -x 2024-04-03T16:06:43.5103690Z set -x 2024-04-03T16:06:43.5103971Z  2024-04-03T16:06:43.5104308Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-04-03T16:06:43.5104839Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-04-03T16:06:43.5105390Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-04-03T16:06:43.5105884Z  TEST_COMMAND=.ci/onnx/test.sh 2024-04-03T16:06:43.5106267Z else 2024-04-03T16:06:43.5106587Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-04-03T16:06:43.5107002Z fi 2024-04-03T16:06:43.5107242Z  2024-04-03T16:06:43.5107693Z # detached container should get cleaned up by teardown_ec2_linux 2024-04-03T16:06:43.5108448Z # TODO: Stop building test binaries as part of the build phase 2024-04-03T16:06:43.5109103Z # Used for GPU_FLAG since that doesn't play nice 2024-04-03T16:06:43.5109872Z # shellcheck disable=SC2086,SC2090 2024-04-03T16:06:43.5110313Z container_name=$(docker run \ 2024-04-03T16:06:43.5110718Z  ${GPU_FLAG:-} \ 2024-04-03T16:06:43.5111075Z  -e BUILD_ENVIRONMENT \ 2024-04-03T16:06:43.5111445Z  -e PR_NUMBER \ 2024-04-03T16:06:43.5111794Z  -e GITHUB_ACTIONS \ 2024-04-03T16:06:43.5112174Z  -e GITHUB_REPOSITORY \ 2024-04-03T16:06:43.5112551Z  -e GITHUB_WORKFLOW \ 2024-04-03T16:06:43.5112918Z  -e GITHUB_JOB \ 2024-04-03T16:06:43.5113261Z  -e GITHUB_RUN_ID \ 2024-04-03T16:06:43.5113634Z  -e GITHUB_RUN_NUMBER \ 2024-04-03T16:06:43.5114010Z  -e GITHUB_RUN_ATTEMPT \ 2024-04-03T16:06:43.5114389Z  -e JOB_ID \ 2024-04-03T16:06:43.5114710Z  -e JOB_NAME \ 2024-04-03T16:06:43.5115026Z  -e BASE_SHA \ 2024-04-03T16:06:43.5115348Z  -e BRANCH \ 2024-04-03T16:06:43.5115666Z  -e SHA1 \ 2024-04-03T16:06:43.5115981Z  -e AWS_DEFAULT_REGION \ 2024-04-03T16:06:43.5116369Z  -e IN_WHEEL_TEST \ 2024-04-03T16:06:43.5116731Z  -e SHARD_NUMBER \ 2024-04-03T16:06:43.5117073Z  -e TEST_CONFIG \ 2024-04-03T16:06:43.5117426Z  -e NUM_TEST_SHARDS \ 2024-04-03T16:06:43.5117803Z  -e REENABLED_ISSUES \ 2024-04-03T16:06:43.5118188Z  -e CONTINUE_THROUGH_ERROR \ 2024-04-03T16:06:43.5118596Z  -e VERBOSE_TEST_LOGS \ 2024-04-03T16:06:43.5118978Z  -e NO_TEST_TIMEOUT \ 2024-04-03T16:06:43.5119324Z  -e NO_TD \ 2024-04-03T16:06:43.5119643Z  -e PR_LABELS \ 2024-04-03T16:06:43.5120030Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-04-03T16:06:43.5120543Z  -e SCCACHE_BUCKET \ 2024-04-03T16:06:43.5120925Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-04-03T16:06:43.5121317Z  -e XLA_CUDA \ 2024-04-03T16:06:43.5121688Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-04-03T16:06:43.5122190Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-04-03T16:06:43.5122705Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-04-03T16:06:43.5123198Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-04-03T16:06:43.5123650Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-04-03T16:06:43.5124052Z  -e DASHBOARD_TAG \ 2024-04-03T16:06:43.5124504Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-04-03T16:06:43.5125021Z  --ulimit stack=10485760:83886080 \ 2024-04-03T16:06:43.5125491Z  --security-opt seccomp=unconfined \ 2024-04-03T16:06:43.5126102Z  --cap-add=SYS_PTRACE \ 2024-04-03T16:06:43.5126462Z  --ipc=host \ 2024-04-03T16:06:43.5126802Z  --shm-size="${SHM_SIZE}" \ 2024-04-03T16:06:43.5127187Z  --tty \ 2024-04-03T16:06:43.5127471Z  --detach \ 2024-04-03T16:06:43.5127815Z  --name="${container_name}" \ 2024-04-03T16:06:43.5128216Z  --user jenkins \ 2024-04-03T16:06:43.5128753Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-04-03T16:06:43.5129305Z  -w /var/lib/jenkins/workspace \ 2024-04-03T16:06:43.5129725Z  "${DOCKER_IMAGE}" 2024-04-03T16:06:43.5130044Z ) 2024-04-03T16:06:43.5130442Z # Propagate download.pytorch.org IP to container 2024-04-03T16:06:43.5131380Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-04-03T16:06:43.5132355Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-04-03T16:06:43.5133281Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-04-03T16:06:43.5140469Z shell: /usr/bin/bash -e {0} 2024-04-03T16:06:43.5140801Z env: 2024-04-03T16:06:43.5141058Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:06:43.5141439Z BUILD_ENVIRONMENT: linux-jammy-py3.8-gcc11 2024-04-03T16:06:43.5141874Z PR_NUMBER: 123154 2024-04-03T16:06:43.5142206Z GITHUB_REPOSITORY: pytorch/pytorch 2024-04-03T16:06:43.5142590Z GITHUB_WORKFLOW: pull 2024-04-03T16:06:43.5142906Z GITHUB_JOB: test 2024-04-03T16:06:43.5143202Z GITHUB_RUN_ID: 8536950919 2024-04-03T16:06:43.5143529Z GITHUB_RUN_NUMBER: 192590 2024-04-03T16:06:43.5143868Z GITHUB_RUN_ATTEMPT: 2 2024-04-03T16:06:43.5144175Z JOB_ID: 2024-04-03T16:06:43.5144414Z JOB_NAME: 2024-04-03T16:06:43.5144684Z BRANCH: pull/123154 2024-04-03T16:06:43.5145050Z SHA1: d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:06:43.5145544Z BASE_SHA: 1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:06:43.5146004Z TEST_CONFIG: distributed 2024-04-03T16:06:43.5146328Z SHARD_NUMBER: 2 2024-04-03T16:06:43.5146597Z NUM_TEST_SHARDS: 2 2024-04-03T16:06:43.5146895Z REENABLED_ISSUES: 2024-04-03T16:06:43.5147207Z CONTINUE_THROUGH_ERROR: False 2024-04-03T16:06:43.5147560Z VERBOSE_TEST_LOGS: False 2024-04-03T16:06:43.5147895Z NO_TEST_TIMEOUT: False 2024-04-03T16:06:43.5148205Z NO_TD: False 2024-04-03T16:06:43.5148554Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-04-03T16:06:43.5149028Z SCCACHE_S3_KEY_PREFIX: pull 2024-04-03T16:06:43.5149365Z SHM_SIZE: 1g 2024-04-03T16:06:43.5150674Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:06:43.5151624Z XLA_CUDA: 2024-04-03T16:06:43.5152098Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-04-03T16:06:43.5152707Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-04-03T16:06:43.5153116Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-04-03T16:06:43.5153504Z DASHBOARD_TAG: 2024-04-03T16:06:43.5153800Z HUGGING_FACE_HUB_TOKEN: 2024-04-03T16:06:43.5154110Z ##[endgroup] 2024-04-03T16:06:43.5174016Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-04-03T16:06:43.5174559Z + [[ linux-jammy-py3.8-gcc11 == *onnx* ]] 2024-04-03T16:06:43.5174990Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-04-03T16:06:43.5181630Z +++ nproc --ignore=2 2024-04-03T16:06:43.5216390Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e PR_LABELS -e MAX_JOBS=6 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_8536950919 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:06:43.9979387Z + container_name=4c1cfaf4a8d048d787858cf2354817432c864f82c94c156c2f90005f40fddbdd 2024-04-03T16:06:43.9981149Z + grep download.pytorch.org /etc/hosts 2024-04-03T16:06:43.9983906Z + docker exec -i 4c1cfaf4a8d048d787858cf2354817432c864f82c94c156c2f90005f40fddbdd sudo bash -c '/bin/cat >> /etc/hosts' 2024-04-03T16:06:44.1217033Z + echo DOCKER_CONTAINER_ID=4c1cfaf4a8d048d787858cf2354817432c864f82c94c156c2f90005f40fddbdd 2024-04-03T16:06:44.1220359Z ++ echo dist/torch-2.4.0a0+gitd253bd7-cp38-cp38-linux_x86_64.whl 2024-04-03T16:06:44.1222650Z + docker exec -t 4c1cfaf4a8d048d787858cf2354817432c864f82c94c156c2f90005f40fddbdd sh -c 'pip install dist/torch-2.4.0a0+gitd253bd7-cp38-cp38-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-04-03T16:06:44.7036776Z Processing ./dist/torch-2.4.0a0+gitd253bd7-cp38-cp38-linux_x86_64.whl 2024-04-03T16:06:45.1220872Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.4.0a0+gitd253bd7) (3.13.1) 2024-04-03T16:06:45.1228103Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.4.0a0+gitd253bd7) (4.10.0) 2024-04-03T16:06:45.1231551Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.4.0a0+gitd253bd7) (1.12) 2024-04-03T16:06:45.1235157Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.4.0a0+gitd253bd7) (2.8.8) 2024-04-03T16:06:45.1238362Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.4.0a0+gitd253bd7) (3.1.3) 2024-04-03T16:06:45.1241967Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.4.0a0+gitd253bd7) (2024.2.0) 2024-04-03T16:06:45.1251764Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.4.0a0+gitd253bd7) (3.3.0) 2024-04-03T16:06:45.1309855Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from opt-einsum>=3.3->torch==2.4.0a0+gitd253bd7) (1.21.2) 2024-04-03T16:06:45.1725818Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from jinja2->torch==2.4.0a0+gitd253bd7) (2.1.5) 2024-04-03T16:06:45.1877698Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from sympy->torch==2.4.0a0+gitd253bd7) (1.3.0) 2024-04-03T16:06:46.3233265Z Installing collected packages: torch 2024-04-03T16:06:53.4605285Z Successfully installed torch-2.4.0a0+gitd253bd7 2024-04-03T16:06:53.5310699Z + echo 'Environment variables:' 2024-04-03T16:06:53.5311127Z Environment variables: 2024-04-03T16:06:53.5311427Z + env 2024-04-03T16:06:53.5328833Z INSTALLED_DB=yes 2024-04-03T16:06:53.5329689Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:06:53.5330327Z CONTINUE_THROUGH_ERROR=False 2024-04-03T16:06:53.5330777Z BUILD_ENVIRONMENT=linux-jammy-py3.8-gcc11 2024-04-03T16:06:53.5331327Z HOSTNAME=4c1cfaf4a8d0 2024-04-03T16:06:53.5332260Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_490bef69-80cc-4961-aa21-4746903cc41b 2024-04-03T16:06:53.5333163Z GITHUB_ACTION=__self 2024-04-03T16:06:53.5333611Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-04-03T16:06:53.5334076Z GITHUB_RUN_NUMBER=192590 2024-04-03T16:06:53.5336323Z TEST_CONFIG=distributed 2024-04-03T16:06:53.5336862Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-04-03T16:06:53.5337771Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-04-03T16:06:53.5338749Z GITHUB_TRIGGERING_ACTOR=soulitzer 2024-04-03T16:06:53.5339601Z GITHUB_REF_TYPE=branch 2024-04-03T16:06:53.5340310Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-04-03T16:06:53.5341215Z BASE_SHA=1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:06:53.5342447Z XLA_CUDA= 2024-04-03T16:06:53.5343015Z HUGGING_FACE_HUB_TOKEN= 2024-04-03T16:06:53.5344197Z *** 2024-04-03T16:06:53.5344765Z GITHUB_REPOSITORY_ID=65600975 2024-04-03T16:06:53.5345533Z GITHUB_ACTIONS=true 2024-04-03T16:06:53.5346324Z SHA1=d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:06:53.5347496Z GITHUB_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:06:53.5349145Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/123154/merge 2024-04-03T16:06:53.5350570Z UCC_HOME=/usr 2024-04-03T16:06:53.5351035Z VERBOSE_TEST_LOGS=False 2024-04-03T16:06:53.5351663Z GITHUB_REF=refs/pull/123154/merge 2024-04-03T16:06:53.5352146Z SHARD_NUMBER=2 2024-04-03T16:06:53.5352436Z GITHUB_REF_PROTECTED=false 2024-04-03T16:06:53.5352760Z HOME=/var/lib/jenkins 2024-04-03T16:06:53.5353093Z GITHUB_API_URL=https://api.github.com 2024-04-03T16:06:53.5353516Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-04-03T16:06:53.5353886Z UCX_COMMIT= 2024-04-03T16:06:53.5354143Z SCCACHE_S3_KEY_PREFIX=pull 2024-04-03T16:06:53.5354477Z NUM_TEST_SHARDS=2 2024-04-03T16:06:53.5354754Z UCX_HOME=/usr 2024-04-03T16:06:53.5355593Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_490bef69-80cc-4961-aa21-4746903cc41b 2024-04-03T16:06:53.5356412Z JOB_NAME= 2024-04-03T16:06:53.5357194Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_490bef69-80cc-4961-aa21-4746903cc41b 2024-04-03T16:06:53.5358315Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-04-03T16:06:53.5358982Z GITHUB_EVENT_NAME=pull_request 2024-04-03T16:06:53.5359329Z DASHBOARD_TAG= 2024-04-03T16:06:53.5359596Z GITHUB_RUN_ID=8536950919 2024-04-03T16:06:53.5360630Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_490bef69-80cc-4961-aa21-4746903cc41b 2024-04-03T16:06:53.5361508Z GITHUB_ACTOR=caic99 2024-04-03T16:06:53.5361836Z PR_NUMBER=123154 2024-04-03T16:06:53.5362104Z DESIRED_CUDA= 2024-04-03T16:06:53.5362379Z GITHUB_RUN_ATTEMPT=2 2024-04-03T16:06:53.5362678Z ANACONDA_PYTHON_VERSION=3.8 2024-04-03T16:06:53.5363106Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-04-03T16:06:53.5363551Z TERM=xterm 2024-04-03T16:06:53.5363797Z INSTALLED_VISION=yes 2024-04-03T16:06:53.5364098Z BRANCH=pull/123154 2024-04-03T16:06:53.5364396Z OPENSSL_ROOT_DIR=/opt/openssl 2024-04-03T16:06:53.5364734Z CUDA_PATH=/usr/local/cuda 2024-04-03T16:06:53.5365490Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-04-03T16:06:53.5366244Z GITHUB_SERVER_URL=https://github.com 2024-04-03T16:06:53.5366616Z UCC_COMMIT= 2024-04-03T16:06:53.5366872Z REENABLED_ISSUES= 2024-04-03T16:06:53.5367145Z DOCS=yes 2024-04-03T16:06:53.5367382Z INSTALLED_ANDROID= 2024-04-03T16:06:53.5367659Z SHLVL=1 2024-04-03T16:06:53.5367893Z MAX_JOBS=6 2024-04-03T16:06:53.5368140Z GITHUB_ACTOR_ID=78061359 2024-04-03T16:06:53.5368587Z GITHUB_WORKFLOW_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:06:53.5369098Z GITHUB_REF_NAME=123154/merge 2024-04-03T16:06:53.5369671Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-04-03T16:06:53.5370238Z GITHUB_JOB=test 2024-04-03T16:06:53.5370516Z NO_TEST_TIMEOUT=False 2024-04-03T16:06:53.5370829Z GITHUB_REPOSITORY=pytorch/pytorch 2024-04-03T16:06:53.5371206Z GITHUB_RETENTION_DAYS=90 2024-04-03T16:06:53.5371537Z OPENSSL_DIR=/opt/openssl 2024-04-03T16:06:53.5371851Z GITHUB_ACTION_REPOSITORY= 2024-04-03T16:06:53.5372918Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:06:53.5374186Z GITHUB_BASE_REF=main 2024-04-03T16:06:53.5374472Z INSTALLED_ACL= 2024-04-03T16:06:53.5374740Z CI=true 2024-04-03T16:06:53.5375009Z GITHUB_REPOSITORY_OWNER=pytorch 2024-04-03T16:06:53.5375345Z JOB_ID= 2024-04-03T16:06:53.5375596Z INSTALLED_PROTOBUF=yes 2024-04-03T16:06:53.5375930Z GITHUB_HEAD_REF=patch-1 2024-04-03T16:06:53.5376330Z GITHUB_ACTION_REF= 2024-04-03T16:06:53.5376756Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-04-03T16:06:53.5377218Z GITHUB_WORKFLOW=pull 2024-04-03T16:06:53.5377526Z DEBIAN_FRONTEND=noninteractive 2024-04-03T16:06:53.5378446Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_490bef69-80cc-4961-aa21-4746903cc41b 2024-04-03T16:06:53.5379272Z NO_TD=False 2024-04-03T16:06:53.5379538Z SKIP_SCCACHE_INITIALIZATION=1 2024-04-03T16:06:53.5379887Z _=/usr/bin/env 2024-04-03T16:06:53.5380339Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-04-03T16:06:53.5508968Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch 2024-04-03T16:06:53.5510961Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-04-03T16:06:53.5511893Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib 2024-04-03T16:06:53.5512983Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/test 2024-04-03T16:06:53.5513973Z + BUILD_DIR=build 2024-04-03T16:06:53.5514280Z + BUILD_RENAMED_DIR=build_renamed 2024-04-03T16:06:53.5514652Z + BUILD_BIN_DIR=build/bin 2024-04-03T16:06:53.5514958Z + SHARD_NUMBER=2 2024-04-03T16:06:53.5515244Z + NUM_TEST_SHARDS=2 2024-04-03T16:06:53.5515543Z + export VALGRIND=ON 2024-04-03T16:06:53.5515831Z + VALGRIND=ON 2024-04-03T16:06:53.5516198Z + [[ linux-jammy-py3.8-gcc11 == *clang9* ]] 2024-04-03T16:06:53.5516612Z + [[ 0 == \1 ]] 2024-04-03T16:06:53.5516875Z + [[ False == \1 ]] 2024-04-03T16:06:53.5517252Z + [[ linux-jammy-py3.8-gcc11 != *bazel* ]] 2024-04-03T16:06:53.5517708Z ++ realpath build/custom_test_artifacts 2024-04-03T16:06:53.5537193Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-04-03T16:06:53.5538361Z + [[ -n '' ]] 2024-04-03T16:06:53.5539809Z ++ dirname .ci/pytorch/test.sh 2024-04-03T16:06:53.5556106Z + source .ci/pytorch/common.sh 2024-04-03T16:06:53.5563452Z +++ dirname .ci/pytorch/common.sh 2024-04-03T16:06:53.5569359Z ++ source .ci/pytorch/common_utils.sh 2024-04-03T16:06:53.5576560Z +++ declare -f -t trap_add 2024-04-03T16:06:53.5580600Z ++ set -ex 2024-04-03T16:06:53.5581332Z ++ [[ linux-jammy-py3.8-gcc11 == *rocm* ]] 2024-04-03T16:06:53.5581810Z ++ BUILD_TEST_LIBTORCH=0 2024-04-03T16:06:53.5582188Z + echo 'Environment variables' 2024-04-03T16:06:53.5582553Z Environment variables 2024-04-03T16:06:53.5582850Z + env 2024-04-03T16:06:53.5588530Z INSTALLED_DB=yes 2024-04-03T16:06:53.5589771Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:06:53.5590795Z CONTINUE_THROUGH_ERROR=False 2024-04-03T16:06:53.5591437Z BUILD_ENVIRONMENT=linux-jammy-py3.8-gcc11 2024-04-03T16:06:53.5591856Z HOSTNAME=4c1cfaf4a8d0 2024-04-03T16:06:53.5592698Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_490bef69-80cc-4961-aa21-4746903cc41b 2024-04-03T16:06:53.5593502Z GITHUB_ACTION=__self 2024-04-03T16:06:53.5593824Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-04-03T16:06:53.5594212Z GITHUB_RUN_NUMBER=192590 2024-04-03T16:06:53.5594531Z TEST_CONFIG=distributed 2024-04-03T16:06:53.5594864Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-04-03T16:06:53.5595313Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-04-03T16:06:53.5595715Z GITHUB_TRIGGERING_ACTOR=soulitzer 2024-04-03T16:06:53.5596082Z GITHUB_REF_TYPE=branch 2024-04-03T16:06:53.5596402Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-04-03T16:06:53.5596785Z BASE_SHA=1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:06:53.5597206Z XLA_CUDA= 2024-04-03T16:06:53.5597464Z HUGGING_FACE_HUB_TOKEN= 2024-04-03T16:06:53.5598055Z *** 2024-04-03T16:06:53.5598314Z GITHUB_REPOSITORY_ID=65600975 2024-04-03T16:06:53.5598661Z GITHUB_ACTIONS=true 2024-04-03T16:06:53.5599001Z SHA1=d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:06:53.5599507Z GITHUB_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:06:53.5600223Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/123154/merge 2024-04-03T16:06:53.5600946Z UCC_HOME=/usr 2024-04-03T16:06:53.5601319Z VERBOSE_TEST_LOGS=False 2024-04-03T16:06:53.5601650Z GITHUB_REF=refs/pull/123154/merge 2024-04-03T16:06:53.5601991Z SHARD_NUMBER=2 2024-04-03T16:06:53.5602279Z GITHUB_REF_PROTECTED=false 2024-04-03T16:06:53.5602607Z HOME=/var/lib/jenkins 2024-04-03T16:06:53.5602938Z GITHUB_API_URL=https://api.github.com 2024-04-03T16:06:53.5603362Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-04-03T16:06:53.5603731Z UCX_COMMIT= 2024-04-03T16:06:53.5603989Z SCCACHE_S3_KEY_PREFIX=pull 2024-04-03T16:06:53.5604313Z NUM_TEST_SHARDS=2 2024-04-03T16:06:53.5604588Z UCX_HOME=/usr 2024-04-03T16:06:53.5605427Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_490bef69-80cc-4961-aa21-4746903cc41b 2024-04-03T16:06:53.5606243Z JOB_NAME= 2024-04-03T16:06:53.5607016Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_490bef69-80cc-4961-aa21-4746903cc41b 2024-04-03T16:06:53.5608138Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-04-03T16:06:53.5608803Z GITHUB_EVENT_NAME=pull_request 2024-04-03T16:06:53.5609150Z DASHBOARD_TAG= 2024-04-03T16:06:53.5609414Z GITHUB_RUN_ID=8536950919 2024-04-03T16:06:53.5610335Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_490bef69-80cc-4961-aa21-4746903cc41b 2024-04-03T16:06:53.5611206Z GITHUB_ACTOR=caic99 2024-04-03T16:06:53.5611485Z PR_NUMBER=123154 2024-04-03T16:06:53.5611761Z DESIRED_CUDA= 2024-04-03T16:06:53.5612033Z GITHUB_RUN_ATTEMPT=2 2024-04-03T16:06:53.5612312Z VALGRIND=ON 2024-04-03T16:06:53.5612626Z ANACONDA_PYTHON_VERSION=3.8 2024-04-03T16:06:53.5613047Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-04-03T16:06:53.5628930Z TERM=xterm 2024-04-03T16:06:53.5629255Z INSTALLED_VISION=yes 2024-04-03T16:06:53.5629856Z BRANCH=pull/123154 2024-04-03T16:06:53.5630183Z OPENSSL_ROOT_DIR=/opt/openssl 2024-04-03T16:06:53.5630530Z CUDA_PATH=/usr/local/cuda 2024-04-03T16:06:53.5631351Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-04-03T16:06:53.5632123Z GITHUB_SERVER_URL=https://github.com 2024-04-03T16:06:53.5632494Z UCC_COMMIT= 2024-04-03T16:06:53.5632758Z REENABLED_ISSUES= 2024-04-03T16:06:53.5633036Z DOCS=yes 2024-04-03T16:06:53.5633273Z INSTALLED_ANDROID= 2024-04-03T16:06:53.5633555Z SHLVL=1 2024-04-03T16:06:53.5633792Z MAX_JOBS=6 2024-04-03T16:06:53.5634039Z GITHUB_ACTOR_ID=78061359 2024-04-03T16:06:53.5634493Z GITHUB_WORKFLOW_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:06:53.5635008Z GITHUB_REF_NAME=123154/merge 2024-04-03T16:06:53.5635595Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-04-03T16:06:53.5636165Z GITHUB_JOB=test 2024-04-03T16:06:53.5636464Z NO_TEST_TIMEOUT=False 2024-04-03T16:06:53.5636784Z GITHUB_REPOSITORY=pytorch/pytorch 2024-04-03T16:06:53.5637164Z GITHUB_RETENTION_DAYS=90 2024-04-03T16:06:53.5637491Z OPENSSL_DIR=/opt/openssl 2024-04-03T16:06:53.5637810Z GITHUB_ACTION_REPOSITORY= 2024-04-03T16:06:53.5638870Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:06:53.5640196Z GITHUB_BASE_REF=main 2024-04-03T16:06:53.5640771Z INSTALLED_ACL= 2024-04-03T16:06:53.5641183Z CI=true 2024-04-03T16:06:53.5641626Z GITHUB_REPOSITORY_OWNER=pytorch 2024-04-03T16:06:53.5642096Z JOB_ID= 2024-04-03T16:06:53.5642514Z INSTALLED_PROTOBUF=yes 2024-04-03T16:06:53.5643156Z GITHUB_HEAD_REF=patch-1 2024-04-03T16:06:53.5643674Z GITHUB_ACTION_REF= 2024-04-03T16:06:53.5644378Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-04-03T16:06:53.5645312Z GITHUB_WORKFLOW=pull 2024-04-03T16:06:53.5645845Z DEBIAN_FRONTEND=noninteractive 2024-04-03T16:06:53.5647576Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_490bef69-80cc-4961-aa21-4746903cc41b 2024-04-03T16:06:53.5649182Z NO_TD=False 2024-04-03T16:06:53.5649693Z SKIP_SCCACHE_INITIALIZATION=1 2024-04-03T16:06:53.5650324Z _=/usr/bin/env 2024-04-03T16:06:53.5651037Z + echo 'Testing pytorch' 2024-04-03T16:06:53.5651565Z Testing pytorch 2024-04-03T16:06:53.5652072Z + export LANG=C.UTF-8 2024-04-03T16:06:53.5652616Z + LANG=C.UTF-8 2024-04-03T16:06:53.5653095Z + PR_NUMBER=123154 2024-04-03T16:06:53.5653644Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-04-03T16:06:53.5654399Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-04-03T16:06:53.5655312Z + [[ linux-jammy-py3.8-gcc11 == *rocm* ]] 2024-04-03T16:06:53.5656050Z + [[ distributed == \s\l\o\w ]] 2024-04-03T16:06:53.5656913Z + [[ linux-jammy-py3.8-gcc11 == *slow-gradcheck* ]] 2024-04-03T16:06:53.5657870Z + [[ linux-jammy-py3.8-gcc11 == *cuda* ]] 2024-04-03T16:06:53.5658749Z + [[ linux-jammy-py3.8-gcc11 == *rocm* ]] 2024-04-03T16:06:53.5659623Z + [[ linux-jammy-py3.8-gcc11 == *xpu* ]] 2024-04-03T16:06:53.5660376Z + [[ distributed == *crossref* ]] 2024-04-03T16:06:53.5661197Z + [[ linux-jammy-py3.8-gcc11 == *rocm* ]] 2024-04-03T16:06:53.5662084Z + [[ linux-jammy-py3.8-gcc11 == *xpu* ]] 2024-04-03T16:06:53.5662991Z + [[ linux-jammy-py3.8-gcc11 != *-bazel-* ]] 2024-04-03T16:06:53.5663872Z + pip_install --user ninja==1.10.2 2024-04-03T16:06:53.5664803Z + pip install --progress-bar off --user ninja==1.10.2 2024-04-03T16:06:53.9651028Z Collecting ninja==1.10.2 2024-04-03T16:06:53.9800481Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-04-03T16:06:53.9954013Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-04-03T16:06:55.0859650Z Installing collected packages: ninja 2024-04-03T16:06:55.0945210Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-04-03T16:06:55.0995770Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-04-03T16:06:55.0996734Z Successfully installed ninja-1.10.2 2024-04-03T16:06:55.1747060Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:06:55.1749295Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:06:55.1750796Z + install_tlparse 2024-04-03T16:06:55.1751245Z + pip_install --user tlparse==0.3.7 2024-04-03T16:06:55.1751801Z + pip install --progress-bar off --user tlparse==0.3.7 2024-04-03T16:06:55.5620081Z Collecting tlparse==0.3.7 2024-04-03T16:06:55.5985353Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-04-03T16:06:55.6107025Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-04-03T16:06:56.7323300Z Installing collected packages: tlparse 2024-04-03T16:06:56.7727232Z Successfully installed tlparse-0.3.7 2024-04-03T16:06:56.8466317Z ++ python -m site --user-base 2024-04-03T16:06:56.8675110Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:06:56.8677896Z + [[ linux-jammy-py3.8-gcc11 == *asan* ]] 2024-04-03T16:06:56.8678729Z + [[ linux-jammy-py3.8-gcc11 == *-debug* ]] 2024-04-03T16:06:56.8679562Z + [[ linux-jammy-py3.8-gcc11 != *-bazel-* ]] 2024-04-03T16:06:56.8680638Z + echo 'We are not in debug mode: linux-jammy-py3.8-gcc11. Expect the assertion to pass' 2024-04-03T16:06:56.8681882Z We are not in debug mode: linux-jammy-py3.8-gcc11. Expect the assertion to pass 2024-04-03T16:06:56.8682475Z + cd test 2024-04-03T16:06:56.8682996Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-04-03T16:06:58.0563494Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-04-03T16:06:58.0564503Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-04-03T16:06:58.0565227Z + DYNAMO_BENCHMARK_FLAGS=() 2024-04-03T16:06:58.0565773Z + [[ distributed == *dynamo_eager* ]] 2024-04-03T16:06:58.0566207Z + [[ distributed == *aot_eager* ]] 2024-04-03T16:06:58.0566617Z + [[ distributed == *aot_inductor* ]] 2024-04-03T16:06:58.0567026Z + [[ distributed == *inductor* ]] 2024-04-03T16:06:58.0567420Z + [[ distributed == *dynamic* ]] 2024-04-03T16:06:58.0567798Z + [[ distributed == *cpu_inductor* ]] 2024-04-03T16:06:58.0568359Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-04-03T16:06:58.0574131Z + [[ linux-jammy-py3.8-gcc11 == *tbb* ]] 2024-04-03T16:06:58.0583213Z + [[ linux-jammy-py3.8-gcc11 == *libtorch* ]] 2024-04-03T16:06:58.0583784Z + [[ linux-jammy-py3.8-gcc11 == *-bazel-* ]] 2024-04-03T16:06:58.0585879Z + cd test 2024-04-03T16:06:58.0586835Z + python -c 'import torch; print(torch.__config__.show())' 2024-04-03T16:06:59.0551005Z PyTorch built with: 2024-04-03T16:06:59.0551551Z - GCC 11.4 2024-04-03T16:06:59.0551874Z - C++ Version: 201703 2024-04-03T16:06:59.0552790Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-04-03T16:06:59.0553872Z - Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361) 2024-04-03T16:06:59.0554531Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-04-03T16:06:59.0555033Z - LAPACK is enabled (usually provided by MKL) 2024-04-03T16:06:59.0555487Z - NNPACK is enabled 2024-04-03T16:06:59.0555855Z - CPU capability usage: AVX512 2024-04-03T16:06:59.0563672Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-04-03T16:06:59.0570849Z 2024-04-03T16:06:59.2337431Z + cd test 2024-04-03T16:06:59.2338106Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-04-03T16:07:00.2141293Z ATen/Parallel: 2024-04-03T16:07:00.2142012Z at::get_num_threads() : 4 2024-04-03T16:07:00.2142732Z at::get_num_interop_threads() : 8 2024-04-03T16:07:00.2143518Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-04-03T16:07:00.2144227Z omp_get_max_threads() : 4 2024-04-03T16:07:00.2146046Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-04-03T16:07:00.2147710Z mkl_get_max_threads() : 4 2024-04-03T16:07:00.2148861Z Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361) 2024-04-03T16:07:00.2150356Z std::thread::hardware_concurrency() : 8 2024-04-03T16:07:00.2151558Z Environment variables: 2024-04-03T16:07:00.2152190Z OMP_NUM_THREADS : [not set] 2024-04-03T16:07:00.2152858Z MKL_NUM_THREADS : [not set] 2024-04-03T16:07:00.2153559Z ATen parallel backend: OpenMP 2024-04-03T16:07:00.2154038Z 2024-04-03T16:07:00.3912553Z + [[ distributed == *backward* ]] 2024-04-03T16:07:00.3913338Z + [[ distributed == *xla* ]] 2024-04-03T16:07:00.3913994Z + [[ distributed == *executorch* ]] 2024-04-03T16:07:00.3914449Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-04-03T16:07:00.3915066Z + [[ linux-jammy-py3.8-gcc11 == *libtorch* ]] 2024-04-03T16:07:00.3915524Z + [[ distributed == distributed ]] 2024-04-03T16:07:00.3915901Z + test_distributed 2024-04-03T16:07:00.3916273Z + echo 'Testing distributed python tests' 2024-04-03T16:07:00.3916712Z Testing distributed python tests 2024-04-03T16:07:00.3917313Z + python test/run_test.py --distributed-tests --shard 2 2 --verbose 2024-04-03T16:07:00.4742647Z test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-04-03T16:07:00.4743665Z import pkg_resources 2024-04-03T16:07:02.7939961Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-04-03T16:07:02.9062752Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-04-03T16:07:02.9663255Z Ignoring disabled issues: [''] 2024-04-03T16:07:02.9731471Z Found test times from artifacts 2024-04-03T16:07:02.9990532Z Found test times from artifacts 2024-04-03T16:07:03.0000088Z Running all tests 2024-04-03T16:07:03.0126617Z Name: tests to run (est. time: 55.07min) 2024-04-03T16:07:03.0127312Z Serial tests (107): 2024-04-03T16:07:03.0127950Z distributed/checkpoint/test_nested_dict 1/1 2024-04-03T16:07:03.0128776Z distributed/_shard/sharded_tensor/test_logger 1/1 2024-04-03T16:07:03.0129722Z distributed/_composable/test_contract 1/1 2024-04-03T16:07:03.0130614Z distributed/checkpoint/test_traverse 1/1 2024-04-03T16:07:03.0131456Z distributed/fsdp/test_fsdp_fx 1/1 2024-04-03T16:07:03.0132283Z distributed/_spmd/test_transformation 1/1 2024-04-03T16:07:03.0133076Z distributed/_tensor/test_init 1/1 2024-04-03T16:07:03.0133584Z distributed/_composable/fsdp/test_fully_shard_state 1/1 2024-04-03T16:07:03.0134144Z distributed/checkpoint/test_dedup_tensors 1/1 2024-04-03T16:07:03.0134919Z distributed/checkpoint/test_compatibility 1/1 2024-04-03T16:07:03.0135689Z distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 2024-04-03T16:07:03.0136176Z distributed/_tensor/test_common_rules 1/1 2024-04-03T16:07:03.0137109Z distributed/_composable/test_checkpoint 1/1 2024-04-03T16:07:03.0137953Z distributed/_tensor/test_dtensor_compile 1/1 2024-04-03T16:07:03.0138796Z distributed/test_c10d_functional_native 1/1 2024-04-03T16:07:03.0139681Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 2024-04-03T16:07:03.0140639Z distributed/checkpoint/test_save_load_api 1/1 2024-04-03T16:07:03.0141641Z distributed/_tensor/test_dtensor_ops 1/1 2024-04-03T16:07:03.0142297Z distributed/test_distributed_spawn 1/3 2024-04-03T16:07:03.0142755Z distributed/test_distributed_spawn 3/3 2024-04-03T16:07:03.0143298Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 2024-04-03T16:07:03.0143848Z distributed/_tensor/test_redistribute 1/1 2024-04-03T16:07:03.0144316Z distributed/test_collective_utils 1/1 2024-04-03T16:07:03.0144830Z distributed/tensor/parallel/test_tp_random_state 1/1 2024-04-03T16:07:03.0145326Z distributed/_tensor/test_math_ops 1/1 2024-04-03T16:07:03.0145813Z distributed/checkpoint/test_hsdp_checkpoint 1/1 2024-04-03T16:07:03.0146306Z distributed/_spmd/test_tracing 1/1 2024-04-03T16:07:03.0146762Z distributed/checkpoint/e2e/test_fine_tuning 1/1 2024-04-03T16:07:03.0147262Z distributed/_tensor/test_matrix_ops 1/1 2024-04-03T16:07:03.0148069Z distributed/_composable/fully_shard/test_fully_shard_compile 1/1 2024-04-03T16:07:03.0148686Z distributed/checkpoint/test_fsdp_model_state 1/1 2024-04-03T16:07:03.0149238Z distributed/checkpoint/test_state_dict_utils 1/1 2024-04-03T16:07:03.0150068Z distributed/tensor/parallel/test_fsdp_2d_parallel 1/1 2024-04-03T16:07:03.0150599Z distributed/_tensor/debug/test_comm_mode 1/1 2024-04-03T16:07:03.0151268Z distributed/tensor/parallel/test_tp_examples 1/1 2024-04-03T16:07:03.0151759Z distributed/test_c10d_pypg 1/1 2024-04-03T16:07:03.0152254Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 2024-04-03T16:07:03.0152825Z distributed/fsdp/test_distributed_checkpoint 1/1 2024-04-03T16:07:03.0153320Z distributed/_tensor/test_utils 1/1 2024-04-03T16:07:03.0153857Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 2024-04-03T16:07:03.0154671Z distributed/fsdp/test_fsdp_flatten_params 1/1 2024-04-03T16:07:03.0155323Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 1/1 2024-04-03T16:07:03.0155956Z distributed/checkpoint/test_utils 1/1 2024-04-03T16:07:03.0156384Z distributed/_tensor/test_view_ops 1/1 2024-04-03T16:07:03.0156933Z distributed/_composable/fully_shard/test_fully_shard_init 1/1 2024-04-03T16:07:03.0157480Z distributed/test_functional_api 1/1 2024-04-03T16:07:03.0157938Z distributed/fsdp/test_fsdp_ignored_modules 1/1 2024-04-03T16:07:03.0158464Z distributed/fsdp/test_fsdp_tp_integration 1/1 2024-04-03T16:07:03.0158971Z distributed/fsdp/test_checkpoint_wrapper 1/1 2024-04-03T16:07:03.0159442Z distributed/fsdp/test_fsdp_grad_acc 1/1 2024-04-03T16:07:03.0159904Z distributed/checkpoint/test_planner 1/1 2024-04-03T16:07:03.0160433Z distributed/fsdp/test_fsdp_apply 1/1 2024-04-03T16:07:03.0160924Z distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 2024-04-03T16:07:03.0161461Z distributed/pipeline/sync/test_pipeline 1/1 2024-04-03T16:07:03.0162021Z distributed/_composable/test_replicate_with_compiler 1/1 2024-04-03T16:07:03.0162565Z distributed/fsdp/test_fsdp_state_dict 1/1 2024-04-03T16:07:03.0163048Z distributed/fsdp/test_fsdp_traversal 1/1 2024-04-03T16:07:03.0163505Z distributed/fsdp/test_fsdp_input 1/1 2024-04-03T16:07:03.0163930Z distributed/fsdp/test_fsdp_uneven 1/1 2024-04-03T16:07:03.0164423Z distributed/fsdp/test_fsdp_multiple_wrapping 1/1 2024-04-03T16:07:03.0165042Z distributed/_composable/fully_shard/test_fully_shard_runtime 1/1 2024-04-03T16:07:03.0165633Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 2024-04-03T16:07:03.0166132Z distributed/fsdp/test_fsdp_exec_order 1/1 2024-04-03T16:07:03.0166683Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 2024-04-03T16:07:03.0167201Z distributed/fsdp/test_wrap 1/1 2024-04-03T16:07:03.0167702Z distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 2024-04-03T16:07:03.0168262Z distributed/fsdp/test_fsdp_comm_hooks 1/1 2024-04-03T16:07:03.0168824Z distributed/algorithms/quantization/test_quantization 1/1 2024-04-03T16:07:03.0169395Z distributed/fsdp/test_fsdp_use_orig_params 1/1 2024-04-03T16:07:03.0169873Z distributed/fsdp/test_utils 1/1 2024-04-03T16:07:03.0170311Z distributed/pipeline/sync/test_phony 1/1 2024-04-03T16:07:03.0170782Z distributed/test_c10d_object_collectives 1/1 2024-04-03T16:07:03.0171293Z distributed/fsdp/test_fsdp_multiple_forward 1/1 2024-04-03T16:07:03.0171785Z distributed/fsdp/test_fsdp_comm 1/1 2024-04-03T16:07:03.0172216Z distributed/pipeline/sync/test_bugs 1/1 2024-04-03T16:07:03.0172722Z distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 2024-04-03T16:07:03.0173251Z distributed/pipeline/sync/skip/test_gpipe 1/1 2024-04-03T16:07:03.0173717Z distributed/algorithms/test_join 1/1 2024-04-03T16:07:03.0174246Z distributed/_shard/sharded_tensor/ops/test_embedding 1/1 2024-04-03T16:07:03.0174870Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 2024-04-03T16:07:03.0175428Z distributed/pipeline/sync/test_inplace 1/1 2024-04-03T16:07:03.0176087Z distributed/rpc/cuda/test_tensorpipe_agent 1/1 2024-04-03T16:07:03.0176599Z distributed/rpc/test_tensorpipe_agent 2/3 2024-04-03T16:07:03.0177062Z distributed/pipeline/sync/test_copy 1/1 2024-04-03T16:07:03.0177560Z distributed/fsdp/test_fsdp_mixed_precision 1/1 2024-04-03T16:07:03.0178136Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 2024-04-03T16:07:03.0178718Z distributed/test_pg_wrapper 1/1 2024-04-03T16:07:03.0179128Z distributed/test_c10d_common 1/1 2024-04-03T16:07:03.0179592Z distributed/pipeline/sync/skip/test_tracker 1/1 2024-04-03T16:07:03.0180101Z distributed/pipeline/sync/skip/test_leak 1/1 2024-04-03T16:07:03.0180697Z distributed/pipeline/sync/skip/test_stash_pop 1/1 2024-04-03T16:07:03.0181194Z distributed/test_c10d_spawn_gloo 1/1 2024-04-03T16:07:03.0181602Z distributed/test_c10d_ucc 1/1 2024-04-03T16:07:03.0182092Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 2024-04-03T16:07:03.0182649Z distributed/pipeline/sync/test_stream 1/1 2024-04-03T16:07:03.0183110Z distributed/test_c10d_spawn_nccl 1/1 2024-04-03T16:07:03.0183528Z distributed/test_c10d_spawn_ucc 1/1 2024-04-03T16:07:03.0183988Z distributed/pipeline/sync/test_balance 1/1 2024-04-03T16:07:03.0184584Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 2024-04-03T16:07:03.0185188Z distributed/pipeline/sync/test_dependency 1/1 2024-04-03T16:07:03.0185700Z distributed/pipeline/sync/skip/test_api 1/1 2024-04-03T16:07:03.0186249Z distributed/pipeline/sync/test_deferred_batch_norm 1/1 2024-04-03T16:07:03.0186785Z distributed/checkpoint/test_state_dict 1/1 2024-04-03T16:07:03.0187236Z distributed/test_c10d_gloo 1/1 2024-04-03T16:07:03.0187627Z distributed/test_c10d_nccl 1/1 2024-04-03T16:07:03.0188044Z distributed/elastic/utils/logging_test 1/1 2024-04-03T16:07:03.0188517Z distributed/elastic/utils/util_test 1/1 2024-04-03T16:07:03.0188946Z Parallel tests (0): 2024-04-03T16:07:03.0189273Z Name: excluded (est. time: 0.0min) 2024-04-03T16:07:03.0189817Z Serial tests (0): 2024-04-03T16:07:03.0190209Z Parallel tests (0): 2024-04-03T16:07:03.0190877Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-04-03T16:07:03.0259218Z Running distributed/checkpoint/test_nested_dict 1/1 ... [2024-04-03 16:07:03.025471] 2024-04-03T16:07:03.0266736Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_nested_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:07:03.026163] 2024-04-03T16:07:06.8999831Z 2024-04-03T16:07:06.9001790Z distributed/checkpoint/test_nested_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_nested_dict_1.1_6dceaf3748e07c38_.log 2024-04-03T16:07:06.9004066Z Running 2 items in this shard: test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_flattening_round_trip, test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_mapping 2024-04-03T16:07:06.9005240Z 2024-04-03T16:07:06.9008610Z Running distributed/_shard/sharded_tensor/test_logger 1/1 ... [2024-04-03 16:07:06.900654] 2024-04-03T16:07:06.9014271Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:07:06.901089] 2024-04-03T16:07:09.4736297Z 2024-04-03T16:07:09.4739067Z distributed/_shard/sharded_tensor/test_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_4227686bc69b9cbc_.log 2024-04-03T16:07:09.4742565Z Running 1 items in this shard: test/distributed/_shard/sharded_tensor/test_logger.py::ShardingSpecLoggerTest::test_get_or_create_logger 2024-04-03T16:07:09.4744253Z 2024-04-03T16:07:09.4745842Z Running distributed/_composable/test_contract 1/1 ... [2024-04-03 16:07:09.474340] 2024-04-03T16:07:09.4753376Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/test_contract.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:07:09.474848] 2024-04-03T16:07:12.0972905Z 2024-04-03T16:07:12.0974956Z distributed/_composable/test_contract 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_contract_1.1_4695269369407fe2_.log 2024-04-03T16:07:12.0978081Z Running 4 items in this shard: test/distributed/_composable/test_contract.py::TestContract::test_add_hooks, test/distributed/_composable/test_contract.py::TestContract::test_modify_fqn, test/distributed/_composable/test_contract.py::TestContract::test_registry, test/distributed/_composable/test_contract.py::TestContract::test_state 2024-04-03T16:07:12.0980219Z 2024-04-03T16:07:12.0980644Z Running distributed/checkpoint/test_traverse 1/1 ... [2024-04-03 16:07:12.097744] 2024-04-03T16:07:12.0985627Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_traverse.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:07:12.098150] 2024-04-03T16:07:14.7207325Z 2024-04-03T16:07:14.7209263Z distributed/checkpoint/test_traverse 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_traverse_1.1_b180125bb6150f25_.log 2024-04-03T16:07:14.7213819Z Running 7 items in this shard: test/distributed/checkpoint/test_traverse.py::TestTraverse::test_get_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_set_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_doesnt_ignore_intermediate_collections, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_dict, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_list, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_shallow, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_with_ordered_dict 2024-04-03T16:07:14.7218038Z 2024-04-03T16:07:14.7218447Z Running distributed/fsdp/test_fsdp_fx 1/1 ... [2024-04-03 16:07:14.721509] 2024-04-03T16:07:14.7233770Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:07:14.723023] 2024-04-03T16:07:17.3455563Z 2024-04-03T16:07:17.3457429Z distributed/fsdp/test_fsdp_fx 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_fx_1.1_ad69edc1ae019671_.log 2024-04-03T16:07:17.3459101Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_fx.py::TestSymbolicTracing::test_symbolic_tracing_outputs 2024-04-03T16:07:17.3459854Z 2024-04-03T16:07:17.3462408Z Running distributed/_spmd/test_transformation 1/1 ... [2024-04-03 16:07:17.346012] 2024-04-03T16:07:17.3467122Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_spmd/test_transformation.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:07:17.346402] 2024-04-03T16:07:19.8669951Z 2024-04-03T16:07:19.8671611Z distributed/_spmd/test_transformation 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._spmd.test_transformation_1.1_81ba4f1dc712fdeb_.log 2024-04-03T16:07:19.8672727Z 2024-04-03T16:07:19.8676053Z Running distributed/_tensor/test_init 1/1 ... [2024-04-03 16:07:19.867362] 2024-04-03T16:07:19.8681760Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:07:19.867780] 2024-04-03T16:07:40.5772228Z 2024-04-03T16:07:40.5775405Z distributed/_tensor/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_init_1.1_702115d1e5b7f6ee_.log 2024-04-03T16:07:40.5779790Z Running 7 items in this shard: test/distributed/_tensor/test_init.py::DTensorInitOpsTest::test_init_ops, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_empty, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_full, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_ones, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros_full_mesh, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros_submesh 2024-04-03T16:07:40.5783043Z 2024-04-03T16:07:40.5783565Z Running distributed/_composable/fsdp/test_fully_shard_state 1/1 ... [2024-04-03 16:07:40.577680] 2024-04-03T16:07:40.5785676Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:07:40.578090] 2024-04-03T16:07:43.2006790Z 2024-04-03T16:07:43.2009331Z distributed/_composable/fsdp/test_fully_shard_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_6f2742ae38126607_.log 2024-04-03T16:07:43.2014168Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_cls, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_deepcopy, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_reapply, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_state, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_unsupported_module_cls 2024-04-03T16:07:43.2017814Z 2024-04-03T16:07:43.2018280Z Running distributed/checkpoint/test_dedup_tensors 1/1 ... [2024-04-03 16:07:43.201160] 2024-04-03T16:07:43.2020493Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_dedup_tensors.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:07:43.201595] 2024-04-03T16:07:45.8744766Z 2024-04-03T16:07:45.8748273Z distributed/checkpoint/test_dedup_tensors 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_382a48f1e7fdd752_.log 2024-04-03T16:07:45.8750229Z Running 1 items in this shard: test/distributed/checkpoint/test_dedup_tensors.py::TestDedupTensor::test_dedup_shards 2024-04-03T16:07:45.8751138Z 2024-04-03T16:07:45.8752465Z Running distributed/checkpoint/test_compatibility 1/1 ... [2024-04-03 16:07:45.875000] 2024-04-03T16:07:45.8757291Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_compatibility.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:07:45.875441] 2024-04-03T16:07:48.4980267Z 2024-04-03T16:07:48.4982038Z distributed/checkpoint/test_compatibility 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_compatibility_1.1_a7c752493a17bc0d_.log 2024-04-03T16:07:48.4984900Z Running 2 items in this shard: test/distributed/checkpoint/test_compatibility.py::TestDCPCompatbility::test_metadata, test/distributed/checkpoint/test_compatibility.py::TestDCPCompatbility::test_sharded_tensor_dependency 2024-04-03T16:07:48.4986382Z 2024-04-03T16:07:48.4986974Z Running distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 ... [2024-04-03 16:07:48.498454] 2024-04-03T16:07:48.4991407Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/fsdp/test_fsdp_dsd.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:07:48.498844] 2024-04-03T16:07:51.1211865Z 2024-04-03T16:07:51.1213888Z distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.fsdp.test_fsdp_dsd_1.1_ce5df97814a5d9f9_.log 2024-04-03T16:07:51.1216707Z Running 2 items in this shard: test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_1d_fsdp_cpu_offload_full_model_state_dict, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_1d_fsdp_get_model_state_dict 2024-04-03T16:07:51.1218820Z 2024-04-03T16:07:51.1219268Z Running distributed/_tensor/test_common_rules 1/1 ... [2024-04-03 16:07:51.121680] 2024-04-03T16:07:51.1223825Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_common_rules.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:07:51.122102] 2024-04-03T16:08:18.8981898Z 2024-04-03T16:08:18.8983793Z distributed/_tensor/test_common_rules 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_common_rules_1.1_6af0d3df98285343_.log 2024-04-03T16:08:18.8990465Z Running 10 items in this shard: test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_basic_propagation, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_errors, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_linearity, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_merge_sharding, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_pointwise_propagation, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_enforce_sharding_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_broadcasting, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_suggestion 2024-04-03T16:08:18.8995906Z 2024-04-03T16:08:18.8996359Z Running distributed/_composable/test_checkpoint 1/1 ... [2024-04-03 16:08:18.898710] 2024-04-03T16:08:18.8998366Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/test_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:08:18.899148] 2024-04-03T16:08:21.5215480Z 2024-04-03T16:08:21.5217581Z distributed/_composable/test_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_checkpoint_1.1_cca6209a41e6ec05_.log 2024-04-03T16:08:21.5221247Z Running 5 items in this shard: test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_clears_state_on_error_in_forward, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_multi_args, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_random_cpu, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_tensor_only_cpu, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_tensor_only_gpu 2024-04-03T16:08:21.5224286Z 2024-04-03T16:08:21.5224787Z Running distributed/_tensor/test_dtensor_compile 1/1 ... [2024-04-03 16:08:21.521982] 2024-04-03T16:08:21.5226988Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_dtensor_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:08:21.522360] 2024-04-03T16:08:53.5049640Z 2024-04-03T16:08:53.5051693Z distributed/_tensor/test_dtensor_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_compile_1.1_7f68c87ffc75bd4b_.log 2024-04-03T16:08:53.5076802Z Running 32 items in this shard: test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_different_gradient_placement, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamo_device_mesh_attrs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_noncontiguous_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_graph_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs_forward_hook, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False_use_native_funcol_True, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True_use_native_funcol_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True_use_native_funcol_True 2024-04-03T16:08:53.5106582Z 2024-04-03T16:08:53.5107327Z Running distributed/test_c10d_functional_native 1/1 ... [2024-04-03 16:08:53.505595] 2024-04-03T16:08:53.5110505Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_functional_native.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:08:53.506026] 2024-04-03T16:08:55.8289925Z 2024-04-03T16:08:55.8291766Z distributed/test_c10d_functional_native 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_functional_native_1.1_d4ed45433d82ad20_.log 2024-04-03T16:08:55.8304818Z Running 23 items in this shard: test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_to_all_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_broadcast, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_threading, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_unwaited, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_to_all_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_broadcast, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_inplace_op_on_view, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reuse_buffer_after_inplace_collective, test/distributed/test_c10d_functional_native.py::CompileTest::test_ranks_and_tag 2024-04-03T16:08:55.8316774Z 2024-04-03T16:08:55.8317253Z Running distributed/checkpoint/e2e/test_fsdp_ep 1/1 ... [2024-04-03 16:08:55.829450] 2024-04-03T16:08:55.8319275Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/e2e/test_fsdp_ep.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:08:55.829827] 2024-04-03T16:08:58.4521240Z 2024-04-03T16:08:58.4523294Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_1d3aa4e78640342b_.log 2024-04-03T16:08:58.4525011Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_fsdp_ep.py::TestFSDPWithEP::test_e2e 2024-04-03T16:08:58.4525676Z 2024-04-03T16:08:58.4528468Z Running distributed/checkpoint/test_save_load_api 1/1 ... [2024-04-03 16:08:58.452643] 2024-04-03T16:08:58.4533582Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_save_load_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:08:58.453071] 2024-04-03T16:09:03.5293962Z 2024-04-03T16:09:03.5296675Z distributed/checkpoint/test_save_load_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_save_load_api_1.1_0ea29ea45c118ffa_.log 2024-04-03T16:09:03.5300013Z Running 1 items in this shard: test/distributed/checkpoint/test_save_load_api.py::TestSaveAndLoadAPI::test_auto_detect 2024-04-03T16:09:03.5301226Z 2024-04-03T16:09:03.5303215Z Running distributed/_tensor/test_dtensor_ops 1/1 ... [2024-04-03 16:09:03.530072] 2024-04-03T16:09:03.5310245Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_dtensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:09:03.530579] 2024-04-03T16:10:07.9796270Z 2024-04-03T16:10:07.9798472Z distributed/_tensor/test_dtensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_ops_1.1_78adfb662063b683_.log 2024-04-03T16:10:08.0482668Z Running 655 items in this shard: test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_H_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_T_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db___getitem___cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db___radd___cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db___rdiv___cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db___rmatmul___cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db___rmod___cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db___rmul___cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db___rpow___cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db___rsub___cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db__batch_norm_with_update_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db__chunk_cat_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db__native_batch_norm_legit_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db__segment_reduce_lengths_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db__segment_reduce_offsets_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db__softmax_backward_data_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db__upsample_bilinear2d_aa_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_abs_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_acos_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_acosh_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_add_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_addbmm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_addcdiv_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_addcmul_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_addmm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_addmm_decomposed_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_addmv_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_addr_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_all_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_allclose_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_amax_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_amin_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_aminmax_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_angle_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_any_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_arange_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_argmax_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_argmin_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_argsort_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_argwhere_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_as_strided_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_as_strided_partial_views_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_as_strided_scatter_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_asin_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_asinh_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_atan2_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_atan_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_atanh_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_atleast_1d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_atleast_2d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_atleast_3d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_baddbmm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_bernoulli_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_bfloat16_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_block_diag_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_bmm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_bool_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_broadcast_shapes_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_broadcast_tensors_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_broadcast_to_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_bucketize_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_byte_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_cartesian_prod_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_cat_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_cauchy_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_cdist_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_cdouble_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_ceil_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_cfloat_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_chalf_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_char_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_cholesky_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_cholesky_inverse_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_cholesky_solve_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_chunk_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_clamp_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_clamp_max_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_clamp_min_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_clone_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_column_stack_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_combinations_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_complex_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_conj_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_conj_physical_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_constant_pad_nd_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_contiguous_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_copysign_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_corrcoef_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_cos_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_cosh_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_count_nonzero_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_cov_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_cross_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_cummax_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_cummin_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_cumprod_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_cumsum_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_cumulative_trapezoid_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_deg2rad_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_diag_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_diag_embed_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_diagflat_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_diagonal_copy_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_diagonal_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_diagonal_scatter_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_diff_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_digamma_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_dist_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_div_floor_rounding_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_div_no_rounding_mode_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_div_trunc_rounding_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_dot_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_double_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_dsplit_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_dstack_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_einsum_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_empty_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_empty_like_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_empty_permuted_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_empty_strided_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_eq_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_equal_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_erf_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_erfc_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_erfinv_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_exp2_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_exp_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_expand_as_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_expand_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_expm1_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_exponential_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_eye_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_fft2_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_fft_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_fftn_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_fftshift_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_hfft2_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_hfft_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_hfftn_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_ifft2_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_ifft_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_ifftn_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_ifftshift_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_ihfft2_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_ihfft_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_ihfftn_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_irfft2_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_irfft_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_irfftn_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_rfft2_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_rfft_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fft_rfftn_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fill_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_flatten_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_flip_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fliplr_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_flipud_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_float_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_float_power_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_floor_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_floor_divide_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fmax_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fmin_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_fmod_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_frac_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_frexp_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_full_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_full_like_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_gather_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_ge_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_geometric_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_geqrf_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_gradient_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_grid_sampler_2d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_gt_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_half_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_heaviside_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_histc_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_histogram_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_histogramdd_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_hsplit_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_hstack_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_hypot_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_i0_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_igamma_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_igammac_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_index_add_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_index_copy_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_index_fill_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_index_put_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_index_reduce_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_index_select_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_inner_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_int_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_isclose_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_isfinite_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_isin_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_isinf_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_isnan_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_isneginf_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_isposinf_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_isreal_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_item_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_jiterator_2inputs_2outputs_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_jiterator_4inputs_with_extra_args_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_jiterator_binary_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_jiterator_binary_return_by_ref_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_jiterator_unary_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_kron_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_kthvalue_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_ldexp_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_le_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_lerp_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_lgamma_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_cholesky_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_cholesky_ex_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_cond_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_cross_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_det_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_det_singular_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_diagonal_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_eig_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_eigh_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_eigvals_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_eigvalsh_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_householder_product_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_inv_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_inv_ex_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_ldl_factor_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_ldl_factor_ex_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_ldl_solve_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_lstsq_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_lstsq_grad_oriented_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_lu_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_lu_factor_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_lu_factor_ex_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_lu_solve_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_matrix_norm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_matrix_power_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_matrix_rank_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_matrix_rank_hermitian_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_multi_dot_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_norm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_norm_subgradients_at_zero_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_pinv_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_pinv_hermitian_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_pinv_singular_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_qr_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_slogdet_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_solve_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_solve_ex_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_solve_triangular_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_svd_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_svdvals_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_tensorinv_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_tensorsolve_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_vander_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_vecdot_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linalg_vector_norm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linspace_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_linspace_tensor_overload_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_log10_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_log1p_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_log2_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_log_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_log_normal_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_log_softmax_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_log_softmax_with_dtype_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_logaddexp2_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_logaddexp_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_logcumsumexp_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_logdet_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_logical_and_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_logical_not_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_logical_or_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_logical_xor_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_logit_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_logspace_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_logspace_tensor_overload_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_logsumexp_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_long_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_lt_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_lu_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_lu_solve_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_lu_unpack_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_mH_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_mT_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_amax_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_amin_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_argmax_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_argmin_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_cumprod_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_cumsum_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_fill_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_log_softmax_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_logaddexp_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_logsumexp_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_mean_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_median_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_norm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_normalize_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_prod_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_scatter_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_select_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_softmax_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_softmin_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_std_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_sum_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_masked_var_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_matmul_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_matrix_exp_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_max_binary_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_max_pool2d_with_indices_backward_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_max_reduction_no_dim_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_max_reduction_with_dim_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_maximum_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_mean_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_median_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_meshgrid_list_of_tensors_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_meshgrid_variadic_tensors_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_min_binary_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_min_reduction_no_dim_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_min_reduction_with_dim_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_minimum_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_mm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_mode_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_movedim_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_msort_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_mul_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_multinomial_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_mv_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_mvlgamma_mvlgamma_p_1_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_mvlgamma_mvlgamma_p_3_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_mvlgamma_mvlgamma_p_5_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nan_to_num_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nanmean_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nanmedian_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nanquantile_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nansum_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_narrow_copy_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_narrow_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_native_batch_norm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_native_dropout_backward_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_native_layer_norm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_ne_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_neg_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_new_empty_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_new_empty_strided_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_new_full_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_new_ones_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_new_zeros_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nextafter_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_adaptive_avg_pool1d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_adaptive_avg_pool2d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_adaptive_avg_pool3d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_adaptive_max_pool1d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_adaptive_max_pool2d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_adaptive_max_pool3d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_alpha_dropout_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_avg_pool1d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_avg_pool2d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_avg_pool3d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_batch_norm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_bilinear_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_binary_cross_entropy_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_binary_cross_entropy_with_logits_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_celu_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_conv1d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_conv2d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_conv3d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_conv_transpose1d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_conv_transpose2d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_conv_transpose3d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_cosine_embedding_loss_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_cosine_similarity_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_cross_entropy_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_ctc_loss_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_dropout2d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_dropout3d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_dropout_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_elu_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_embedding_bag_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_embedding_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_feature_alpha_dropout_with_train_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_feature_alpha_dropout_without_train_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_fractional_max_pool2d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_fractional_max_pool3d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_gaussian_nll_loss_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_gelu_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_glu_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_grid_sample_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_group_norm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_hardshrink_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_hardsigmoid_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_hardswish_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_hardtanh_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_hinge_embedding_loss_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_huber_loss_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_instance_norm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_interpolate_area_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_interpolate_bicubic_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_interpolate_bilinear_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_interpolate_linear_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_interpolate_nearest-exact_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_interpolate_nearest_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_interpolate_trilinear_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_kl_div_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_l1_loss_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_layer_norm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_leaky_relu_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_linear_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_local_response_norm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_logsigmoid_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_margin_ranking_loss_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_max_pool1d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_max_pool2d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_max_pool3d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_max_unpool1d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_max_unpool1d_grad_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_max_unpool2d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_max_unpool2d_grad_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_max_unpool3d_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_max_unpool3d_grad_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_mish_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_mse_loss_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_multi_head_attention_forward_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_multi_margin_loss_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_multilabel_margin_loss_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_multilabel_soft_margin_loss_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_nll_loss_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_normalize_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_pad_circular_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_pad_constant_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_pad_reflect_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_pad_replicate_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_pad_replicate_negative_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_pairwise_distance_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_pdist_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_pixel_shuffle_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_pixel_unshuffle_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_poisson_nll_loss_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_prelu_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_relu6_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_relu_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_rms_norm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_rrelu_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_scaled_dot_product_attention_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_selu_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_silu_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_smooth_l1_loss_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_soft_margin_loss_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_softmin_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_softmin_with_dtype_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_softplus_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_softshrink_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_softsign_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_tanhshrink_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_threshold_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_triplet_margin_loss_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_triplet_margin_with_distance_loss_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_unfold_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_upsample_bilinear_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nn_functional_upsample_nearest_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nonzero_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_nonzero_static_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_norm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_norm_fro_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_norm_inf_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_norm_nuc_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_normal_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_normal_in_place_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_normal_number_mean_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_ones_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_ones_like_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_ormqr_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_outer_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_pca_lowrank_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_permute_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_pinverse_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_polar_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_polygamma_polygamma_n_0_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_polygamma_polygamma_n_1_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_polygamma_polygamma_n_2_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_polygamma_polygamma_n_3_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_polygamma_polygamma_n_4_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_positive_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_pow_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_prod_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_put_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_qr_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_quantile_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_rad2deg_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_rand_like_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_randint_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_randint_like_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_randn_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_randn_like_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_ravel_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_real_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_reciprocal_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_remainder_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_renorm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_repeat_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_repeat_interleave_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_reshape_as_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_reshape_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_resize__cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_resize_as__cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_resolve_conj_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_resolve_neg_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_roll_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_rot90_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_round_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_round_decimals_0_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_round_decimals_3_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_round_decimals_neg_3_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_rsqrt_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_rsub_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_scalar_tensor_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_scatter_add_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_scatter_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_scatter_reduce_amax_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_scatter_reduce_amin_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_scatter_reduce_mean_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_scatter_reduce_prod_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_scatter_reduce_sum_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_searchsorted_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_select_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_select_scatter_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_sgn_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_short_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_sigmoid_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_sign_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_signal_windows_bartlett_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_signal_windows_blackman_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_signal_windows_cosine_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_signal_windows_exponential_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_signal_windows_gaussian_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_signal_windows_general_cosine_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_signal_windows_general_hamming_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_signal_windows_hamming_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_signal_windows_hann_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_signal_windows_kaiser_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_signal_windows_nuttall_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_signbit_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_sin_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_sinc_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_sinh_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_slice_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_slice_scatter_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_softmax_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_softmax_with_dtype_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_sort_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_sparse_mm_reduce_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_sparse_sampled_addmm_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_airy_ai_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_bessel_j0_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_bessel_j1_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_bessel_y0_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_bessel_y1_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_chebyshev_polynomial_t_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_chebyshev_polynomial_u_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_chebyshev_polynomial_v_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_chebyshev_polynomial_w_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_entr_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_erfcx_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_hermite_polynomial_h_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_hermite_polynomial_he_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_i0e_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_i1_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_i1e_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_laguerre_polynomial_l_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_legendre_polynomial_p_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_log_ndtr_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_modified_bessel_i0_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_modified_bessel_i1_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_modified_bessel_k0_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_modified_bessel_k1_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_ndtr_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_ndtri_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_polygamma_special_polygamma_n_0_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_scaled_modified_bessel_k0_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_scaled_modified_bessel_k1_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_shifted_chebyshev_polynomial_t_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_shifted_chebyshev_polynomial_u_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_shifted_chebyshev_polynomial_v_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_shifted_chebyshev_polynomial_w_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_spherical_bessel_j0_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_xlog1py_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_special_zeta_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_split_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_split_list_args_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_split_with_sizes_copy_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_split_with_sizes_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_sqrt_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_square_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_squeeze_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_squeeze_multiple_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_stack_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_std_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_std_mean_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_std_mean_unbiased_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_std_unbiased_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_stft_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_sub_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_sum_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_sum_to_size_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_svd_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_svd_lowrank_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_t_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_take_along_dim_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_take_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_tan_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_tanh_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_tensor_split_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_tensordot_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_tile_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_to_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_to_sparse_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_topk_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_trace_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_transpose_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_trapezoid_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_trapz_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_triangular_solve_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_tril_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_triu_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_true_divide_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_trunc_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_unbind_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_unflatten_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_unfold_copy_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_unfold_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_uniform_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_unique_consecutive_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_unique_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_unsafe_chunk_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_unsafe_split_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_unsqueeze_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_var_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_var_mean_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_var_mean_unbiased_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_var_unbiased_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_vdot_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_view_as_complex_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_view_as_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_view_copy_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_view_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_vsplit_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_vstack_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_where_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_xlogy_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_zero__cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_zeros_cpu_float32, test/distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_zeros_like_cpu_float32 2024-04-03T16:10:08.0852636Z 2024-04-03T16:10:08.0853156Z Running distributed/test_distributed_spawn 1/3 ... [2024-04-03 16:10:08.067405] 2024-04-03T16:10:08.0853929Z MPI not available -- MPI backend tests will be skipped 2024-04-03T16:10:08.0854573Z Running distributed tests for the test backend with env init_method 2024-04-03T16:10:08.0856560Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:10:08.073614] 2024-04-03T16:10:10.5592738Z 2024-04-03T16:10:10.5594650Z distributed/test_distributed_spawn 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.3_78d84d0ca1565f0d_.log 2024-04-03T16:10:10.5596367Z Running 0 items in this shard: 2024-04-03T16:10:10.5596767Z 2024-04-03T16:10:10.5602557Z Running distributed tests for the test backend with file init_method 2024-04-03T16:10:10.5608035Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:10:10.560501] 2024-04-03T16:10:13.0273280Z 2024-04-03T16:10:13.0275166Z distributed/test_distributed_spawn 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.3_1760b607f4684e68_.log 2024-04-03T16:10:13.0276361Z Running 0 items in this shard: 2024-04-03T16:10:13.0276628Z 2024-04-03T16:10:13.0281445Z Running distributed tests for the gloo backend with env init_method 2024-04-03T16:10:13.0287360Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:10:13.028432] 2024-04-03T16:18:21.7349519Z 2024-04-03T16:18:21.7351342Z distributed/test_distributed_spawn 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.3_29ba5b46ee91226d_.log 2024-04-03T16:18:21.7418511Z Running 100 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_default_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-04-03T16:18:21.7479465Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2024-04-03T16:18:21.7481270Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2024-04-03T16:18:21.7482692Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2024-04-03T16:18:21.7484081Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2024-04-03T16:18:21.7485633Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2024-04-03T16:18:21.7487299Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2024-04-03T16:18:21.7489077Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value 2024-04-03T16:18:21.7490935Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2024-04-03T16:18:21.7492532Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2024-04-03T16:18:21.7494026Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync 2024-04-03T16:18:21.7495689Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2024-04-03T16:18:21.7497364Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2024-04-03T16:18:21.7498781Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2024-04-03T16:18:21.7500170Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2024-04-03T16:18:21.7501795Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg 2024-04-03T16:18:21.7503212Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup 2024-04-03T16:18:21.7504671Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2024-04-03T16:18:21.7506173Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2024-04-03T16:18:21.7507697Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2024-04-03T16:18:21.7509208Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2024-04-03T16:18:21.7510838Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2024-04-03T16:18:21.7512471Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2024-04-03T16:18:21.7513904Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2024-04-03T16:18:21.7515388Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2024-04-03T16:18:21.7516882Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min 2024-04-03T16:18:21.7518286Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max 2024-04-03T16:18:21.7519683Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min 2024-04-03T16:18:21.7521127Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2024-04-03T16:18:21.7522451Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2024-04-03T16:18:21.7523757Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2024-04-03T16:18:21.7525072Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum 2024-04-03T16:18:21.7526379Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async 2024-04-03T16:18:21.7527757Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2024-04-03T16:18:21.7529140Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group 2024-04-03T16:18:21.7530519Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2024-04-03T16:18:21.7531976Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2024-04-03T16:18:21.7533351Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split 2024-04-03T16:18:21.7534922Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2024-04-03T16:18:21.7536477Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group 2024-04-03T16:18:21.7538043Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda 2024-04-03T16:18:21.7539624Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group 2024-04-03T16:18:21.7541196Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda 2024-04-03T16:18:21.7542629Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group 2024-04-03T16:18:21.7543901Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier 2024-04-03T16:18:21.7545142Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2024-04-03T16:18:21.7546474Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2024-04-03T16:18:21.7547844Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group 2024-04-03T16:18:21.7549245Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2024-04-03T16:18:21.7551277Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err 2024-04-03T16:18:21.7552764Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2024-04-03T16:18:21.7554093Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2024-04-03T16:18:21.7555391Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2024-04-03T16:18:21.7556742Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2024-04-03T16:18:21.7558108Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2024-04-03T16:18:21.7559668Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2024-04-03T16:18:21.7561416Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping 2024-04-03T16:18:21.7563024Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2024-04-03T16:18:21.7564563Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2024-04-03T16:18:21.7565901Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2024-04-03T16:18:21.7567257Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook 2024-04-03T16:18:21.7568791Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2024-04-03T16:18:21.7570188Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg 2024-04-03T16:18:21.7571612Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2024-04-03T16:18:21.7573075Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params 2024-04-03T16:18:21.7574542Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged 2024-04-03T16:18:21.7575970Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params 2024-04-03T16:18:21.7577409Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states 2024-04-03T16:18:21.7578822Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2024-04-03T16:18:21.7580202Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2024-04-03T16:18:21.7581628Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn 2024-04-03T16:18:21.7583182Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_default_group 2024-04-03T16:18:21.7584604Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group 2024-04-03T16:18:21.7585901Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group 2024-04-03T16:18:21.7587232Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params 2024-04-03T16:18:21.7588548Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future 2024-04-03T16:18:21.7589938Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2024-04-03T16:18:21.7591232Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2024-04-03T16:18:21.7592491Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv 2024-04-03T16:18:21.7593786Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler 2024-04-03T16:18:21.7595220Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order 2024-04-03T16:18:21.7596641Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo 2024-04-03T16:18:21.7598055Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup 2024-04-03T16:18:21.7599498Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2024-04-03T16:18:21.7600913Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort 2024-04-03T16:18:21.7602186Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups 2024-04-03T16:18:21.7603748Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size 2024-04-03T16:18:21.7605334Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2024-04-03T16:18:21.7606856Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product 2024-04-03T16:18:21.7608207Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2024-04-03T16:18:21.7609504Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2024-04-03T16:18:21.7610764Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2024-04-03T16:18:21.7612031Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2024-04-03T16:18:21.7613302Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda 2024-04-03T16:18:21.7614578Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2024-04-03T16:18:21.7615962Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2024-04-03T16:18:21.7617422Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2024-04-03T16:18:21.7618825Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2024-04-03T16:18:21.7620288Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2024-04-03T16:18:21.7621724Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2024-04-03T16:18:21.7623165Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-04-03T16:18:21.7624005Z 2024-04-03T16:18:21.7624288Z Running distributed tests for the gloo backend with file init_method 2024-04-03T16:18:21.7626353Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:18:21.740329] 2024-04-03T16:26:27.6804810Z 2024-04-03T16:26:27.6807268Z distributed/test_distributed_spawn 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.3_b5f415309e51b4bd_.log 2024-04-03T16:26:27.6907276Z Running 100 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_default_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-04-03T16:26:27.6980625Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2024-04-03T16:26:27.6982340Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2024-04-03T16:26:27.6983771Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2024-04-03T16:26:27.6985173Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2024-04-03T16:26:27.6986736Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2024-04-03T16:26:27.6988404Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2024-04-03T16:26:27.6990352Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value 2024-04-03T16:26:27.6992118Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2024-04-03T16:26:27.6993732Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2024-04-03T16:26:27.6995221Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync 2024-04-03T16:26:27.6996754Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2024-04-03T16:26:27.6998336Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2024-04-03T16:26:27.6999778Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2024-04-03T16:26:27.7001296Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2024-04-03T16:26:27.7002773Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg 2024-04-03T16:26:27.7004203Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup 2024-04-03T16:26:27.7005883Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2024-04-03T16:26:27.7007583Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2024-04-03T16:26:27.7009219Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2024-04-03T16:26:27.7010746Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2024-04-03T16:26:27.7012242Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2024-04-03T16:26:27.7013700Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2024-04-03T16:26:27.7015139Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2024-04-03T16:26:27.7016710Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2024-04-03T16:26:27.7018219Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min 2024-04-03T16:26:27.7019618Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max 2024-04-03T16:26:27.7021034Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min 2024-04-03T16:26:27.7022420Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2024-04-03T16:26:27.7023743Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2024-04-03T16:26:27.7025063Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2024-04-03T16:26:27.7026405Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum 2024-04-03T16:26:27.7027737Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async 2024-04-03T16:26:27.7029115Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2024-04-03T16:26:27.7030610Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group 2024-04-03T16:26:27.7032005Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2024-04-03T16:26:27.7033372Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2024-04-03T16:26:27.7034735Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split 2024-04-03T16:26:27.7036259Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2024-04-03T16:26:27.7037822Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group 2024-04-03T16:26:27.7039413Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda 2024-04-03T16:26:27.7041077Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group 2024-04-03T16:26:27.7042750Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda 2024-04-03T16:26:27.7044197Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group 2024-04-03T16:26:27.7045547Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier 2024-04-03T16:26:27.7046780Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2024-04-03T16:26:27.7048119Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2024-04-03T16:26:27.7049508Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group 2024-04-03T16:26:27.7050903Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2024-04-03T16:26:27.7052373Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err 2024-04-03T16:26:27.7053856Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2024-04-03T16:26:27.7055209Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2024-04-03T16:26:27.7056499Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2024-04-03T16:26:27.7057854Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2024-04-03T16:26:27.7059225Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2024-04-03T16:26:27.7060809Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2024-04-03T16:26:27.7062445Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping 2024-04-03T16:26:27.7064060Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2024-04-03T16:26:27.7065598Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2024-04-03T16:26:27.7066963Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2024-04-03T16:26:27.7068310Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook 2024-04-03T16:26:27.7069908Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2024-04-03T16:26:27.7071312Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg 2024-04-03T16:26:27.7072688Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2024-04-03T16:26:27.7074151Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params 2024-04-03T16:26:27.7075639Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged 2024-04-03T16:26:27.7077089Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params 2024-04-03T16:26:27.7078618Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states 2024-04-03T16:26:27.7080025Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2024-04-03T16:26:27.7081577Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2024-04-03T16:26:27.7083027Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn 2024-04-03T16:26:27.7084584Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_default_group 2024-04-03T16:26:27.7085999Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group 2024-04-03T16:26:27.7087308Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group 2024-04-03T16:26:27.7088653Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params 2024-04-03T16:26:27.7089955Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future 2024-04-03T16:26:27.7091198Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2024-04-03T16:26:27.7092505Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2024-04-03T16:26:27.7093786Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv 2024-04-03T16:26:27.7095072Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler 2024-04-03T16:26:27.7096514Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order 2024-04-03T16:26:27.7097945Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo 2024-04-03T16:26:27.7099383Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup 2024-04-03T16:26:27.7100819Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2024-04-03T16:26:27.7102180Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort 2024-04-03T16:26:27.7103475Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups 2024-04-03T16:26:27.7104963Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size 2024-04-03T16:26:27.7106545Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2024-04-03T16:26:27.7108003Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product 2024-04-03T16:26:27.7109371Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2024-04-03T16:26:27.7110852Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2024-04-03T16:26:27.7112117Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2024-04-03T16:26:27.7113395Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2024-04-03T16:26:27.7114768Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda 2024-04-03T16:26:27.7116045Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2024-04-03T16:26:27.7117516Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2024-04-03T16:26:27.7118986Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2024-04-03T16:26:27.7120483Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2024-04-03T16:26:27.7121936Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2024-04-03T16:26:27.7123367Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2024-04-03T16:26:27.7124830Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-04-03T16:26:27.7125677Z 2024-04-03T16:26:27.7126178Z Running distributed/test_distributed_spawn 3/3 ... [2024-04-03 16:26:27.686373] 2024-04-03T16:26:27.7126922Z MPI not available -- MPI backend tests will be skipped 2024-04-03T16:26:27.7127572Z Running distributed tests for the test backend with env init_method 2024-04-03T16:26:27.7129532Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:26:27.693607] 2024-04-03T16:26:30.1501669Z 2024-04-03T16:26:30.1503551Z distributed/test_distributed_spawn 3/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.3_3a6a35090cdb944e_.log 2024-04-03T16:26:30.1504780Z Running 0 items in this shard: 2024-04-03T16:26:30.1505051Z 2024-04-03T16:26:30.1508845Z Running distributed tests for the test backend with file init_method 2024-04-03T16:26:30.1514859Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:26:30.151190] 2024-04-03T16:26:32.6459298Z 2024-04-03T16:26:32.6461035Z distributed/test_distributed_spawn 3/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.3_8d0b48a9ae1d9492_.log 2024-04-03T16:26:32.6462321Z Running 0 items in this shard: 2024-04-03T16:26:32.6462634Z 2024-04-03T16:26:32.6467046Z Running distributed tests for the gloo backend with env init_method 2024-04-03T16:26:32.6473111Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:26:32.647010] 2024-04-03T16:34:14.9668767Z 2024-04-03T16:34:14.9671984Z distributed/test_distributed_spawn 3/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.3_22204db9bce4e8a4_.log 2024-04-03T16:34:14.9731435Z Running 90 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_new_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-04-03T16:34:14.9786558Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2024-04-03T16:34:14.9788328Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2024-04-03T16:34:14.9790215Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2024-04-03T16:34:14.9791856Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2024-04-03T16:34:14.9793313Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2024-04-03T16:34:14.9794663Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2024-04-03T16:34:14.9796105Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2024-04-03T16:34:14.9797490Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2024-04-03T16:34:14.9798833Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2024-04-03T16:34:14.9800285Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2024-04-03T16:34:14.9801796Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2024-04-03T16:34:14.9803267Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2024-04-03T16:34:14.9804703Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2024-04-03T16:34:14.9806124Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2024-04-03T16:34:14.9807549Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2024-04-03T16:34:14.9808908Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2024-04-03T16:34:14.9810230Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2024-04-03T16:34:14.9811591Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2024-04-03T16:34:14.9813061Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2024-04-03T16:34:14.9814683Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2024-04-03T16:34:14.9816120Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2024-04-03T16:34:14.9817739Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2024-04-03T16:34:14.9819263Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2024-04-03T16:34:14.9820694Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2024-04-03T16:34:14.9822023Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2024-04-03T16:34:14.9823359Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2024-04-03T16:34:14.9824724Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2024-04-03T16:34:14.9826131Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2024-04-03T16:34:14.9827583Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2024-04-03T16:34:14.9829085Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2024-04-03T16:34:14.9830715Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2024-04-03T16:34:14.9832115Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2024-04-03T16:34:14.9833688Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2024-04-03T16:34:14.9835427Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2024-04-03T16:34:14.9836986Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2024-04-03T16:34:14.9838413Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2024-04-03T16:34:14.9839879Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2024-04-03T16:34:14.9841471Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2024-04-03T16:34:14.9842833Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2024-04-03T16:34:14.9844237Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2024-04-03T16:34:14.9845743Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2024-04-03T16:34:14.9847150Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2024-04-03T16:34:14.9848584Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2024-04-03T16:34:14.9850098Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2024-04-03T16:34:14.9851716Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2024-04-03T16:34:14.9853133Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2024-04-03T16:34:14.9854729Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2024-04-03T16:34:14.9856533Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2024-04-03T16:34:14.9858120Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2024-04-03T16:34:14.9859558Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2024-04-03T16:34:14.9860998Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2024-04-03T16:34:14.9862436Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2024-04-03T16:34:14.9863901Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2024-04-03T16:34:14.9865363Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_new_group 2024-04-03T16:34:14.9866835Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2024-04-03T16:34:14.9868156Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2024-04-03T16:34:14.9869401Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2024-04-03T16:34:14.9870845Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2024-04-03T16:34:14.9872167Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2024-04-03T16:34:14.9873586Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2024-04-03T16:34:14.9875066Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2024-04-03T16:34:14.9876504Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2024-04-03T16:34:14.9877928Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2024-04-03T16:34:14.9879340Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2024-04-03T16:34:14.9880952Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2024-04-03T16:34:14.9882559Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2024-04-03T16:34:14.9884111Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2024-04-03T16:34:14.9885603Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2024-04-03T16:34:14.9887244Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2024-04-03T16:34:14.9888929Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2024-04-03T16:34:14.9890591Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2024-04-03T16:34:14.9891966Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2024-04-03T16:34:14.9893307Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2024-04-03T16:34:14.9894568Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2024-04-03T16:34:14.9895901Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2024-04-03T16:34:14.9897289Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2024-04-03T16:34:14.9898580Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2024-04-03T16:34:14.9899818Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2024-04-03T16:34:14.9901110Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2024-04-03T16:34:14.9902442Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2024-04-03T16:34:14.9903786Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2024-04-03T16:34:14.9905134Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2024-04-03T16:34:14.9906427Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-04-03T16:34:14.9907825Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2024-04-03T16:34:14.9909249Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2024-04-03T16:34:14.9910832Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-04-03T16:34:14.9912298Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-04-03T16:34:14.9913725Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-04-03T16:34:14.9915203Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-04-03T16:34:14.9916744Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-04-03T16:34:14.9917597Z 2024-04-03T16:34:14.9917881Z Running distributed tests for the gloo backend with file init_method 2024-04-03T16:34:14.9919955Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:34:14.970701] 2024-04-03T16:41:57.5434226Z 2024-04-03T16:41:57.5437062Z distributed/test_distributed_spawn 3/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.3_c5d5d8d4291dfc47_.log 2024-04-03T16:41:57.5496274Z Running 90 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_new_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-04-03T16:41:57.5549746Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2024-04-03T16:41:57.5551480Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2024-04-03T16:41:57.5553204Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2024-04-03T16:41:57.5554853Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2024-04-03T16:41:57.5556298Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2024-04-03T16:41:57.5557648Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2024-04-03T16:41:57.5559078Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2024-04-03T16:41:57.5560522Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2024-04-03T16:41:57.5561872Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2024-04-03T16:41:57.5563281Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2024-04-03T16:41:57.5564774Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2024-04-03T16:41:57.5566253Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2024-04-03T16:41:57.5567685Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2024-04-03T16:41:57.5569109Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2024-04-03T16:41:57.5570531Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2024-04-03T16:41:57.5571890Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2024-04-03T16:41:57.5573357Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2024-04-03T16:41:57.5574717Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2024-04-03T16:41:57.5576209Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2024-04-03T16:41:57.5577574Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2024-04-03T16:41:57.5579007Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2024-04-03T16:41:57.5580528Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2024-04-03T16:41:57.5582070Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2024-04-03T16:41:57.5583477Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2024-04-03T16:41:57.5584818Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2024-04-03T16:41:57.5586143Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2024-04-03T16:41:57.5587513Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2024-04-03T16:41:57.5588898Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2024-04-03T16:41:57.5590911Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2024-04-03T16:41:57.5592425Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2024-04-03T16:41:57.5593901Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2024-04-03T16:41:57.5595305Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2024-04-03T16:41:57.5596868Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2024-04-03T16:41:57.5598593Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2024-04-03T16:41:57.5600164Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2024-04-03T16:41:57.5601704Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2024-04-03T16:41:57.5603183Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2024-04-03T16:41:57.5604708Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2024-04-03T16:41:57.5606066Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2024-04-03T16:41:57.5607463Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2024-04-03T16:41:57.5609064Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2024-04-03T16:41:57.5610467Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2024-04-03T16:41:57.5611982Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2024-04-03T16:41:57.5613502Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2024-04-03T16:41:57.5615023Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2024-04-03T16:41:57.5616436Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2024-04-03T16:41:57.5617959Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2024-04-03T16:41:57.5619758Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2024-04-03T16:41:57.5621344Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2024-04-03T16:41:57.5622775Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2024-04-03T16:41:57.5624208Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2024-04-03T16:41:57.5625635Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2024-04-03T16:41:57.5627096Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2024-04-03T16:41:57.5628572Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_new_group 2024-04-03T16:41:57.5630186Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2024-04-03T16:41:57.5631511Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2024-04-03T16:41:57.5632760Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2024-04-03T16:41:57.5634046Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2024-04-03T16:41:57.5635343Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2024-04-03T16:41:57.5636772Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2024-04-03T16:41:57.5638275Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2024-04-03T16:41:57.5639817Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2024-04-03T16:41:57.5641289Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2024-04-03T16:41:57.5642700Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2024-04-03T16:41:57.5644231Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2024-04-03T16:41:57.5645942Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2024-04-03T16:41:57.5647486Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2024-04-03T16:41:57.5649053Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2024-04-03T16:41:57.5650585Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2024-04-03T16:41:57.5652285Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2024-04-03T16:41:57.5653864Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2024-04-03T16:41:57.5655222Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2024-04-03T16:41:57.5656557Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2024-04-03T16:41:57.5657843Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2024-04-03T16:41:57.5659163Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2024-04-03T16:41:57.5660551Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2024-04-03T16:41:57.5661841Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2024-04-03T16:41:57.5663094Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2024-04-03T16:41:57.5664383Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2024-04-03T16:41:57.5665704Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2024-04-03T16:41:57.5667043Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2024-04-03T16:41:57.5668364Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2024-04-03T16:41:57.5669797Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-04-03T16:41:57.5671202Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2024-04-03T16:41:57.5672653Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2024-04-03T16:41:57.5674048Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-04-03T16:41:57.5675502Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-04-03T16:41:57.5676918Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-04-03T16:41:57.5678392Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-04-03T16:41:57.5679914Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-04-03T16:41:57.5680911Z 2024-04-03T16:41:57.5681517Z Running distributed/_composable/fsdp/test_fully_shard_overlap 1/1 ... [2024-04-03 16:41:57.547258] 2024-04-03T16:41:57.5683791Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_overlap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:41:57.547660] 2024-04-03T16:42:00.1701294Z 2024-04-03T16:42:00.1703383Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_overlap_1.1_45f7177e77197ac1_.log 2024-04-03T16:42:00.1705629Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_training_overlap 2024-04-03T16:42:00.1706675Z 2024-04-03T16:42:00.1708154Z Running distributed/_tensor/test_redistribute 1/1 ... [2024-04-03 16:42:00.170593] 2024-04-03T16:42:00.1713250Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_redistribute.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:42:00.171019] 2024-04-03T16:42:31.2015731Z 2024-04-03T16:42:31.2017340Z distributed/_tensor/test_redistribute 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_redistribute_1.1_9c615097b35bdbf4_.log 2024-04-03T16:42:31.2029864Z Running 10 items in this shard: test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_partial_to_replicate_forward_backward, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_partial_to_shard, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_redistribute_negative_shard_dim, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_redistribute_uneven_sharding, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_local_partial_grad, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_partial, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_replicate_forward_backward, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_shard_forward_backward, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_shard_to_replicate_forward_backward, test/distributed/_tensor/test_redistribute.py::MultiDimRedistributeTest::test_multi_dim_mesh 2024-04-03T16:42:31.2038594Z 2024-04-03T16:42:31.2039281Z Running distributed/test_collective_utils 1/1 ... [2024-04-03 16:42:31.202319] 2024-04-03T16:42:31.2042168Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_collective_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:42:31.202865] 2024-04-03T16:42:33.3589109Z 2024-04-03T16:42:33.3590918Z distributed/test_collective_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_collective_utils_1.1_c5fe5238939fce8e_.log 2024-04-03T16:42:33.3592108Z 2024-04-03T16:42:33.3595439Z Running distributed/tensor/parallel/test_tp_random_state 1/1 ... [2024-04-03 16:42:33.359322] 2024-04-03T16:42:33.3600366Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_random_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:42:33.359726] 2024-04-03T16:42:38.5377753Z 2024-04-03T16:42:38.5379541Z distributed/tensor/parallel/test_tp_random_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_random_state_1.1_9f9a1434487b9b97_.log 2024-04-03T16:42:38.5381805Z Running 1 items in this shard: test/distributed/tensor/parallel/test_tp_random_state.py::TensorParallelRandomStateTests::test_model_init 2024-04-03T16:42:38.5382651Z 2024-04-03T16:42:38.5384981Z Running distributed/_tensor/test_math_ops 1/1 ... [2024-04-03 16:42:38.538264] 2024-04-03T16:42:38.5389840Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_math_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:42:38.538648] 2024-04-03T16:43:03.1074550Z 2024-04-03T16:43:03.1076342Z distributed/_tensor/test_math_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_math_ops_1.1_d8a43948892d2350_.log 2024-04-03T16:43:03.1081457Z Running 8 items in this shard: test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_bwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_fwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_linear_op_reductions, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_mean, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_nll_loss_and_cross_entropy, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_shard_math_ops, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_softmax_fwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_softmax_with_bwd 2024-04-03T16:43:03.1085268Z 2024-04-03T16:43:03.1085750Z Running distributed/checkpoint/test_hsdp_checkpoint 1/1 ... [2024-04-03 16:43:03.108053] 2024-04-03T16:43:03.1087875Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_hsdp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:43:03.108455] 2024-04-03T16:43:15.8019125Z 2024-04-03T16:43:15.8021215Z distributed/checkpoint/test_hsdp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_hsdp_checkpoint_1.1_fc96828091851eff_.log 2024-04-03T16:43:15.8025866Z Running 4 items in this shard: test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_checkpoint_is_even_sharded_model_False, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_checkpoint_is_even_sharded_model_True, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_fsdp_checkpoint_conversion_is_even_sharded_model_False, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_fsdp_checkpoint_conversion_is_even_sharded_model_True 2024-04-03T16:43:15.8029057Z 2024-04-03T16:43:15.8029531Z Running distributed/_spmd/test_tracing 1/1 ... [2024-04-03 16:43:15.802345] 2024-04-03T16:43:15.8031617Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_spmd/test_tracing.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:43:15.802752] 2024-04-03T16:43:17.9724487Z 2024-04-03T16:43:17.9726914Z distributed/_spmd/test_tracing 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._spmd.test_tracing_1.1_6fd9dbacc2d7d792_.log 2024-04-03T16:43:17.9728835Z 2024-04-03T16:43:17.9732632Z Running distributed/checkpoint/e2e/test_fine_tuning 1/1 ... [2024-04-03 16:43:17.973007] 2024-04-03T16:43:17.9739432Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/e2e/test_fine_tuning.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:43:17.973498] 2024-04-03T16:43:20.5977852Z 2024-04-03T16:43:20.5980332Z distributed/checkpoint/e2e/test_fine_tuning 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_fine_tuning_1.1_c93330a8a6180a83_.log 2024-04-03T16:43:20.5982103Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_fine_tuning.py::TestFineTuning::test_fine_tuning 2024-04-03T16:43:20.5982843Z 2024-04-03T16:43:20.5985615Z Running distributed/_tensor/test_matrix_ops 1/1 ... [2024-04-03 16:43:20.598308] 2024-04-03T16:43:20.5990734Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_matrix_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:43:20.598748] 2024-04-03T16:43:48.3763506Z 2024-04-03T16:43:48.3765115Z distributed/_tensor/test_matrix_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_matrix_ops_1.1_14155b64e6a156a1_.log 2024-04-03T16:43:48.3771476Z Running 10 items in this shard: test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm_auto_redistribute, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm_empty_operand, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_baddbmm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_bmm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_mm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_scaled_dot_product_attention, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_t, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_t_partial_use_native_funcol_False, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_t_partial_use_native_funcol_True 2024-04-03T16:43:48.3776274Z 2024-04-03T16:43:48.3776866Z Running distributed/_composable/fully_shard/test_fully_shard_compile 1/1 ... [2024-04-03 16:43:48.376922] 2024-04-03T16:43:48.3779099Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:43:48.377328] 2024-04-03T16:43:50.6994146Z 2024-04-03T16:43:50.6996647Z distributed/_composable/fully_shard/test_fully_shard_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_compile_1.1_df569d378076b0d3_.log 2024-04-03T16:43:50.6999551Z Running 1 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_compile.py::TestCompile::test_compile 2024-04-03T16:43:50.7000444Z 2024-04-03T16:43:50.7003834Z Running distributed/checkpoint/test_fsdp_model_state 1/1 ... [2024-04-03 16:43:50.700090] 2024-04-03T16:43:50.7009718Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_model_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:43:50.700611] 2024-04-03T16:43:58.3347458Z 2024-04-03T16:43:58.3349481Z distributed/checkpoint/test_fsdp_model_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_model_state_1.1_cd09346898256b10_.log 2024-04-03T16:43:58.3352187Z Running 2 items in this shard: test/distributed/checkpoint/test_fsdp_model_state.py::FsdpModelStateCheckpoint::test_fsdp_model_state_no_resharding, test/distributed/checkpoint/test_fsdp_model_state.py::FsdpModelStateCheckpoint::test_fsdp_model_state_with_resharding 2024-04-03T16:43:58.3354019Z 2024-04-03T16:43:58.3354568Z Running distributed/checkpoint/test_state_dict_utils 1/1 ... [2024-04-03 16:43:58.335202] 2024-04-03T16:43:58.3359248Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_state_dict_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:43:58.335617] 2024-04-03T16:44:01.0082904Z 2024-04-03T16:44:01.0085220Z distributed/checkpoint/test_state_dict_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_state_dict_utils_1.1_3ebecad4494b433f_.log 2024-04-03T16:44:01.0090036Z Running 5 items in this shard: test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_complicated_dict, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_cpu_and_ranks_only, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_create_cpu_state_dict, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_gather_state_dict_dtensor, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_gather_with_cpu_and_ranks_only 2024-04-03T16:44:01.0093705Z 2024-04-03T16:44:01.0094246Z Running distributed/tensor/parallel/test_fsdp_2d_parallel 1/1 ... [2024-04-03 16:44:01.008641] 2024-04-03T16:44:01.0096377Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/tensor/parallel/test_fsdp_2d_parallel.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:44:01.009039] 2024-04-03T16:44:26.6786468Z 2024-04-03T16:44:26.6788538Z distributed/tensor/parallel/test_fsdp_2d_parallel 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_fsdp_2d_parallel_1.1_2d41b63a1e23b4c6_.log 2024-04-03T16:44:26.6797183Z Running 9 items in this shard: test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_2d_e2e_training_default, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_2d_e2e_training_not_use_orig_params, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_2d_e2e_training_use_orig_params, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_2d_fsdp_state_enable_extension, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_raise_invalid_tp_composition, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_2d_load_state_dict_is_even_sharded_model_True, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_2d_optim_state_dict_is_even_sharded_model_True, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_2d_state_dict_is_even_sharded_model_True, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_fsdp_2d_extension 2024-04-03T16:44:26.6803887Z 2024-04-03T16:44:26.6804342Z Running distributed/_tensor/debug/test_comm_mode 1/1 ... [2024-04-03 16:44:26.679121] 2024-04-03T16:44:26.6806352Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/debug/test_comm_mode.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:44:26.679523] 2024-04-03T16:44:29.2519534Z 2024-04-03T16:44:29.2521416Z distributed/_tensor/debug/test_comm_mode 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_comm_mode_1.1_2718bdb55ceb6146_.log 2024-04-03T16:44:29.2523574Z Running 2 items in this shard: test/distributed/_tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode, test/distributed/_tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode_with_dtensor 2024-04-03T16:44:29.2525163Z 2024-04-03T16:44:29.2525812Z Running distributed/tensor/parallel/test_tp_examples 1/1 ... [2024-04-03 16:44:29.252379] 2024-04-03T16:44:29.2530537Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_examples.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:44:29.252758] 2024-04-03T16:44:50.0648958Z 2024-04-03T16:44:50.0651599Z distributed/tensor/parallel/test_tp_examples 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_examples_1.1_33ae65a825fbf68f_.log 2024-04-03T16:44:50.0658298Z Running 7 items in this shard: test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_loss_parallel, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_inference, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_False_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_True_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_True, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_weight_tying 2024-04-03T16:44:50.0664047Z 2024-04-03T16:44:50.0664430Z Running distributed/test_c10d_pypg 1/1 ... [2024-04-03 16:44:50.065482] 2024-04-03T16:44:50.0666272Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:44:50.065905] 2024-04-03T16:46:25.5219003Z 2024-04-03T16:46:25.5222909Z distributed/test_c10d_pypg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_pypg_1.1_cc5e0d3e5c55e44b_.log 2024-04-03T16:46:25.5260900Z Running 42 items in this shard: test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_dataclass_output, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_dataclass_output_unused_param, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_invoke_work_object, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_with_pypg, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_with_pypg_with_grad_views, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_invalid_powerSGD_state, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_sync_batch_norm_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_dataclass_output, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_dataclass_output_unused_param, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_invoke_work_object, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_with_pypg, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_with_pypg_with_grad_views, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_invalid_powerSGD_state, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_sync_batch_norm_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_sync_batch_norm_only_empty_input 2024-04-03T16:46:25.5287772Z 2024-04-03T16:46:25.5288380Z Running distributed/_composable/fsdp/test_fully_shard_frozen 1/1 ... [2024-04-03 16:46:25.522618] 2024-04-03T16:46:25.5290518Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_frozen.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:46:25.523170] 2024-04-03T16:46:28.1477213Z 2024-04-03T16:46:28.1480145Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_f814de743d600ba2_.log 2024-04-03T16:46:28.1486254Z Running 3 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_multi_forward_mixed_requires_grad, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_across_groups, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_per_group 2024-04-03T16:46:28.1489971Z 2024-04-03T16:46:28.1490847Z Running distributed/fsdp/test_distributed_checkpoint 1/1 ... [2024-04-03 16:46:28.148389] 2024-04-03T16:46:28.1494422Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:46:28.148919] 2024-04-03T16:46:30.7733162Z 2024-04-03T16:46:30.7735482Z distributed/fsdp/test_distributed_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_distributed_checkpoint_1.1_3a3316f23d073049_.log 2024-04-03T16:46:30.7738493Z Running 2 items in this shard: test/distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpoint::test_distributed_checkpoint_state_dict_type0, test/distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpoint::test_distributed_checkpoint_state_dict_type1 2024-04-03T16:46:30.7740483Z 2024-04-03T16:46:30.7740881Z Running distributed/_tensor/test_utils 1/1 ... [2024-04-03 16:46:30.773681] 2024-04-03T16:46:30.7743861Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:46:30.774097] 2024-04-03T16:46:43.3165564Z 2024-04-03T16:46:43.3167426Z distributed/_tensor/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_utils_1.1_d0b4df22fb759b6e_.log 2024-04-03T16:46:43.3169987Z Running 3 items in this shard: test/distributed/_tensor/test_utils.py::UtilTest::test_compute_local_shape_2d_uneven, test/distributed/_tensor/test_utils.py::UtilTest::test_compute_local_shape_and_global_offset_1D, test/distributed/_tensor/test_utils.py::UtilTest::test_compute_local_shape_and_global_offset_2D 2024-04-03T16:46:43.3172015Z 2024-04-03T16:46:43.3172549Z Running distributed/_composable/fsdp/test_fully_shard_comm 1/1 ... [2024-04-03 16:46:43.317012] 2024-04-03T16:46:43.3177524Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_comm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:46:43.317443] 2024-04-03T16:46:46.0402499Z 2024-04-03T16:46:46.0404449Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_comm_1.1_9af651d7a164c21e_.log 2024-04-03T16:46:46.0408796Z Running 4 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_all_gather_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_reduce_scatter_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_fully_shard_communication_count, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardBackwardPrefetch::test_fully_shard_backward_prefetch 2024-04-03T16:46:46.0411640Z 2024-04-03T16:46:46.0412206Z Running distributed/fsdp/test_fsdp_flatten_params 1/1 ... [2024-04-03 16:46:46.040725] 2024-04-03T16:46:46.0414303Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_flatten_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:46:46.041123] 2024-04-03T16:47:13.2997710Z 2024-04-03T16:47:13.2999798Z distributed/fsdp/test_fsdp_flatten_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_flatten_params_1.1_126f9ddf5171f8ad_.log 2024-04-03T16:47:13.3009426Z Running 11 items in this shard: test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_empty_module, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_aligned_full_precision, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_aligned_mixed_precision, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_unaligned, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flatten_nothing, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_numel_with_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_numel_without_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_output_with_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_output_without_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_partial_flattening, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_pnorm_after_step_with_shared_params 2024-04-03T16:47:13.3016139Z 2024-04-03T16:47:13.3016815Z Running distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 1/1 ... [2024-04-03 16:47:13.300312] 2024-04-03T16:47:13.3019137Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_model_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:47:13.300707] 2024-04-03T16:47:25.4876568Z 2024-04-03T16:47:25.4878527Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_model_checkpoint_1.1_f19ae5709be18bf0_.log 2024-04-03T16:47:25.4883263Z Running 4 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_model_checkpoint.py::TestModelCheckpointing::test_full_state_dict_save_load_mixed_sharding, test/distributed/_composable/fully_shard/test_fully_shard_model_checkpoint.py::TestModelCheckpointing::test_state_dict_save_load_flow, test/distributed/_composable/fully_shard/test_fully_shard_model_checkpoint.py::TestModelCheckpointing::test_state_dict_save_load_root_fully_shard, test/distributed/_composable/fully_shard/test_fully_shard_model_checkpoint.py::TestModelCheckpointing::test_state_dict_save_load_submodule_fully_shard 2024-04-03T16:47:25.4886622Z 2024-04-03T16:47:25.4887035Z Running distributed/checkpoint/test_utils 1/1 ... [2024-04-03 16:47:25.488078] 2024-04-03T16:47:25.4888980Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:47:25.488467] 2024-04-03T16:47:28.1108554Z 2024-04-03T16:47:28.1110996Z distributed/checkpoint/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_utils_1.1_8c2160cf0a44936f_.log 2024-04-03T16:47:28.1114895Z Running 5 items in this shard: test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_flat_data, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_index_hint_ignored_on_equals, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_index_hint_ignored_on_hash, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_init_convert_offset, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_sharded_tensor_lookup 2024-04-03T16:47:28.1118033Z 2024-04-03T16:47:28.1118587Z Running distributed/_tensor/test_view_ops 1/1 ... [2024-04-03 16:47:28.111337] 2024-04-03T16:47:28.1120987Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_view_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:47:28.111736] 2024-04-03T16:47:57.4085873Z 2024-04-03T16:47:57.4087827Z distributed/_tensor/test_view_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_view_ops_1.1_f8669f1e375bd7c4_.log 2024-04-03T16:47:57.4090434Z Running 3 items in this shard: test/distributed/_tensor/test_view_ops.py::TestViewOps::test_complex_view_ops, test/distributed/_tensor/test_view_ops.py::TestViewOps::test_view_groups, test/distributed/_tensor/test_view_ops.py::TestViewOps::test_view_ops 2024-04-03T16:47:57.4092278Z 2024-04-03T16:47:57.4093875Z Running distributed/_composable/fully_shard/test_fully_shard_init 1/1 ... [2024-04-03 16:47:57.409010] 2024-04-03T16:47:57.4098026Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:47:57.409452] 2024-04-03T16:48:14.2060727Z 2024-04-03T16:48:14.2062703Z distributed/_composable/fully_shard/test_fully_shard_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_init_1.1_edfc122e0068c659_.log 2024-04-03T16:48:14.2068009Z Running 6 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_device_id, test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_manual_fully_shard, test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_materialize_meta_module, test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_nested_fully_shard_shared_state, test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_policy, test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_sync_module_states 2024-04-03T16:48:14.2072207Z 2024-04-03T16:48:14.2072620Z Running distributed/test_functional_api 1/1 ... [2024-04-03 16:48:14.206494] 2024-04-03T16:48:14.2074533Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_functional_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:48:14.207011] 2024-04-03T16:49:35.6177153Z 2024-04-03T16:49:35.6178632Z distributed/test_functional_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_functional_api_1.1_d9807308dbc8baef_.log 2024-04-03T16:49:35.6229859Z Running 77 items in this shard: test/distributed/test_functional_api.py::TestExpand::test_expand_1d_rank_list, test/distributed/test_functional_api.py::TestExpand::test_expand_2d_rank_list, test/distributed/test_functional_api.py::TestExpand::test_expand_device_mesh, test/distributed/test_functional_api.py::TestExpand::test_expand_device_mesh_tuple, test/distributed/test_functional_api.py::TestExpand::test_expand_process_group, test/distributed/test_functional_api.py::TestPgTag::test_find_or_create_pg, test/distributed/test_functional_api.py::TestPgTag::test_find_root_pg, test/distributed/test_functional_api.py::TestPgTag::test_pg_creation_with_tag, test/distributed/test_functional_api.py::TestPgTag::test_pg_lookup_roundtrip, test/distributed/test_functional_api.py::TestPgTag::test_pg_lookup_with_tag, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_into_tensor_coalesced_device_cpu_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_into_tensor_coalesced_device_cpu_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_into_tensor_coalesced_device_cuda_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_into_tensor_coalesced_device_cuda_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_tensor_device_cpu_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_tensor_device_cpu_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_tensor_device_cuda_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_tensor_device_cuda_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_coalesced_eager_device_cpu_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_coalesced_eager_device_cpu_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_coalesced_eager_device_cuda_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_coalesced_eager_device_cuda_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_eager_device_cpu_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_eager_device_cpu_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_eager_device_cuda_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_eager_device_cuda_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_broadcast_device_cpu_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_broadcast_device_cpu_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_broadcast_device_cuda_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_broadcast_device_cuda_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_into_tensor_coalesced_device_cpu_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_into_tensor_coalesced_device_cpu_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_into_tensor_coalesced_device_cuda_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_into_tensor_coalesced_device_cuda_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_tensor_device_cpu_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_tensor_device_cpu_use_native_funcol_True, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_tensor_device_cuda_use_native_funcol_False, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_tensor_device_cuda_use_native_funcol_True, test/distributed/test_functional_api.py::TestMetaCollectives::test_all_reduce_use_native_funcol_False, test/distributed/test_functional_api.py::TestMetaCollectives::test_all_reduce_use_native_funcol_True, test/distributed/test_functional_api.py::TestGradCollectives::test_all_reduce_use_native_funcol_False, test/distributed/test_functional_api.py::TestGradCollectives::test_all_reduce_use_native_funcol_True, test/distributed/test_functional_api.py::TestMakeFx::test_all_reduce_tracing_use_native_funcol_False, test/distributed/test_functional_api.py::TestMakeFx::test_all_reduce_tracing_use_native_funcol_True, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_gather_into_tensor_coalesced_use_native_funcol_False, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_gather_into_tensor_coalesced_use_native_funcol_True, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_to_all_single_1d_input_use_native_funcol_False, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_to_all_single_1d_input_use_native_funcol_True, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_to_all_single_input_split_sizes_none, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_to_all_single_output_split_sizes_none, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_to_all_single_split_sizes_none, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_to_all_single_use_native_funcol_False, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_to_all_single_use_native_funcol_True, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_tracing_use_native_funcol_False, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_tracing_use_native_funcol_True, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_tracing_with_fakepg_use_native_funcol_False, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_tracing_with_fakepg_use_native_funcol_True, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_gather_into_tensor_coalesced_use_native_funcol_False, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_gather_into_tensor_coalesced_use_native_funcol_True, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_to_all_single_1d_input_use_native_funcol_False, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_to_all_single_1d_input_use_native_funcol_True, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_to_all_single_input_split_sizes_none, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_to_all_single_output_split_sizes_none, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_to_all_single_split_sizes_none, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_to_all_single_use_native_funcol_False, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_to_all_single_use_native_funcol_True, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_permute_tensor_with_sub_group_use_native_funcol_False, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_permute_tensor_with_sub_group_use_native_funcol_True, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_tracing_use_native_funcol_False, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_tracing_use_native_funcol_True, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_tracing_with_fakepg_use_native_funcol_False, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_tracing_with_fakepg_use_native_funcol_True, test/distributed/test_functional_api.py::TestOpWaitiness::test_add_triggers_wait, test/distributed/test_functional_api.py::TestOpWaitiness::test_dead_wrapper_plus_view, test/distributed/test_functional_api.py::TestOpWaitiness::test_dead_wrapper_triggers_wait, test/distributed/test_functional_api.py::TestOpWaitiness::test_view_does_not_trigger_wait, test/distributed/test_functional_api.py::TestOpWaitiness::test_wait_reduce_outstanding_work_count 2024-04-03T16:49:35.6276545Z 2024-04-03T16:49:35.6277077Z Running distributed/fsdp/test_fsdp_ignored_modules 1/1 ... [2024-04-03 16:49:35.619678] 2024-04-03T16:49:35.6279110Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_ignored_modules.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:49:35.620115] 2024-04-03T16:49:38.2929415Z 2024-04-03T16:49:38.2931089Z distributed/fsdp/test_fsdp_ignored_modules 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_ignored_modules_1.1_ebf12540cb17e94a_.log 2024-04-03T16:49:38.2941782Z Running 13 items in this shard: test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_diff_ignored_modules_across_ranks, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_invalid_composable_False, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_invalid_composable_True, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_nested, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_nested_composable, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_False_composable_False, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_False_composable_True, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_True_composable_False, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_True_composable_True, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_transformer, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_transformer_composable, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_states_auto_wrap, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_states_check 2024-04-03T16:49:38.2951274Z 2024-04-03T16:49:38.2951760Z Running distributed/fsdp/test_fsdp_tp_integration 1/1 ... [2024-04-03 16:49:38.293468] 2024-04-03T16:49:38.2953773Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:49:38.293864] 2024-04-03T16:49:56.1987372Z 2024-04-03T16:49:56.1989300Z distributed/fsdp/test_fsdp_tp_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_1b1e2f78c89a6fb0_.log 2024-04-03T16:49:56.1994762Z Running 6 items in this shard: test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_extension_grad, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload0, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload1, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload0, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload1, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_sync_module_state 2024-04-03T16:49:56.1999285Z 2024-04-03T16:49:56.1999762Z Running distributed/fsdp/test_checkpoint_wrapper 1/1 ... [2024-04-03 16:49:56.199154] 2024-04-03T16:49:56.2001868Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:49:56.199589] 2024-04-03T16:49:58.8723708Z 2024-04-03T16:49:58.8725747Z distributed/fsdp/test_checkpoint_wrapper 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_checkpoint_wrapper_1.1_6441f6d4343a8d1a_.log 2024-04-03T16:49:58.8732183Z Running 8 items in this shard: test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_apply_activation_checkpointing, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_args_kwargs, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_cpu_offload, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_kwarg_support, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_parity, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_forward_missing_attributes, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_fqn, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_load_activation_checkpointed_module 2024-04-03T16:49:58.8737506Z 2024-04-03T16:49:58.8737998Z Running distributed/fsdp/test_fsdp_grad_acc 1/1 ... [2024-04-03 16:49:58.872925] 2024-04-03T16:49:58.8740177Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:49:58.873377] 2024-04-03T16:50:15.5705168Z 2024-04-03T16:50:15.5707297Z distributed/fsdp/test_fsdp_grad_acc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_grad_acc_1.1_df192f7be621c155_.log 2024-04-03T16:50:15.5712454Z Running 6 items in this shard: test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs0_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs0_use_orig_params_True, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs1_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs1_use_orig_params_True, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_cpu_offload_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_cpu_offload_use_orig_params_True 2024-04-03T16:50:15.5716837Z 2024-04-03T16:50:15.5717315Z Running distributed/checkpoint/test_planner 1/1 ... [2024-04-03 16:50:15.571210] 2024-04-03T16:50:15.5720605Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_planner.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:50:15.571725] 2024-04-03T16:50:18.2955638Z 2024-04-03T16:50:18.2958197Z distributed/checkpoint/test_planner 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_planner_1.1_0df65cd45a29d038_.log 2024-04-03T16:50:18.2965673Z Running 6 items in this shard: test/distributed/checkpoint/test_planner.py::TestSavePlan::test_global_plan, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_load_with_resharding, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_load_with_world_size_diff_by_one, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_local_load_plan, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_local_plan, test/distributed/checkpoint/test_planner.py::TestPlannerHelpers::test_create_read_item_from_chunks 2024-04-03T16:50:18.2970984Z 2024-04-03T16:50:18.2971737Z Running distributed/fsdp/test_fsdp_apply 1/1 ... [2024-04-03 16:50:18.296299] 2024-04-03T16:50:18.2974462Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:50:18.296825] 2024-04-03T16:50:28.1310362Z 2024-04-03T16:50:28.1312346Z distributed/fsdp/test_fsdp_apply 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_apply_1.1_56ded0589b8d0e0d_.log 2024-04-03T16:50:28.1315112Z Running 3 items in this shard: test/distributed/fsdp/test_fsdp_apply.py::TestApply::test_apply_in_summon_raises_error, test/distributed/fsdp/test_fsdp_apply.py::TestApply::test_nested_module_apply, test/distributed/fsdp/test_fsdp_apply.py::TestApply::test_transformer_module_apply 2024-04-03T16:50:28.1317058Z 2024-04-03T16:50:28.1317649Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 ... [2024-04-03 16:50:28.131535] 2024-04-03T16:50:28.1323048Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:50:28.131977] 2024-04-03T16:51:13.9447224Z 2024-04-03T16:51:13.9449513Z distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_sharded_grad_scaler_1.1_f2b7b15cc3744c35_.log 2024-04-03T16:51:13.9478740Z Running 20 items in this shard: test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_grad_scaling, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_inf_gradients_skip_optim_step, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_scaling_unscaling_sparse, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_sharded_grad_scaler_found_inf 2024-04-03T16:51:13.9501936Z 2024-04-03T16:51:13.9502467Z Running distributed/pipeline/sync/test_pipeline 1/1 ... [2024-04-03 16:51:13.945413] 2024-04-03T16:51:13.9504500Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/test_pipeline.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:51:13.945944] 2024-04-03T16:51:15.5680508Z 2024-04-03T16:51:15.5682756Z distributed/pipeline/sync/test_pipeline 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_pipeline_1.1_6e37c2b58fbff0b7_.log 2024-04-03T16:51:15.5684833Z Running 1 items in this shard: test/distributed/pipeline/sync/test_pipeline.py::test_clock_cycles 2024-04-03T16:51:15.5685490Z 2024-04-03T16:51:15.5687150Z Running distributed/_composable/test_replicate_with_compiler 1/1 ... [2024-04-03 16:51:15.568487] 2024-04-03T16:51:15.5691596Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/test_replicate_with_compiler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:51:15.568866] 2024-04-03T16:51:18.2413577Z 2024-04-03T16:51:18.2415675Z distributed/_composable/test_replicate_with_compiler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_replicate_with_compiler_1.1_2974bd799da43978_.log 2024-04-03T16:51:18.2421835Z Running 9 items in this shard: test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_bucketing_coalesced_op, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_bucketing_concat_op, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_backward_only, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_bf16, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_cpu, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_cpu_no_sync, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_fp16, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_gpu, test/distributed/_composable/test_replicate_with_compiler.py::DDP_TP_Test::test_ddp_tp 2024-04-03T16:51:18.2426923Z 2024-04-03T16:51:18.2427368Z Running distributed/fsdp/test_fsdp_state_dict 1/1 ... [2024-04-03 16:51:18.241815] 2024-04-03T16:51:18.2429343Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:51:18.242233] 2024-04-03T16:51:21.2655327Z 2024-04-03T16:51:21.2657546Z distributed/fsdp/test_fsdp_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_state_dict_1.1_7ff3f4d88a04542f_.log 2024-04-03T16:51:21.2837508Z Running 179 items in this shard: test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_full_state_dict_missing_unexpected_keys_cleaned, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_local_state_dict_with_empty_ranks, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_sharded_load_multi_backend_pg, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_shared_module_and_shared_parameter, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_rank0_offload_save_load_flow_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_rank0_offload_save_load_flow_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_type, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_torch_save_load, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_world_size_one, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_wrong_state_dict_config, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict4GPUs::test_local_state_dict_reshard 2024-04-03T16:51:21.3013027Z 2024-04-03T16:51:21.3013567Z Running distributed/fsdp/test_fsdp_traversal 1/1 ... [2024-04-03 16:51:21.266672] 2024-04-03T16:51:21.3015546Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:51:21.267098] 2024-04-03T16:51:26.3444475Z 2024-04-03T16:51:26.3446260Z distributed/fsdp/test_fsdp_traversal 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_traversal_1.1_4f5f4356f76e5148_.log 2024-04-03T16:51:26.3448008Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_traversal.py::TestTraversal::test_fsdp_modules 2024-04-03T16:51:26.3448765Z 2024-04-03T16:51:26.3450666Z Running distributed/fsdp/test_fsdp_input 1/1 ... [2024-04-03 16:51:26.344869] 2024-04-03T16:51:26.3455602Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:51:26.345251] 2024-04-03T16:51:33.5741918Z 2024-04-03T16:51:33.5743486Z distributed/fsdp/test_fsdp_input 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_input_1.1_22b7dbc91fcec567_.log 2024-04-03T16:51:33.5745717Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_input.py::TestInput::test_input_type_dict, test/distributed/fsdp/test_fsdp_input.py::TestInput::test_input_type_list 2024-04-03T16:51:33.5746777Z 2024-04-03T16:51:33.5748922Z Running distributed/fsdp/test_fsdp_uneven 1/1 ... [2024-04-03 16:51:33.574640] 2024-04-03T16:51:33.5753607Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:51:33.575039] 2024-04-03T16:51:38.8027529Z 2024-04-03T16:51:38.8029393Z distributed/fsdp/test_fsdp_uneven 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_uneven_1.1_144fd96c7f1d0440_.log 2024-04-03T16:51:38.8031209Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShard::test_one_iteration 2024-04-03T16:51:38.8032020Z 2024-04-03T16:51:38.8033962Z Running distributed/fsdp/test_fsdp_multiple_wrapping 1/1 ... [2024-04-03 16:51:38.803148] 2024-04-03T16:51:38.8038926Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:51:38.803540] 2024-04-03T16:51:43.9814958Z 2024-04-03T16:51:43.9816806Z distributed/fsdp/test_fsdp_multiple_wrapping 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_multiple_wrapping_1.1_aa57bcf528c1d1ab_.log 2024-04-03T16:51:43.9818844Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_multiple_wrapping.py::TestMultipleWrapping::test_multiple_wrapping 2024-04-03T16:51:43.9819689Z 2024-04-03T16:51:43.9825672Z Running distributed/_composable/fully_shard/test_fully_shard_runtime 1/1 ... [2024-04-03 16:51:43.982314] 2024-04-03T16:51:43.9831750Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_runtime.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:51:43.982858] 2024-04-03T16:51:46.7066005Z 2024-04-03T16:51:46.7068246Z distributed/_composable/fully_shard/test_fully_shard_runtime 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_runtime_1.1_c4aedd956e85e9cf_.log 2024-04-03T16:51:46.7071153Z Running 2 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_runtime.py::TestRuntime::test_training, test/distributed/_composable/fully_shard/test_fully_shard_runtime.py::TestRuntime::test_unshard_reshard_order 2024-04-03T16:51:46.7072477Z 2024-04-03T16:51:46.7072926Z Running distributed/fsdp/test_fsdp_clip_grad_norm 1/1 ... [2024-04-03 16:51:46.706932] 2024-04-03T16:51:46.7077741Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:51:46.707392] 2024-04-03T16:51:59.6015636Z 2024-04-03T16:51:59.6017682Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_clip_grad_norm_1.1_6011e1c571e7fae4_.log 2024-04-03T16:51:59.6020811Z Running 4 items in this shard: test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNorm::test_ddp_parity, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNorm::test_low_precision_grads, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNorm::test_no_gradients, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNorm::test_non_root 2024-04-03T16:51:59.6023300Z 2024-04-03T16:51:59.6023737Z Running distributed/fsdp/test_fsdp_exec_order 1/1 ... [2024-04-03 16:51:59.602011] 2024-04-03T16:51:59.6027210Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_exec_order.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:51:59.602397] 2024-04-03T16:52:22.7183111Z 2024-04-03T16:52:22.7185336Z distributed/fsdp/test_fsdp_exec_order 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_exec_order_1.1_62f87da35cf9acff_.log 2024-04-03T16:52:22.7191970Z Running 8 items in this shard: test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_invalid_first_iter_order_sharding_strategy0, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_invalid_first_iter_order_sharding_strategy1, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_invalid_later_iter_order_sharding_strategy0_iters_before_path_change_1, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_invalid_later_iter_order_sharding_strategy0_iters_before_path_change_3, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_invalid_later_iter_order_sharding_strategy1_iters_before_path_change_1, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_invalid_later_iter_order_sharding_strategy1_iters_before_path_change_3, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_train_eval_sharding_strategy0, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrder::test_train_eval_sharding_strategy1 2024-04-03T16:52:22.7197693Z 2024-04-03T16:52:22.7198258Z Running distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 ... [2024-04-03 16:52:22.718828] 2024-04-03T16:52:22.7200475Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:52:22.719283] 2024-04-03T16:52:25.0416515Z 2024-04-03T16:52:25.0419436Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks_1.1_4284e4eba063f87d_.log 2024-04-03T16:52:25.0429959Z Running 6 items in this shard: test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_allreduce_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_fp16compress_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_noop_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_quantize_per_channel_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_quantize_per_tensor_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_is_last_hook 2024-04-03T16:52:25.0437256Z 2024-04-03T16:52:25.0437881Z Running distributed/fsdp/test_wrap 1/1 ... [2024-04-03 16:52:25.042368] 2024-04-03T16:52:25.0440723Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:52:25.042905] 2024-04-03T16:53:33.1067452Z 2024-04-03T16:53:33.1069451Z distributed/fsdp/test_wrap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_wrap_1.1_598ef885ae7677ce_.log 2024-04-03T16:53:33.1103312Z Running 51 items in this shard: test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_bn_always_wrapped_individually, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_False_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_False_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_True_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_True_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_False_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_False_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_True_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_True_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_False_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_False_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_True_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_True_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_False_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_False_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_True_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_True_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_False_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_False_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_True_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_True_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_wrap_batchnorm_individually_use_or_policy_False, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_wrap_batchnorm_individually_use_or_policy_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap_with_ignored_modules_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap_with_ignored_modules_wrap_method1, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_api, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_exclude_wrap, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_exclude_wrap_include_children, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_force_leaf, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_force_leaf_custom, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_cuda_init_mode0_cpu_offload0_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_cuda_init_mode0_cpu_offload0_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_cuda_init_mode0_cpu_offload1_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_cuda_init_mode0_cpu_offload1_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_cuda_init_mode1_cpu_offload0_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_cuda_init_mode1_cpu_offload0_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_cuda_init_mode1_cpu_offload1_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_cuda_init_mode1_cpu_offload1_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_with_ignored_modules_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_with_ignored_modules_wrap_method1, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_custom_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_frozen_params, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_module_wrap_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_module_wrap_policy_callable, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_transformer_auto_wrap_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_disabled_outside_context, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_override_defaults, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_wrap_method1, test/distributed/fsdp/test_wrap.py::TestWrapUtils::test_validate_frozen_params 2024-04-03T16:53:33.1133186Z 2024-04-03T16:53:33.1133786Z Running distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 ... [2024-04-03 16:53:33.107438] 2024-04-03T16:53:33.1135950Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint_cpu.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:53:33.107837] 2024-04-03T16:54:00.2690834Z 2024-04-03T16:54:00.2694879Z distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_cpu_1.1_15a5985f7e646b42_.log 2024-04-03T16:54:00.2707382Z Running 12 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_save_load_bytes_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_save_load_bytes_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor_thread_count_2 2024-04-03T16:54:00.2718811Z 2024-04-03T16:54:00.2719276Z Running distributed/fsdp/test_fsdp_comm_hooks 1/1 ... [2024-04-03 16:54:00.269737] 2024-04-03T16:54:00.2721694Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm_hooks.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:54:00.270135] 2024-04-03T16:54:43.4269046Z 2024-04-03T16:54:43.4270880Z distributed/fsdp/test_fsdp_comm_hooks 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_comm_hooks_1.1_4505069e43e3c6e1_.log 2024-04-03T16:54:43.4293307Z Running 28 items in this shard: test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_False_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_True_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_behavior_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_behavior_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_behavior_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_False_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_True_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_hybrid_strategy, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_non_root_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_non_root_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_non_root_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_submodules_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_submodules_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_submodules_sharding_strategy2 2024-04-03T16:54:43.4313079Z 2024-04-03T16:54:43.4313710Z Running distributed/algorithms/quantization/test_quantization 1/1 ... [2024-04-03 16:54:43.427449] 2024-04-03T16:54:43.4328476Z MPI not available -- MPI backend tests will be skipped 2024-04-03T16:54:43.4330142Z Running distributed tests for the test backend with env init_method 2024-04-03T16:54:43.4337470Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:54:43.433448] 2024-04-03T16:54:45.6818211Z 2024-04-03T16:54:45.6821171Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_b5dcf5ab27f7f30b_.log 2024-04-03T16:54:45.6824019Z Running 0 items in this shard: 2024-04-03T16:54:45.6824447Z 2024-04-03T16:54:45.6829213Z Running distributed tests for the test backend with file init_method 2024-04-03T16:54:45.6837696Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:54:45.683304] 2024-04-03T16:54:47.9296528Z 2024-04-03T16:54:47.9298819Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_d4b79d84255014b8_.log 2024-04-03T16:54:47.9300289Z Running 0 items in this shard: 2024-04-03T16:54:47.9300558Z 2024-04-03T16:54:47.9303832Z Running distributed tests for the gloo backend with env init_method 2024-04-03T16:54:47.9310196Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:54:47.930672] 2024-04-03T16:55:09.4016350Z 2024-04-03T16:55:09.4018587Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_19bd0b51fd36c3bd_.log 2024-04-03T16:55:09.4026250Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-04-03T16:55:09.4031227Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16 2024-04-03T16:55:09.4032840Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16 2024-04-03T16:55:09.4034611Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16 2024-04-03T16:55:09.4036143Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16 2024-04-03T16:55:09.4037819Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16 2024-04-03T16:55:09.4039437Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-04-03T16:55:09.4040394Z 2024-04-03T16:55:09.4040681Z Running distributed tests for the gloo backend with file init_method 2024-04-03T16:55:09.4042863Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:55:09.402949] 2024-04-03T16:55:30.9476759Z 2024-04-03T16:55:30.9478624Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_8e7b78bc95c25a27_.log 2024-04-03T16:55:30.9485164Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-04-03T16:55:30.9489846Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16 2024-04-03T16:55:30.9491842Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16 2024-04-03T16:55:30.9493498Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16 2024-04-03T16:55:30.9495513Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16 2024-04-03T16:55:30.9497080Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16 2024-04-03T16:55:30.9498691Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-04-03T16:55:30.9499592Z 2024-04-03T16:55:30.9500103Z Running distributed/fsdp/test_fsdp_use_orig_params 1/1 ... [2024-04-03 16:55:30.948947] 2024-04-03T16:55:30.9502128Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_use_orig_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:55:30.949293] 2024-04-03T16:56:23.2046075Z 2024-04-03T16:56:23.2047861Z distributed/fsdp/test_fsdp_use_orig_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_use_orig_params_1.1_62278d94efd278c8_.log 2024-04-03T16:56:23.2069084Z Running 25 items in this shard: test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_full_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_no_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_shard_grad_op, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_full_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_no_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_shard_grad_op, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_trainability, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_fsdp_compile, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_multiple_optimizers, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_multiple_forward_offload_params_False, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_multiple_forward_offload_params_True, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_summon_between_two_forwards_offload_params_False, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_summon_between_two_forwards_offload_params_True, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsParamAccess::test_access_params_after_forward, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_grad_writeback, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_no_reshard_and_mixed_precision, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_param_writeback, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_writeback_between_fwd_and_bwd_for_no_reshard_raises, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_writeback_shape_mismatch, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsFQNs::test_named_parameters_in_forward, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsNoSync::test_no_sync_correctness, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsNoSync::test_no_sync_mixed_precision, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsInit::test_non_uniform_requires_grad, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestMultiTensorApply::test_multi_tensor_apply_size0_tensors_cpu, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestMultiTensorApply::test_multi_tensor_apply_size0_tensors_cuda 2024-04-03T16:56:23.2088365Z 2024-04-03T16:56:23.2088803Z Running distributed/fsdp/test_utils 1/1 ... [2024-04-03 16:56:23.205186] 2024-04-03T16:56:23.2090744Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:56:23.205598] 2024-04-03T16:56:25.8281977Z 2024-04-03T16:56:25.8284460Z distributed/fsdp/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_utils_1.1_8d1c4ce8050de0fa_.log 2024-04-03T16:56:25.8289985Z Running 5 items in this shard: test/distributed/fsdp/test_utils.py::TestUtils::test_apply_to_tensors_cpu_cuda, test/distributed/fsdp/test_utils.py::TestUtils::test_apply_to_tensors_devices0, test/distributed/fsdp/test_utils.py::TestUtils::test_apply_to_tensors_devices1, test/distributed/fsdp/test_utils.py::TestUtils::test_packed_sequence, test/distributed/fsdp/test_utils.py::TestUtils::test_replace_by_prefix 2024-04-03T16:56:25.8293896Z 2024-04-03T16:56:25.8294386Z Running distributed/pipeline/sync/test_phony 1/1 ... [2024-04-03 16:56:25.828713] 2024-04-03T16:56:25.8296516Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/test_phony.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:56:25.829173] 2024-04-03T16:56:27.4501632Z 2024-04-03T16:56:27.4503465Z distributed/pipeline/sync/test_phony 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_phony_1.1_fb816301847105d5_.log 2024-04-03T16:56:27.4506592Z Running 4 items in this shard: test/distributed/pipeline/sync/test_phony.py::test_phony_size, test/distributed/pipeline/sync/test_phony.py::test_phony_requires_grad, test/distributed/pipeline/sync/test_phony.py::test_cached_phony, test/distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function 2024-04-03T16:56:27.4508729Z 2024-04-03T16:56:27.4509217Z Running distributed/test_c10d_object_collectives 1/1 ... [2024-04-03 16:56:27.450621] 2024-04-03T16:56:27.4514087Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:56:27.451068] 2024-04-03T16:56:48.6036302Z 2024-04-03T16:56:48.6037875Z distributed/test_c10d_object_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_object_collectives_1.1_39c4eb4096740940_.log 2024-04-03T16:56:48.6044810Z Running 8 items in this shard: test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_all_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_broadcast_object_list, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_scatter_object_list, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_all_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_broadcast_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_scatter_object 2024-04-03T16:56:48.6049785Z 2024-04-03T16:56:48.6050307Z Running distributed/fsdp/test_fsdp_multiple_forward 1/1 ... [2024-04-03 16:56:48.604161] 2024-04-03T16:56:48.6052573Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:56:48.604579] 2024-04-03T16:56:53.8328561Z 2024-04-03T16:56:53.8330818Z distributed/fsdp/test_fsdp_multiple_forward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_multiple_forward_1.1_09fd75e637af4be9_.log 2024-04-03T16:56:53.8332622Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_multiple_forward.py::TestMultiForward::test_multi_forward 2024-04-03T16:56:53.8334641Z 2024-04-03T16:56:53.8335452Z Running distributed/fsdp/test_fsdp_comm 1/1 ... [2024-04-03 16:56:53.833293] 2024-04-03T16:56:53.8340257Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:56:53.833697] 2024-04-03T16:57:16.9483621Z 2024-04-03T16:57:16.9485662Z distributed/fsdp/test_fsdp_comm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_comm_1.1_6268768ce900e0e3_.log 2024-04-03T16:57:16.9495307Z Running 8 items in this shard: test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_False_use_no_sync_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_False_use_no_sync_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_False_use_no_sync_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_False_use_no_sync_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_True_use_no_sync_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_True_use_no_sync_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_True_use_no_sync_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_True_use_no_sync_True_sharding_strategy1 2024-04-03T16:57:16.9501331Z 2024-04-03T16:57:16.9501823Z Running distributed/pipeline/sync/test_bugs 1/1 ... [2024-04-03 16:57:16.948823] 2024-04-03T16:57:16.9503808Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/test_bugs.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:57:16.949256] 2024-04-03T16:57:19.3224413Z 2024-04-03T16:57:19.3226329Z distributed/pipeline/sync/test_bugs 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_bugs_1.1_0666cd215a5202ad_.log 2024-04-03T16:57:19.3229185Z Running 4 items in this shard: test/distributed/pipeline/sync/test_bugs.py::test_python_autograd_function, test/distributed/pipeline/sync/test_bugs.py::test_exception_no_hang, test/distributed/pipeline/sync/test_bugs.py::test_tuple_wait, test/distributed/pipeline/sync/test_bugs.py::test_parallel_randoms 2024-04-03T16:57:19.3231285Z 2024-04-03T16:57:19.3231855Z Running distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 ... [2024-04-03 16:57:19.322865] 2024-04-03T16:57:19.3236079Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_hsdp_dtensor_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:57:19.323316] 2024-04-03T16:57:42.3025641Z 2024-04-03T16:57:42.3027580Z distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_hsdp_dtensor_state_dict_1.1_7c15ae9047c4e7a5_.log 2024-04-03T16:57:42.3036121Z Running 8 items in this shard: test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_False, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_True, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_False, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_True, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_False, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_True, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_hsdp_init_with_device_mesh, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_root_module_is_not_FSDP 2024-04-03T16:57:42.3044120Z 2024-04-03T16:57:42.3044616Z Running distributed/pipeline/sync/skip/test_gpipe 1/1 ... [2024-04-03 16:57:42.303131] 2024-04-03T16:57:42.3047016Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_gpipe.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:57:42.303577] 2024-04-03T16:57:44.2752552Z 2024-04-03T16:57:44.2754590Z distributed/pipeline/sync/skip/test_gpipe 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_gpipe_1.1_70f63913abe31b57_.log 2024-04-03T16:57:44.2762676Z Running 13 items in this shard: test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip 2024-04-03T16:57:44.2768383Z 2024-04-03T16:57:44.2768793Z Running distributed/algorithms/test_join 1/1 ... [2024-04-03 16:57:44.275712] 2024-04-03T16:57:44.2770723Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/algorithms/test_join.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:57:44.276136] 2024-04-03T16:58:07.7873797Z 2024-04-03T16:58:07.7876417Z distributed/algorithms/test_join 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.test_join_1.1_0b3cede2e48cd9ae_.log 2024-04-03T16:58:07.7885444Z Running 9 items in this shard: test/distributed/algorithms/test_join.py::TestJoin::test_join_kwargs, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinable_disable, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinables, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinables_throw, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_disable, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_main_hooks, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_post_hooks, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_throw 2024-04-03T16:58:07.7891982Z 2024-04-03T16:58:07.7892808Z Running distributed/_shard/sharded_tensor/ops/test_embedding 1/1 ... [2024-04-03 16:58:07.787999] 2024-04-03T16:58:07.7895907Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:58:07.788543] 2024-04-03T16:58:10.1110424Z 2024-04-03T16:58:10.1112820Z distributed/_shard/sharded_tensor/ops/test_embedding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_embedding_1.1_e31eacf45c3e85ad_.log 2024-04-03T16:58:10.1115973Z Running 2 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_embedding.py::TestShardedEmbedding::test_sharded_embedding_colwise, test/distributed/_shard/sharded_tensor/ops/test_embedding.py::TestShardedEmbedding::test_sharded_embedding_rowwise 2024-04-03T16:58:10.1117752Z 2024-04-03T16:58:10.1118331Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 ... [2024-04-03 16:58:10.111568] 2024-04-03T16:58:10.1123988Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:58:10.112037] 2024-04-03T16:58:12.3840243Z 2024-04-03T16:58:12.3842534Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_embedding_bag_1.1_c25966ca0239cc11_.log 2024-04-03T16:58:12.3845348Z Running 2 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_embedding_bag.py::TestShardedEmbeddingBag::test_sharded_embedding_bag_colwise, test/distributed/_shard/sharded_tensor/ops/test_embedding_bag.py::TestShardedEmbeddingBag::test_sharded_embedding_bag_rowwise 2024-04-03T16:58:12.3847443Z 2024-04-03T16:58:12.3847901Z Running distributed/pipeline/sync/test_inplace 1/1 ... [2024-04-03 16:58:12.384431] 2024-04-03T16:58:12.3851933Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/test_inplace.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:58:12.384853] 2024-04-03T16:58:14.1564202Z 2024-04-03T16:58:14.1566078Z distributed/pipeline/sync/test_inplace 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_inplace_1.1_2ac9f1e9c8fba849_.log 2024-04-03T16:58:14.1568692Z Running 3 items in this shard: test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad 2024-04-03T16:58:14.1570658Z 2024-04-03T16:58:14.1571130Z Running distributed/rpc/cuda/test_tensorpipe_agent 1/1 ... [2024-04-03 16:58:14.156903] 2024-04-03T16:58:14.1576373Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:58:14.157321] 2024-04-03T17:08:13.0267635Z 2024-04-03T17:08:13.0270100Z distributed/rpc/cuda/test_tensorpipe_agent 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.cuda.test_tensorpipe_agent_1.1_9106023c6652bd57_.log 2024-04-03T17:08:13.0360896Z Running 107 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRpcTest::test_profiler_remote_cuda, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_simple, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_to_cpu_continuation, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_to_cpu_continuation_gpu_root, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_input_moved_to_cuda_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_input_moved_to_cuda_device_script, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_invalid_devices, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_valid_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDdpComparisonTest::test_ddp_dist_autograd_local_vs_remote_gpu, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_always, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_except_last, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_never, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_never_find_unused, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_always, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_except_last, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_never, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_never_find_unused, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_async_execution_nested_with_cuda_future, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_async_execution_with_cuda_future, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_callback_changes_devices, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_cuda_sparse_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_cuda_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_custom_class_with_cuda_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_list_with_cuda_sparse_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_list_with_cuda_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_int, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_str, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_not_cuda, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_modify_tensor_inplace, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_replace_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_value_on_bad_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_multi, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_nested, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_nested_multi, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu_to_gpu_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu_to_gpu_non_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_default_to_non_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_5, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_6, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_7, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_8, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_5, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_6, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_7, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_8, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_non_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_non_default_to_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_to_cpu_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_to_cpu_non_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_gpu, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_in_options, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_max_local_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_max_remote_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_min_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_many_to_one, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_loop, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_not_timeout, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_remote, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_remote_response, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_response, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_response_loop, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_multi_gpu, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_multi_gpu_self, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_one_to_many, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_remote, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_return_to_gpu, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_return_to_gpu_self, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_wrong_worker_name, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_mismatch, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_devices_option_mismatch, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_devices_option_mismatch_reverse, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization5, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_with_unpickleable_attributes, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_tensor_view_as_return_value, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_device_maps_backward_pass, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_dist_autograd_sync_streams, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_gradients_synchronizations 2024-04-03T17:08:13.0436366Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRpcTest::test_profiler_remote_cuda 2024-04-03T17:08:13.0437930Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_simple 2024-04-03T17:08:13.0439466Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_to_cpu_continuation 2024-04-03T17:08:13.0441115Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_to_cpu_continuation_gpu_root 2024-04-03T17:08:13.0442856Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_input_moved_to_cuda_device 2024-04-03T17:08:13.0444515Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_input_moved_to_cuda_device_script 2024-04-03T17:08:13.0446108Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_invalid_devices 2024-04-03T17:08:13.0447607Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_valid_device 2024-04-03T17:08:13.0449213Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDdpComparisonTest::test_ddp_dist_autograd_local_vs_remote_gpu 2024-04-03T17:08:13.0450852Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_always 2024-04-03T17:08:13.0452396Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_except_last 2024-04-03T17:08:13.0453930Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_never 2024-04-03T17:08:13.0455482Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_never_find_unused 2024-04-03T17:08:13.0457048Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_always 2024-04-03T17:08:13.0458591Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_except_last 2024-04-03T17:08:13.0460127Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_never 2024-04-03T17:08:13.0461664Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_never_find_unused 2024-04-03T17:08:13.0463408Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_async_execution_nested_with_cuda_future 2024-04-03T17:08:13.0465209Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_async_execution_with_cuda_future 2024-04-03T17:08:13.0466992Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_callback_changes_devices 2024-04-03T17:08:13.0468806Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_cuda_sparse_tensor 2024-04-03T17:08:13.0470772Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_cuda_tensor 2024-04-03T17:08:13.0472792Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor 2024-04-03T17:08:13.0474868Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_custom_class_with_cuda_tensor 2024-04-03T17:08:13.0476837Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_list_with_cuda_sparse_tensor 2024-04-03T17:08:13.0478739Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_list_with_cuda_tensor 2024-04-03T17:08:13.0480549Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_device 2024-04-03T17:08:13.0482257Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_int 2024-04-03T17:08:13.0484023Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_str 2024-04-03T17:08:13.0485709Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_not_cuda 2024-04-03T17:08:13.0487446Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_modify_tensor_inplace 2024-04-03T17:08:13.0489180Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_replace_tensor 2024-04-03T17:08:13.0490905Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_value_on_bad_device 2024-04-03T17:08:13.0492561Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream 2024-04-03T17:08:13.0494167Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_multi 2024-04-03T17:08:13.0495814Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_nested 2024-04-03T17:08:13.0497490Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_nested_multi 2024-04-03T17:08:13.0499196Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu 2024-04-03T17:08:13.0500869Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu_to_gpu_default 2024-04-03T17:08:13.0502622Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu_to_gpu_non_default 2024-04-03T17:08:13.0504349Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_default 2024-04-03T17:08:13.0506073Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_default_to_non_default 2024-04-03T17:08:13.0507815Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_1 2024-04-03T17:08:13.0509488Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_2 2024-04-03T17:08:13.0511395Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_3 2024-04-03T17:08:13.0513046Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_4 2024-04-03T17:08:13.0514788Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_5 2024-04-03T17:08:13.0516459Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_6 2024-04-03T17:08:13.0518119Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_7 2024-04-03T17:08:13.0519771Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_8 2024-04-03T17:08:13.0521469Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_1 2024-04-03T17:08:13.0523264Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_2 2024-04-03T17:08:13.0524976Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_3 2024-04-03T17:08:13.0526661Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_4 2024-04-03T17:08:13.0528369Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_5 2024-04-03T17:08:13.0530093Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_6 2024-04-03T17:08:13.0531805Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_7 2024-04-03T17:08:13.0533505Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_8 2024-04-03T17:08:13.0535211Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_non_default 2024-04-03T17:08:13.0536965Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_non_default_to_default 2024-04-03T17:08:13.0538730Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_to_cpu_default 2024-04-03T17:08:13.0540480Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_to_cpu_non_default 2024-04-03T17:08:13.0542166Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_gpu 2024-04-03T17:08:13.0543809Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_in_options 2024-04-03T17:08:13.0545537Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_max_local_device 2024-04-03T17:08:13.0547328Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_max_remote_device 2024-04-03T17:08:13.0549114Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_min_device 2024-04-03T17:08:13.0551135Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_many_to_one 2024-04-03T17:08:13.0552928Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config 2024-04-03T17:08:13.0554646Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_loop 2024-04-03T17:08:13.0556432Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_not_timeout 2024-04-03T17:08:13.0558230Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_remote 2024-04-03T17:08:13.0560050Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_remote_response 2024-04-03T17:08:13.0561857Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_response 2024-04-03T17:08:13.0563747Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_response_loop 2024-04-03T17:08:13.0565495Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_multi_gpu 2024-04-03T17:08:13.0567171Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_multi_gpu_self 2024-04-03T17:08:13.0568871Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_one_to_many 2024-04-03T17:08:13.0570514Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_remote 2024-04-03T17:08:13.0572174Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_return_to_gpu 2024-04-03T17:08:13.0573888Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_return_to_gpu_self 2024-04-03T17:08:13.0575631Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_wrong_worker_name 2024-04-03T17:08:13.0577277Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_mismatch 2024-04-03T17:08:13.0578918Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_devices_option_mismatch 2024-04-03T17:08:13.0580628Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_devices_option_mismatch_reverse 2024-04-03T17:08:13.0582410Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization1 2024-04-03T17:08:13.0584187Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization2 2024-04-03T17:08:13.0585973Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization3 2024-04-03T17:08:13.0587747Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization4 2024-04-03T17:08:13.0589698Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization1 2024-04-03T17:08:13.0591536Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization2 2024-04-03T17:08:13.0593267Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization3 2024-04-03T17:08:13.0595001Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization4 2024-04-03T17:08:13.0596728Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization5 2024-04-03T17:08:13.0598446Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization1 2024-04-03T17:08:13.0600282Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization2 2024-04-03T17:08:13.0602036Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization3 2024-04-03T17:08:13.0603834Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization4 2024-04-03T17:08:13.0605548Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization1 2024-04-03T17:08:13.0607278Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization2 2024-04-03T17:08:13.0609011Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization3 2024-04-03T17:08:13.0610745Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization4 2024-04-03T17:08:13.0612484Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_with_unpickleable_attributes 2024-04-03T17:08:13.0614230Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_tensor_view_as_return_value 2024-04-03T17:08:13.0615963Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_device_maps_backward_pass 2024-04-03T17:08:13.0617721Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_dist_autograd_sync_streams 2024-04-03T17:08:13.0619459Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_gradients_synchronizations 2024-04-03T17:08:13.0620423Z 2024-04-03T17:08:13.0620950Z Running distributed/rpc/test_tensorpipe_agent 2/3 ... [2024-04-03 17:08:13.030923] 2024-04-03T17:08:13.0623000Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/rpc/test_tensorpipe_agent.py', '--shard-id=2', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:08:13.031352] 2024-04-03T17:20:30.0525507Z 2024-04-03T17:20:30.0527051Z distributed/rpc/test_tensorpipe_agent 2/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.test_tensorpipe_agent_2.3_7657fa56eb108670_.log 2024-04-03T17:20:30.0616124Z Running 116 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_chained_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_raise_async, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_wrong_return_type_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_build_rpc_profiling_key, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_multi, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_none, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_simple, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_dist_init_decorator, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_duplicate_name_2, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_function_not_on_callee, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_future_done, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_future_done_exception, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_future_in_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_future_wait_twice, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_get_worker_infos, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_init_rpc_twice, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_int_callee, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_multi_builtin_remote_ret, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_nested_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_non_garbage_collected_user_rref_due_to_local_circular_dependency, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_owner_rref_backward, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_async_rpc_builtin_single_threaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_remote_udf, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_async_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_remote_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_sync_rpc_builtin, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_sync_rpc_builtin_single_threaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_class_instance_method, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_multi_async_call, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_nested_pickle, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_rpc_rref_args, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_reinit, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_context_debug_info, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_forward_chain, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_get_future, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_slow_init, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_scalar_add, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_self_add, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_self_remote_rref_as_rpc_arg, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_send_to_rank, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_shutdown_followed_by_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_use_rref_after_shutdown, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_raise_in_user_func, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_with_partial_exception, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_workers_dense, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_workers_twice_dense, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_worker_info_pickle, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_world_size_one, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_autograd_context, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_different_tensor_dims, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_multiple_round_trips, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_node_failure, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_rref_nested, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_simple_script_call, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_simple_self, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_unused_send_function, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_context_cleanup_tensor_with_grad, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_dist_autograd_profiling, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_grad_only_on_return_value_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_py_nested_call, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_python_remote_call, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_nested_context, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_rpc_complex_args, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistOptimizerTest::test_dist_optim_none_grads, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_add_done_callback, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_args_and_kwargs_contain_different_types, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_function_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_function_remote_multi, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_call_fork_in_jit_with_profiling, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_call_script_function_that_not_exists_remotely_from_script, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_callback_chain, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_local_rref_local_value, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_record_function_jit_end_callbacks_with_fork, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_remote_script_module, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_remote_script_throw, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_rref_as_arg_and_return, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_torchscript_functions_not_supported, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_unexepected_kwarg_is_specified, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitDistAutogradTest::test_dist_backward, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitDistAutogradTest::test_get_gradients, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitDistAutogradTest::test_jit_fork_within_context, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitDistAutogradTest::test_restore_context_after_swtich_to_jit_thread, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_forward_sync_script, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_train_eval, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeThreeWorkersRemoteModuleTest::test_create_remote_module_from_module_rref, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeThreeWorkersRemoteModuleTest::test_send_remote_module_over_the_wire, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeThreeWorkersRemoteModuleTest::test_send_remote_module_over_the_wire_script_not_supported, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDdpUnderDistAutogradTest::test_backward_ddp_inside, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDdpUnderDistAutogradTest::test_backward_no_ddp, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDdpComparisonTest::test_ddp_dist_autograd_local_vs_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_builtin_remote_ret_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_dynamic_rpc_init_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_dynamic_rpc_init_rpc_without_rank, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_multi_py_udf_remote_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_my_parameter_server_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_nested_remote_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_nested_rref_stress_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_py_rpc_rref_args_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_py_rref_args_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_rref_get_type_timeout_blocking, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_rref_proxy_timeout, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_self_py_udf_remote_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_self_remote_rref_as_self_remote_arg_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_set_and_get_num_worker_threads, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_no_grad_on_tensor_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_rref_multi_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_simple_self_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_simple_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_context_cleanup_tensor_no_grad_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_graph_for_builtin_call_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_graph_for_py_nested_call_itself_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_multiple_backward_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_nested_backward_accumulate_grads_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_no_graph_with_tensors_not_require_grad_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_trainer_ps_sparse 2024-04-03T17:20:30.0683694Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_chained_remote 2024-04-03T17:20:30.0685115Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_raise_async 2024-04-03T17:20:30.0686687Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_wrong_return_type_remote 2024-04-03T17:20:30.0688122Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_build_rpc_profiling_key 2024-04-03T17:20:30.0689422Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_multi 2024-04-03T17:20:30.0690659Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_none 2024-04-03T17:20:30.0691913Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_simple 2024-04-03T17:20:30.0693189Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_dist_init_decorator 2024-04-03T17:20:30.0694478Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_duplicate_name_2 2024-04-03T17:20:30.0695772Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_function_not_on_callee 2024-04-03T17:20:30.0697042Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_future_done 2024-04-03T17:20:30.0698311Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_future_done_exception 2024-04-03T17:20:30.0699585Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_future_in_rpc 2024-04-03T17:20:30.0700833Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_future_wait_twice 2024-04-03T17:20:30.0702105Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_get_worker_infos 2024-04-03T17:20:30.0703352Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_init_rpc_twice 2024-04-03T17:20:30.0704569Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_int_callee 2024-04-03T17:20:30.0705852Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_multi_builtin_remote_ret 2024-04-03T17:20:30.0707143Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_nested_remote 2024-04-03T17:20:30.0708641Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_non_garbage_collected_user_rref_due_to_local_circular_dependency 2024-04-03T17:20:30.0710624Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_owner_rref_backward 2024-04-03T17:20:30.0712183Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_async_rpc_builtin_single_threaded 2024-04-03T17:20:30.0713661Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_remote_udf 2024-04-03T17:20:30.0715054Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_async_rpc 2024-04-03T17:20:30.0716466Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_remote_rpc 2024-04-03T17:20:30.0717888Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_sync_rpc_builtin 2024-04-03T17:20:30.0719494Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_sync_rpc_builtin_single_threaded 2024-04-03T17:20:30.0721045Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_class_instance_method 2024-04-03T17:20:30.0722433Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_multi_async_call 2024-04-03T17:20:30.0723722Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_nested_pickle 2024-04-03T17:20:30.0724985Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_rpc_rref_args 2024-04-03T17:20:30.0726197Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_reinit 2024-04-03T17:20:30.0727442Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_context_debug_info 2024-04-03T17:20:30.0728768Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_forward_chain 2024-04-03T17:20:30.0730039Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_get_future 2024-04-03T17:20:30.0731307Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_slow_init 2024-04-03T17:20:30.0732563Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_scalar_add 2024-04-03T17:20:30.0733759Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_self_add 2024-04-03T17:20:30.0735040Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_self_remote_rref_as_rpc_arg 2024-04-03T17:20:30.0736332Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_send_to_rank 2024-04-03T17:20:30.0737633Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_shutdown_followed_by_rpc 2024-04-03T17:20:30.0738975Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_use_rref_after_shutdown 2024-04-03T17:20:30.0740338Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_raise_in_user_func 2024-04-03T17:20:30.0741726Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_with_partial_exception 2024-04-03T17:20:30.0743102Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_workers_dense 2024-04-03T17:20:30.0744467Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_workers_twice_dense 2024-04-03T17:20:30.0745809Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_worker_info_pickle 2024-04-03T17:20:30.0747062Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_world_size_one 2024-04-03T17:20:30.0748372Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_autograd_context 2024-04-03T17:20:30.0750013Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_different_tensor_dims 2024-04-03T17:20:30.0751550Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_multiple_round_trips 2024-04-03T17:20:30.0753039Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_node_failure 2024-04-03T17:20:30.0754472Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_rref_nested 2024-04-03T17:20:30.0756034Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_simple_script_call 2024-04-03T17:20:30.0757488Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_simple_self 2024-04-03T17:20:30.0759047Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_unused_send_function 2024-04-03T17:20:30.0760704Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_context_cleanup_tensor_with_grad 2024-04-03T17:20:30.0762218Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_dist_autograd_profiling 2024-04-03T17:20:30.0763712Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_grad_only_on_return_value_remote 2024-04-03T17:20:30.0765233Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_py_nested_call 2024-04-03T17:20:30.0766729Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_python_remote_call 2024-04-03T17:20:30.0768170Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_nested_context 2024-04-03T17:20:30.0769528Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_rpc_complex_args 2024-04-03T17:20:30.0770945Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistOptimizerTest::test_dist_optim_none_grads 2024-04-03T17:20:30.0772322Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_add_done_callback 2024-04-03T17:20:30.0773757Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_args_and_kwargs_contain_different_types 2024-04-03T17:20:30.0775196Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_function_remote 2024-04-03T17:20:30.0776601Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_function_remote_multi 2024-04-03T17:20:30.0778039Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_call_fork_in_jit_with_profiling 2024-04-03T17:20:30.0779621Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_call_script_function_that_not_exists_remotely_from_script 2024-04-03T17:20:30.0781109Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_callback_chain 2024-04-03T17:20:30.0782438Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_local_rref_local_value 2024-04-03T17:20:30.0783910Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_record_function_jit_end_callbacks_with_fork 2024-04-03T17:20:30.0785376Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_remote_script_module 2024-04-03T17:20:30.0786705Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_remote_script_throw 2024-04-03T17:20:30.0788058Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_rref_as_arg_and_return 2024-04-03T17:20:30.0789500Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_torchscript_functions_not_supported 2024-04-03T17:20:30.0791096Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_unexepected_kwarg_is_specified 2024-04-03T17:20:30.0792625Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitDistAutogradTest::test_dist_backward 2024-04-03T17:20:30.0794014Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitDistAutogradTest::test_get_gradients 2024-04-03T17:20:30.0795528Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitDistAutogradTest::test_jit_fork_within_context 2024-04-03T17:20:30.0797108Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitDistAutogradTest::test_restore_context_after_swtich_to_jit_thread 2024-04-03T17:20:30.0798659Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_forward_sync_script 2024-04-03T17:20:30.0800388Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire 2024-04-03T17:20:30.0801994Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_train_eval 2024-04-03T17:20:30.0803526Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeThreeWorkersRemoteModuleTest::test_create_remote_module_from_module_rref 2024-04-03T17:20:30.0805272Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeThreeWorkersRemoteModuleTest::test_send_remote_module_over_the_wire 2024-04-03T17:20:30.0807082Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeThreeWorkersRemoteModuleTest::test_send_remote_module_over_the_wire_script_not_supported 2024-04-03T17:20:30.0808824Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDdpUnderDistAutogradTest::test_backward_ddp_inside 2024-04-03T17:20:30.0810329Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDdpUnderDistAutogradTest::test_backward_no_ddp 2024-04-03T17:20:30.0811863Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDdpComparisonTest::test_ddp_dist_autograd_local_vs_remote 2024-04-03T17:20:30.0813437Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_builtin_remote_ret_sparse 2024-04-03T17:20:30.0814978Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_dynamic_rpc_init_rpc 2024-04-03T17:20:30.0816537Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_dynamic_rpc_init_rpc_without_rank 2024-04-03T17:20:30.0818144Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_multi_py_udf_remote_sparse 2024-04-03T17:20:30.0819718Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_my_parameter_server_sparse 2024-04-03T17:20:30.0821262Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_nested_remote_sparse 2024-04-03T17:20:30.0822787Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_nested_rref_stress_sparse 2024-04-03T17:20:30.0824342Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_py_rpc_rref_args_sparse 2024-04-03T17:20:30.0825863Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_py_rref_args_sparse 2024-04-03T17:20:30.0827420Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_rref_get_type_timeout_blocking 2024-04-03T17:20:30.0829039Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_rref_proxy_timeout 2024-04-03T17:20:30.0830714Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_self_py_udf_remote_sparse 2024-04-03T17:20:30.0832458Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_self_remote_rref_as_self_remote_arg_sparse 2024-04-03T17:20:30.0834127Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_set_and_get_num_worker_threads 2024-04-03T17:20:30.0835796Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_no_grad_on_tensor_sparse 2024-04-03T17:20:30.0837551Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_rref_multi_sparse 2024-04-03T17:20:30.0839243Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_simple_self_sparse 2024-04-03T17:20:30.0841004Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_simple_sparse 2024-04-03T17:20:30.0842754Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_context_cleanup_tensor_no_grad_sparse 2024-04-03T17:20:30.0844523Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_graph_for_builtin_call_sparse 2024-04-03T17:20:30.0846279Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_graph_for_py_nested_call_itself_sparse 2024-04-03T17:20:30.0848033Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_multiple_backward_sparse 2024-04-03T17:20:30.0849794Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_nested_backward_accumulate_grads_sparse 2024-04-03T17:20:30.0851650Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_no_graph_with_tensors_not_require_grad_sparse 2024-04-03T17:20:30.0853386Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_trainer_ps_sparse 2024-04-03T17:20:30.0854275Z 2024-04-03T17:20:30.0854796Z Running distributed/pipeline/sync/test_copy 1/1 ... [2024-04-03 17:20:30.058734] 2024-04-03T17:20:30.0856769Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/test_copy.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:20:30.059191] 2024-04-03T17:20:31.7302060Z 2024-04-03T17:20:31.7304059Z distributed/pipeline/sync/test_copy 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_copy_1.1_cee55925a554a231_.log 2024-04-03T17:20:31.7307477Z Running 5 items in this shard: test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda, test/distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors 2024-04-03T17:20:31.7309873Z 2024-04-03T17:20:31.7310331Z Running distributed/fsdp/test_fsdp_mixed_precision 1/1 ... [2024-04-03 17:20:31.730601] 2024-04-03T17:20:31.7313541Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:20:31.731030] 2024-04-03T17:22:45.3717238Z 2024-04-03T17:22:45.3720189Z distributed/fsdp/test_fsdp_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_mixed_precision_1.1_93c7c9bfb0078674_.log 2024-04-03T17:22:45.3768364Z Running 57 items in this shard: test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_eval_root_cast_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval_buffers, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval_comm, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_grads_reduced_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_input_grads_with_param_mixed_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_no_reshard_after_forward, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_resnet, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_batchnorm_convert_sync_bn_False, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_batchnorm_convert_sync_bn_True, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_default, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_only_params_and_bufs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_params_and_reduce_diff, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_reduce, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_grads_reduced_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_mixed_precision_e2e_full_shard, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_mixed_precision_no_reshard_after_forward, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionIgnoredModules::test_mixed_precision_with_ignored_module, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule_skip_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule_skip_inputs_error, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_different_precisions, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_different_precisions_error, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_external_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPTrainEval::test_train_ema_eval_flow 2024-04-03T17:22:45.3812972Z 2024-04-03T17:22:45.3813548Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 ... [2024-04-03 17:22:45.372373] 2024-04-03T17:22:45.3815696Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:22:45.372805] 2024-04-03T17:22:47.6958745Z 2024-04-03T17:22:47.6961484Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_binary_cmp_1.1_be12aeec5db0d753_.log 2024-04-03T17:22:47.6965931Z Running 4 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_allclose, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_allclose_tensor_specs, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_equal, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_equal_tensor_specs 2024-04-03T17:22:47.6968661Z 2024-04-03T17:22:47.6969118Z Running distributed/test_pg_wrapper 1/1 ... [2024-04-03 17:22:47.696328] 2024-04-03T17:22:47.6971403Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_pg_wrapper.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:22:47.696758] 2024-04-03T17:23:51.8025206Z 2024-04-03T17:23:51.8030737Z distributed/test_pg_wrapper 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_pg_wrapper_1.1_9e48b0ebd9c2cab4_.log 2024-04-03T17:23:51.8041531Z Running 15 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_coalescing_manager_debug_mode_detail, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_hang, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_detail, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_off, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_hang, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode_off, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_debug_mode 2024-04-03T17:23:51.8051343Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_coalescing_manager_debug_mode_detail 2024-04-03T17:23:51.8052768Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_hang 2024-04-03T17:23:51.8054201Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_detail 2024-04-03T17:23:51.8055751Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_off 2024-04-03T17:23:51.8057191Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch 2024-04-03T17:23:51.8058617Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch_debug_mode 2024-04-03T17:23:51.8059994Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_hang 2024-04-03T17:23:51.8061352Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda 2024-04-03T17:23:51.8064803Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda_debug_mode 2024-04-03T17:23:51.8066333Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode 2024-04-03T17:23:51.8067842Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode_off 2024-04-03T17:23:51.8069303Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch 2024-04-03T17:23:51.8070868Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda 2024-04-03T17:23:51.8072343Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda_debug_mode 2024-04-03T17:23:51.8073851Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_debug_mode 2024-04-03T17:23:51.8074664Z 2024-04-03T17:23:51.8075095Z Running distributed/test_c10d_common 1/1 ... [2024-04-03 17:23:51.803493] 2024-04-03T17:23:51.8077010Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_common.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:23:51.803931] 2024-04-03T17:25:40.1571408Z 2024-04-03T17:25:40.1573666Z distributed/test_c10d_common 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_common_1.1_83a4b6dc0c98832d_.log 2024-04-03T17:25:40.1601181Z Running 19 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype, test/distributed/test_c10d_common.py::CommTest::test_debug_level, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle 2024-04-03T17:25:40.1611850Z Running 1 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier 2024-04-03T17:25:40.1613109Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype 2024-04-03T17:25:40.1614489Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype 2024-04-03T17:25:40.1615856Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype 2024-04-03T17:25:40.1617211Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype 2024-04-03T17:25:40.1618403Z Running 1 items in this shard: test/distributed/test_c10d_common.py::CommTest::test_debug_level 2024-04-03T17:25:40.1619591Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr 2024-04-03T17:25:40.1620940Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config 2024-04-03T17:25:40.1622241Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives 2024-04-03T17:25:40.1623563Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name 2024-04-03T17:25:40.1625031Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends 2024-04-03T17:25:40.1626516Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available 2024-04-03T17:25:40.1627821Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv 2024-04-03T17:25:40.1629307Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-04-03T17:25:40.1631222Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-04-03T17:25:40.1632659Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop 2024-04-03T17:25:40.1633808Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable 2024-04-03T17:25:40.1634908Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal 2024-04-03T17:25:40.1635993Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle 2024-04-03T17:25:40.1636620Z 2024-04-03T17:25:40.1637252Z Running distributed/pipeline/sync/skip/test_tracker 1/1 ... [2024-04-03 17:25:40.158077] 2024-04-03T17:25:40.1639323Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:25:40.158481] 2024-04-03T17:25:41.7800562Z 2024-04-03T17:25:41.7802407Z distributed/pipeline/sync/skip/test_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_tracker_1.1_f437e5b526a84669_.log 2024-04-03T17:25:41.7807155Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker, test/distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel, test/distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal, test/distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal, test/distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing, test/distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing 2024-04-03T17:25:41.7810625Z 2024-04-03T17:25:41.7811280Z Running distributed/pipeline/sync/skip/test_leak 1/1 ... [2024-04-03 17:25:41.780524] 2024-04-03T17:25:41.7813348Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_leak.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:25:41.780955] 2024-04-03T17:25:43.9032457Z 2024-04-03T17:25:43.9035118Z distributed/pipeline/sync/skip/test_leak 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_leak_1.1_2e724b94d551734d_.log 2024-04-03T17:25:43.9041765Z Running 8 items in this shard: test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] 2024-04-03T17:25:43.9046469Z 2024-04-03T17:25:43.9046977Z Running distributed/pipeline/sync/skip/test_stash_pop 1/1 ... [2024-04-03 17:25:43.903661] 2024-04-03T17:25:43.9049064Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_stash_pop.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:25:43.904079] 2024-04-03T17:25:45.5252351Z 2024-04-03T17:25:45.5254442Z distributed/pipeline/sync/skip/test_stash_pop 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_stash_pop_1.1_07bd87ac4f9d6309_.log 2024-04-03T17:25:45.5259337Z Running 7 items in this shard: test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none 2024-04-03T17:25:45.5263436Z 2024-04-03T17:25:45.5263838Z Running distributed/test_c10d_spawn_gloo 1/1 ... [2024-04-03 17:25:45.525685] 2024-04-03T17:25:45.5265814Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:25:45.526089] 2024-04-03T17:26:46.8770349Z 2024-04-03T17:26:46.8773928Z distributed/test_c10d_spawn_gloo 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_gloo_1.1_523842e9c3091119_.log 2024-04-03T17:26:46.8784555Z Running 15 items in this shard: test/distributed/test_c10d_spawn_gloo.py::ProcessGroupShareTensorTest::test_shared_allgather_chunk_gloo, test/distributed/test_c10d_spawn_gloo.py::ProcessGroupShareTensorTest::test_shared_allgather_gloo, test/distributed/test_c10d_spawn_gloo.py::ProcessGroupShareTensorTest::test_shared_allreduce_gloo, test/distributed/test_c10d_spawn_gloo.py::ProcessGroupShareTensorTest::test_shared_broadcast_gloo, test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cpu, test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cuda, test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_rnn, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all_single, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_allreduce, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_broadcast, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_gather, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_reduce, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_scatter 2024-04-03T17:26:46.8793551Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::ProcessGroupShareTensorTest::test_shared_allgather_chunk_gloo 2024-04-03T17:26:46.8794984Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::ProcessGroupShareTensorTest::test_shared_allgather_gloo 2024-04-03T17:26:46.8796376Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::ProcessGroupShareTensorTest::test_shared_allreduce_gloo 2024-04-03T17:26:46.8797747Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::ProcessGroupShareTensorTest::test_shared_broadcast_gloo 2024-04-03T17:26:46.8799126Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cpu 2024-04-03T17:26:46.8800587Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cuda 2024-04-03T17:26:46.8801989Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_rnn 2024-04-03T17:26:46.8803317Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather 2024-04-03T17:26:46.8804620Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all 2024-04-03T17:26:46.8805957Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all_single 2024-04-03T17:26:46.8807283Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_allreduce 2024-04-03T17:26:46.8808568Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_broadcast 2024-04-03T17:26:46.8809836Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_gather 2024-04-03T17:26:46.8811312Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_reduce 2024-04-03T17:26:46.8812565Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_scatter 2024-04-03T17:26:46.8813287Z 2024-04-03T17:26:46.8813808Z Running distributed/test_c10d_ucc 1/1 ... [2024-04-03 17:26:46.877715] 2024-04-03T17:26:46.8815723Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_ucc.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:26:46.878150] 2024-04-03T17:26:48.0764261Z 2024-04-03T17:26:48.0765954Z distributed/test_c10d_ucc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_ucc_1.1_bd4ec77c32c58086_.log 2024-04-03T17:26:48.0766946Z 2024-04-03T17:26:48.0770588Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 ... [2024-04-03 17:26:48.076838] 2024-04-03T17:26:48.0775856Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:26:48.077286] 2024-04-03T17:26:50.3492306Z 2024-04-03T17:26:50.3494445Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_tensor_ops_1.1_fae1d865887ce4d2_.log 2024-04-03T17:26:50.3498284Z Running 5 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_clone, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_deep_copy, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_detach, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_inplace_copy, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_set_requires_grad 2024-04-03T17:26:50.3501136Z 2024-04-03T17:26:50.3501588Z Running distributed/pipeline/sync/test_stream 1/1 ... [2024-04-03 17:26:50.349786] 2024-04-03T17:26:50.3505311Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/test_stream.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:26:50.350213] 2024-04-03T17:26:51.9712584Z 2024-04-03T17:26:51.9715176Z distributed/pipeline/sync/test_stream 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_stream_1.1_948332f82422634d_.log 2024-04-03T17:26:51.9733254Z Running 19 items in this shard: test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view 2024-04-03T17:26:51.9747690Z 2024-04-03T17:26:51.9748338Z Running distributed/test_c10d_spawn_nccl 1/1 ... [2024-04-03 17:26:51.972017] 2024-04-03T17:26:51.9751561Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_spawn_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:26:51.972554] 2024-04-03T17:27:28.2682738Z 2024-04-03T17:27:28.2688037Z distributed/test_c10d_spawn_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_nccl_1.1_ae651aff4a3b4061_.log 2024-04-03T17:27:28.2696165Z Running 13 items in this shard: test/distributed/test_c10d_spawn_nccl.py::ProcessGroupShareTensorTest::test_shared_allgather_nccl, test/distributed/test_c10d_spawn_nccl.py::ProcessGroupShareTensorTest::test_shared_allreduce_nccl, test/distributed/test_c10d_spawn_nccl.py::ProcessGroupShareTensorTest::test_shared_broadcast_nccl, test/distributed/test_c10d_spawn_nccl.py::ProcessGroupShareTensorTest::test_shared_reduce_nccl, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather_base, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all_single, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_allreduce, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_broadcast, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter_non_contiguous 2024-04-03T17:27:28.2704004Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::ProcessGroupShareTensorTest::test_shared_allgather_nccl 2024-04-03T17:27:28.2705403Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::ProcessGroupShareTensorTest::test_shared_allreduce_nccl 2024-04-03T17:27:28.2706780Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::ProcessGroupShareTensorTest::test_shared_broadcast_nccl 2024-04-03T17:27:28.2708140Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::ProcessGroupShareTensorTest::test_shared_reduce_nccl 2024-04-03T17:27:28.2709470Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather 2024-04-03T17:27:28.2711668Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather_base 2024-04-03T17:27:28.2713006Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all 2024-04-03T17:27:28.2714340Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all_single 2024-04-03T17:27:28.2715676Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_allreduce 2024-04-03T17:27:28.2717251Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_broadcast 2024-04-03T17:27:28.2718523Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce 2024-04-03T17:27:28.2719816Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter 2024-04-03T17:27:28.2746147Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter_non_contiguous 2024-04-03T17:27:28.2747029Z 2024-04-03T17:27:28.2747502Z Running distributed/test_c10d_spawn_ucc 1/1 ... [2024-04-03 17:27:28.269193] 2024-04-03T17:27:28.2749471Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_spawn_ucc.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:27:28.269642] 2024-04-03T17:27:54.0809471Z 2024-04-03T17:27:54.0811468Z distributed/test_c10d_spawn_ucc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_ucc_1.1_2141fd603bdecac5_.log 2024-04-03T17:27:54.0825550Z Running 9 items in this shard: test/distributed/test_c10d_spawn_ucc.py::ProcessGroupShareTensorTest::test_shared_allgather_ucc, test/distributed/test_c10d_spawn_ucc.py::ProcessGroupShareTensorTest::test_shared_allreduce_ucc, test/distributed/test_c10d_spawn_ucc.py::ProcessGroupShareTensorTest::test_shared_broadcast_ucc, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_gather, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all_single, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_allreduce, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_broadcast, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_reduce 2024-04-03T17:27:54.0831074Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::ProcessGroupShareTensorTest::test_shared_allgather_ucc 2024-04-03T17:27:54.0832451Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::ProcessGroupShareTensorTest::test_shared_allreduce_ucc 2024-04-03T17:27:54.0833824Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::ProcessGroupShareTensorTest::test_shared_broadcast_ucc 2024-04-03T17:27:54.0835146Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_gather 2024-04-03T17:27:54.0836412Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all 2024-04-03T17:27:54.0837717Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all_single 2024-04-03T17:27:54.0839033Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_allreduce 2024-04-03T17:27:54.0840302Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_broadcast 2024-04-03T17:27:54.0841640Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_reduce 2024-04-03T17:27:54.0842352Z 2024-04-03T17:27:54.0842856Z Running distributed/pipeline/sync/test_balance 1/1 ... [2024-04-03 17:27:54.081665] 2024-04-03T17:27:54.0844862Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/test_balance.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:27:54.082104] 2024-04-03T17:27:59.8087820Z 2024-04-03T17:27:59.8089953Z distributed/pipeline/sync/test_balance 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_balance_1.1_e53cafbf15c3fd1f_.log 2024-04-03T17:27:59.8099431Z Running 15 items in this shard: test/distributed/pipeline/sync/test_balance.py::test_blockpartition, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale, test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu], test/distributed/pipeline/sync/test_balance.py::test_not_training, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple, test/distributed/pipeline/sync/test_balance.py::test_already_has_grad 2024-04-03T17:27:59.8106297Z 2024-04-03T17:27:59.8120685Z Running distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 ... [2024-04-03 17:27:59.809186] 2024-04-03T17:27:59.8123587Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:27:59.809589] 2024-04-03T17:28:02.1818202Z 2024-04-03T17:28:02.1820427Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_reshard_1.1_20eb55a5d8820481_.log 2024-04-03T17:28:02.1823223Z Running 2 items in this shard: test/distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py::TestReshard::test_sharded_tensor_reshard, test/distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py::TestReshard::test_sharded_tensor_reshard_errors 2024-04-03T17:28:02.1824898Z 2024-04-03T17:28:02.1825387Z Running distributed/pipeline/sync/test_dependency 1/1 ... [2024-04-03 17:28:02.182315] 2024-04-03T17:28:02.1831077Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/test_dependency.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:28:02.182758] 2024-04-03T17:28:03.8037046Z 2024-04-03T17:28:03.8038787Z distributed/pipeline/sync/test_dependency 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_dependency_1.1_c12249169cdcd565_.log 2024-04-03T17:28:03.8042830Z Running 6 items in this shard: test/distributed/pipeline/sync/test_dependency.py::test_fork_join, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_leak, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad 2024-04-03T17:28:03.8045699Z 2024-04-03T17:28:03.8046155Z Running distributed/pipeline/sync/skip/test_api 1/1 ... [2024-04-03 17:28:03.804170] 2024-04-03T17:28:03.8048978Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:28:03.804594] 2024-04-03T17:28:05.4256322Z 2024-04-03T17:28:05.4258017Z distributed/pipeline/sync/skip/test_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_api_1.1_0602854db6b362d4_.log 2024-04-03T17:28:05.4260819Z Running 3 items in this shard: test/distributed/pipeline/sync/skip/test_api.py::test_namespace_difference, test/distributed/pipeline/sync/skip/test_api.py::test_namespace_copy, test/distributed/pipeline/sync/skip/test_api.py::test_skippable_repr 2024-04-03T17:28:05.4262361Z 2024-04-03T17:28:05.4263006Z Running distributed/pipeline/sync/test_deferred_batch_norm 1/1 ... [2024-04-03 17:28:05.426045] 2024-04-03T17:28:05.4267742Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/test_deferred_batch_norm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:28:05.426450] 2024-04-03T17:28:08.4505424Z 2024-04-03T17:28:08.4507559Z distributed/pipeline/sync/test_deferred_batch_norm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_deferred_batch_norm_1.1_f8598ef2576e2b60_.log 2024-04-03T17:28:08.4515834Z Running 11 items in this shard: test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad 2024-04-03T17:28:08.4521579Z 2024-04-03T17:28:08.4522048Z Running distributed/checkpoint/test_state_dict 1/1 ... [2024-04-03 17:28:08.451020] 2024-04-03T17:28:08.4524038Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:28:08.451461] 2024-04-03T17:28:39.7213832Z 2024-04-03T17:28:39.7215853Z distributed/checkpoint/test_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_state_dict_1.1_2c650f09024bb5e6_.log 2024-04-03T17:28:39.7227998Z Running 11 items in this shard: test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_activation_ckpt_fqns, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_compiled_fsdp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_cpu_offload_full_state_dict, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_ddp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_frozen_parameters, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_fsdp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_fsdp2, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_fsdp_ddp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_partial, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_single_gpu, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_strict 2024-04-03T17:28:39.7236968Z 2024-04-03T17:28:39.7237612Z Running distributed/test_c10d_gloo 1/1 ... [2024-04-03 17:28:39.721855] 2024-04-03T17:28:39.7241658Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:28:39.722281] 2024-04-03T17:39:41.7267732Z 2024-04-03T17:39:41.7269724Z distributed/test_c10d_gloo 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_gloo_1.1_c69adfe41352467b_.log 2024-04-03T17:39:41.7347279Z Running 133 items in this shard: test/distributed/test_c10d_gloo.py::RendezvousEnvTest::test_logging_init, test/distributed/test_c10d_gloo.py::TimeoutTest::test_default_store_timeout_gloo, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_into_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_barrier_implies_wait, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_empty_tensors, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_multi_device_constructor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_round_robin, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_round_robin_create_destroy, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_send_recv_all_to_all, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_set_gloo_pg_timeout, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_cuda_dispatched, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_gloo, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_2gpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_4gpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_sharded_tensor, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_save_load_checkpoint, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_optimizer, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_unused_parameters, test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_multi_bucket, test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_single_bucket, test/distributed/test_c10d_gloo.py::ReducerTest::test_single_dtype_single_bucket, test/distributed/test_c10d_gloo.py::CommTest::test_bool_tensors, test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cpu, test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cuda, test/distributed/test_c10d_gloo.py::CommTest::test_gloo_rank_membership, test/distributed/test_c10d_gloo.py::CommTest::test_gloo_warn_not_in_group, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_default, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_subgroup, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_default_pg_gloo, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_gloo_new_group, test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allgather_coalesced, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_monitored_barrier, test/distributed/test_c10d_gloo.py::CompilerTest::test_allgather_work_wait_cpu, test/distributed/test_c10d_gloo.py::CompilerTest::test_allgather_work_wait_gpu, test/distributed/test_c10d_gloo.py::CompilerTest::test_allreduce_work_wait_cpu, test/distributed/test_c10d_gloo.py::CompilerTest::test_allreduce_work_wait_gpu, test/distributed/test_c10d_gloo.py::CompilerTest::test_broadcast_work_wait_cpu, test/distributed/test_c10d_gloo.py::CompilerTest::test_broadcast_work_wait_gpu, test/distributed/test_c10d_gloo.py::CompilerTest::test_consecutive_comm_work_wait_cpu, test/distributed/test_c10d_gloo.py::CompilerTest::test_consecutive_comm_work_wait_gpu, test/distributed/test_c10d_gloo.py::CompilerTest::test_nested_comm_tensor_wrapping, test/distributed/test_c10d_gloo.py::CompilerTest::test_scatter_work_wait_cpu, test/distributed/test_c10d_gloo.py::CompilerTest::test_scatter_work_wait_gpu, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_duplicate_pg, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_sanity_check 2024-04-03T17:39:41.7412993Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::RendezvousEnvTest::test_logging_init 2024-04-03T17:39:41.7414136Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::TimeoutTest::test_default_store_timeout_gloo 2024-04-03T17:39:41.7415322Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics 2024-04-03T17:39:41.7416540Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics_cuda 2024-04-03T17:39:41.7417736Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_checks 2024-04-03T17:39:41.7418967Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_async 2024-04-03T17:39:41.7420250Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_checks 2024-04-03T17:39:41.7421567Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_into_tensor_coalesced 2024-04-03T17:39:41.7422874Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_noncontiguous_input 2024-04-03T17:39:41.7424113Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress 2024-04-03T17:39:41.7425314Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress_cuda 2024-04-03T17:39:41.7426509Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics 2024-04-03T17:39:41.7427696Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics_cuda 2024-04-03T17:39:41.7428894Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_checks 2024-04-03T17:39:41.7430199Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_async 2024-04-03T17:39:41.7431463Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_basics 2024-04-03T17:39:41.7432747Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks 2024-04-03T17:39:41.7434073Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks_cuda 2024-04-03T17:39:41.7435379Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_stress 2024-04-03T17:39:41.7436597Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress 2024-04-03T17:39:41.7437790Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress_cuda 2024-04-03T17:39:41.7439013Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_barrier_implies_wait 2024-04-03T17:39:41.7440207Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics 2024-04-03T17:39:41.7441457Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics_cuda 2024-04-03T17:39:41.7442815Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_checks 2024-04-03T17:39:41.7443992Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress 2024-04-03T17:39:41.7445277Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress_cuda 2024-04-03T17:39:41.7446464Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_empty_tensors 2024-04-03T17:39:41.7447609Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics 2024-04-03T17:39:41.7448775Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics_cuda 2024-04-03T17:39:41.7449919Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_checks 2024-04-03T17:39:41.7451137Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_noncontiguous_input 2024-04-03T17:39:41.7452348Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress 2024-04-03T17:39:41.7453521Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress_cuda 2024-04-03T17:39:41.7454739Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_multi_device_constructor 2024-04-03T17:39:41.7455937Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics 2024-04-03T17:39:41.7457103Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics_cuda 2024-04-03T17:39:41.7458258Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_checks 2024-04-03T17:39:41.7459443Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor 2024-04-03T17:39:41.7460724Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor_coalesced 2024-04-03T17:39:41.7461951Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress 2024-04-03T17:39:41.7463106Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress_cuda 2024-04-03T17:39:41.7464259Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_round_robin 2024-04-03T17:39:41.7465460Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_round_robin_create_destroy 2024-04-03T17:39:41.7466654Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics 2024-04-03T17:39:41.7467813Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics_cuda 2024-04-03T17:39:41.7468993Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_checks 2024-04-03T17:39:41.7470251Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress 2024-04-03T17:39:41.7471422Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress_cuda 2024-04-03T17:39:41.7472627Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_send_recv_all_to_all 2024-04-03T17:39:41.7473838Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_set_gloo_pg_timeout 2024-04-03T17:39:41.7475071Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics 2024-04-03T17:39:41.7476333Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics_cuda 2024-04-03T17:39:41.7477707Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_checks 2024-04-03T17:39:41.7479006Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_cuda_dispatched 2024-04-03T17:39:41.7480458Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output 2024-04-03T17:39:41.7481787Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output_unused_param 2024-04-03T17:39:41.7483211Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2024-04-03T17:39:41.7484695Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2024-04-03T17:39:41.7486216Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2024-04-03T17:39:41.7487743Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2024-04-03T17:39:41.7489343Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False 2024-04-03T17:39:41.7491016Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2024-04-03T17:39:41.7492600Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2024-04-03T17:39:41.7494136Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2024-04-03T17:39:41.7495657Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing 2024-04-03T17:39:41.7497218Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False 2024-04-03T17:39:41.7498821Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2024-04-03T17:39:41.7500445Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False 2024-04-03T17:39:41.7502074Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2024-04-03T17:39:41.7503594Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu 2024-04-03T17:39:41.7505036Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_gloo 2024-04-03T17:39:41.7506489Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once 2024-04-03T17:39:41.7507915Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients 2024-04-03T17:39:41.7509463Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init 2024-04-03T17:39:41.7510962Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type 2024-04-03T17:39:41.7512494Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty 2024-04-03T17:39:41.7514109Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad 2024-04-03T17:39:41.7515621Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view 2024-04-03T17:39:41.7517273Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph 2024-04-03T17:39:41.7518860Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_integer_list 2024-04-03T17:39:41.7520539Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_torch_device_list 2024-04-03T17:39:41.7522037Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_2gpu_module 2024-04-03T17:39:41.7523374Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_4gpu_module 2024-04-03T17:39:41.7524720Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module 2024-04-03T17:39:41.7526126Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module_grad_is_view 2024-04-03T17:39:41.7527478Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output 2024-04-03T17:39:41.7528834Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters 2024-04-03T17:39:41.7530238Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_sharded_tensor 2024-04-03T17:39:41.7531573Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_invalid_powerSGD_state 2024-04-03T17:39:41.7532892Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_save_load_checkpoint 2024-04-03T17:39:41.7534162Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients 2024-04-03T17:39:41.7535502Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view 2024-04-03T17:39:41.7536902Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2024-04-03T17:39:41.7538298Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input 2024-04-03T17:39:41.7539531Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward 2024-04-03T17:39:41.7540659Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_optimizer 2024-04-03T17:39:41.7541868Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_unused_parameters 2024-04-03T17:39:41.7543057Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_multi_bucket 2024-04-03T17:39:41.7544214Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_single_bucket 2024-04-03T17:39:41.7545368Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_single_dtype_single_bucket 2024-04-03T17:39:41.7546441Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_bool_tensors 2024-04-03T17:39:41.7547496Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cpu 2024-04-03T17:39:41.7548650Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cuda 2024-04-03T17:39:41.7549976Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_gloo_rank_membership 2024-04-03T17:39:41.7551055Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_gloo_warn_not_in_group 2024-04-03T17:39:41.7552318Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_default 2024-04-03T17:39:41.7553562Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_subgroup 2024-04-03T17:39:41.7554783Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_default_pg_gloo 2024-04-03T17:39:41.7555954Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_gloo_new_group 2024-04-03T17:39:41.7557079Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_complex 2024-04-03T17:39:41.7558168Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_mismatch 2024-04-03T17:39:41.7559494Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single 2024-04-03T17:39:41.7561106Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allgather_coalesced 2024-04-03T17:39:41.7562678Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced 2024-04-03T17:39:41.7564209Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_collectives 2024-04-03T17:39:41.7565817Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-04-03T17:39:41.7567536Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-04-03T17:39:41.7569168Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_monitored_barrier 2024-04-03T17:39:41.7570534Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_allgather_work_wait_cpu 2024-04-03T17:39:41.7571669Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_allgather_work_wait_gpu 2024-04-03T17:39:41.7572818Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_allreduce_work_wait_cpu 2024-04-03T17:39:41.7573965Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_allreduce_work_wait_gpu 2024-04-03T17:39:41.7575112Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_broadcast_work_wait_cpu 2024-04-03T17:39:41.7576248Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_broadcast_work_wait_gpu 2024-04-03T17:39:41.7577433Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_consecutive_comm_work_wait_cpu 2024-04-03T17:39:41.7578663Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_consecutive_comm_work_wait_gpu 2024-04-03T17:39:41.7579860Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_nested_comm_tensor_wrapping 2024-04-03T17:39:41.7580997Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_scatter_work_wait_cpu 2024-04-03T17:39:41.7582124Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_scatter_work_wait_gpu 2024-04-03T17:39:41.7583251Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync 2024-04-03T17:39:41.7584515Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_duplicate_pg 2024-04-03T17:39:41.7585777Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_sanity_check 2024-04-03T17:39:41.7586498Z 2024-04-03T17:39:41.7586948Z Running distributed/test_c10d_nccl 1/1 ... [2024-04-03 17:39:41.730528] 2024-04-03T17:39:41.7588925Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:39:41.730920] 2024-04-03T17:39:42.9473278Z 2024-04-03T17:39:42.9474824Z distributed/test_c10d_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_1.1_5c67086c104f5154_.log 2024-04-03T17:39:42.9475924Z 2024-04-03T17:39:42.9479558Z Running distributed/elastic/utils/logging_test 1/1 ... [2024-04-03 17:39:42.947759] 2024-04-03T17:39:42.9484925Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:39:42.948217] 2024-04-03T17:39:45.5710168Z 2024-04-03T17:39:45.5712159Z distributed/elastic/utils/logging_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.logging_test_1.1_ff3e28ebe613a78a_.log 2024-04-03T17:39:45.5714362Z Running 2 items in this shard: test/distributed/elastic/utils/logging_test.py::LoggingTest::test_derive_module_name, test/distributed/elastic/utils/logging_test.py::LoggingTest::test_logger_name 2024-04-03T17:39:45.5715696Z 2024-04-03T17:39:45.5716944Z Running distributed/elastic/utils/util_test 1/1 ... [2024-04-03 17:39:45.571464] 2024-04-03T17:39:45.5722401Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/elastic/utils/util_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:39:45.571866] 2024-04-03T17:39:48.1943446Z 2024-04-03T17:39:48.1945679Z distributed/elastic/utils/util_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.util_test_1.1_823d7ecff9c83a88_.log 2024-04-03T17:39:48.1950621Z Running 7 items in this shard: test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_get_all_rank_0, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_get_all_rank_n, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_synchronize, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_custom_name, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_different, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_none 2024-04-03T17:39:48.1956101Z 2024-04-03T17:39:48.7069060Z 2024-04-03T17:39:48.7069533Z real 92m48.316s 2024-04-03T17:39:48.7070226Z user 175m21.299s 2024-04-03T17:39:48.7071477Z sys 33m9.453s 2024-04-03T17:39:48.7072001Z + assert_git_not_dirty 2024-04-03T17:39:48.7072881Z + [[ linux-jammy-py3.8-gcc11 != *rocm* ]] 2024-04-03T17:39:48.7073647Z + [[ linux-jammy-py3.8-gcc11 != *xla* ]] 2024-04-03T17:39:48.7074562Z ++ git status --porcelain 2024-04-03T17:39:48.7076349Z ++ grep -v '?? third_party' 2024-04-03T17:40:01.5462729Z ++ true 2024-04-03T17:40:01.5463506Z + git_status= 2024-04-03T17:40:01.5464009Z + [[ -n '' ]] 2024-04-03T17:40:01.5464504Z + [[ linux-jammy-py3.8-gcc11 == *cuda* ]] 2024-04-03T17:40:01.5465024Z + [[ linux-jammy-py3.8-gcc11 == *rocm* ]] 2024-04-03T17:40:01.5465414Z + [[ 2 == 1 ]] 2024-04-03T17:40:01.5532974Z ##[group]Run cat test/**/*_toprint.log || true 2024-04-03T17:40:01.5533612Z cat test/**/*_toprint.log || true 2024-04-03T17:40:01.5541098Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T17:40:01.5541597Z env: 2024-04-03T17:40:01.5541865Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:40:01.5542457Z DOCKER_CONTAINER_ID: 4c1cfaf4a8d048d787858cf2354817432c864f82c94c156c2f90005f40fddbdd 2024-04-03T17:40:01.5543110Z ##[endgroup] 2024-04-03T17:40:01.5606645Z cat: test/**/*_toprint.log: No such file or directory 2024-04-03T17:40:01.5636323Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-04-03T17:40:01.5636773Z kill "$MONITOR_SCRIPT_PID" 2024-04-03T17:40:01.5643893Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T17:40:01.5644377Z env: 2024-04-03T17:40:01.5644635Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:40:01.5645234Z DOCKER_CONTAINER_ID: 4c1cfaf4a8d048d787858cf2354817432c864f82c94c156c2f90005f40fddbdd 2024-04-03T17:40:01.5645891Z MONITOR_SCRIPT_PID: 24817 2024-04-03T17:40:01.5646236Z ##[endgroup] 2024-04-03T17:40:01.5826052Z Prepare all required actions 2024-04-03T17:40:01.5826500Z Getting action download info 2024-04-03T17:40:01.7400856Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-04-03T17:40:01.9009363Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-04-03T17:40:01.9009808Z with: 2024-04-03T17:40:01.9010150Z file-suffix: test-distributed-2-2-linux.2xlarge_ 2024-04-03T17:40:01.9010660Z s3-bucket: gha-artifacts 2024-04-03T17:40:01.9011196Z env: 2024-04-03T17:40:01.9011658Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:40:01.9012252Z DOCKER_CONTAINER_ID: 4c1cfaf4a8d048d787858cf2354817432c864f82c94c156c2f90005f40fddbdd 2024-04-03T17:40:01.9013155Z ##[endgroup] 2024-04-03T17:40:01.9038077Z ##[group]Run # Remove any previous test jsons if they exist 2024-04-03T17:40:01.9038692Z # Remove any previous test jsons if they exist 2024-04-03T17:40:01.9039192Z rm -f test-jsons-*.zip 2024-04-03T17:40:01.9039746Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-04-03T17:40:01.9047274Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T17:40:01.9047757Z env: 2024-04-03T17:40:01.9048005Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:40:01.9048583Z DOCKER_CONTAINER_ID: 4c1cfaf4a8d048d787858cf2354817432c864f82c94c156c2f90005f40fddbdd 2024-04-03T17:40:01.9049320Z FILE_SUFFIX: test-distributed-2-2-linux.2xlarge_ 2024-04-03T17:40:01.9049776Z ##[endgroup] 2024-04-03T17:40:01.9303432Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-04-03T17:40:01.9330783Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-04-03T17:40:01.9331617Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-04-03T17:40:01.9337487Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-04-03T17:40:01.9338418Z adding: test/.pytorch-slow-tests.json (deflated 59%) 2024-04-03T17:40:01.9345431Z adding: test/.pytorch-disabled-tests.json (deflated 84%) 2024-04-03T17:40:01.9372643Z ##[group]Run # Remove any previous test reports if they exist 2024-04-03T17:40:01.9373277Z # Remove any previous test reports if they exist 2024-04-03T17:40:01.9373796Z rm -f test-reports-*.zip 2024-04-03T17:40:01.9374372Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-04-03T17:40:01.9381802Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T17:40:01.9382275Z env: 2024-04-03T17:40:01.9382532Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:40:01.9383121Z DOCKER_CONTAINER_ID: 4c1cfaf4a8d048d787858cf2354817432c864f82c94c156c2f90005f40fddbdd 2024-04-03T17:40:01.9383842Z FILE_SUFFIX: test-distributed-2-2-linux.2xlarge_ 2024-04-03T17:40:01.9384301Z ##[endgroup] 2024-04-03T17:40:01.9645424Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_nested_dict/distributed.checkpoint.test_nested_dict-6983123085d762bb.xml (deflated 51%) 2024-04-03T17:40:01.9647420Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_logger/distributed._shard.sharded_tensor.test_logger-07b4d73166deccb4.xml (deflated 37%) 2024-04-03T17:40:01.9649527Z adding: test/test-reports/python-pytest/distributed._composable.test_contract/distributed._composable.test_contract-17e12ed23ee82abb.xml (deflated 65%) 2024-04-03T17:40:01.9651339Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_traverse/distributed.checkpoint.test_traverse-b8f74ce3437bea34.xml (deflated 75%) 2024-04-03T17:40:01.9653046Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_fx/distributed.fsdp.test_fsdp_fx-249be565827395ea.xml (deflated 37%) 2024-04-03T17:40:01.9654653Z adding: test/test-reports/python-pytest/distributed._tensor.test_init/distributed._tensor.test_init-e94470b2c7be09b3.xml (deflated 75%) 2024-04-03T17:40:01.9656644Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_state/distributed._composable.fsdp.test_fully_shard_state-c669d79557ad0b14.xml (deflated 81%) 2024-04-03T17:40:01.9658660Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dedup_tensors/distributed.checkpoint.test_dedup_tensors-a5c78e26f49c94eb.xml (deflated 38%) 2024-04-03T17:40:01.9660583Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_compatibility/distributed.checkpoint.test_compatibility-374d8abee418cdd1.xml (deflated 65%) 2024-04-03T17:40:01.9662517Z adding: test/test-reports/python-pytest/distributed.checkpoint.fsdp.test_fsdp_dsd/distributed.checkpoint.fsdp.test_fsdp_dsd-b19811d02287cf7a.xml (deflated 54%) 2024-04-03T17:40:01.9664358Z adding: test/test-reports/python-pytest/distributed._tensor.test_common_rules/distributed._tensor.test_common_rules-9bfcee4f0da829a5.xml (deflated 80%) 2024-04-03T17:40:01.9666162Z adding: test/test-reports/python-pytest/distributed._composable.test_checkpoint/distributed._composable.test_checkpoint-b5ae35079d4bbfde.xml (deflated 67%) 2024-04-03T17:40:01.9668018Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_compile/distributed._tensor.test_dtensor_compile-0f86aaace7b58b97.xml (deflated 90%) 2024-04-03T17:40:01.9670051Z adding: test/test-reports/python-pytest/distributed.test_c10d_functional_native/distributed.test_c10d_functional_native-260752971a5145b2.xml (deflated 93%) 2024-04-03T17:40:01.9671908Z adding: test/test-reports/python-pytest/distributed.checkpoint.e2e.test_fsdp_ep/distributed.checkpoint.e2e.test_fsdp_ep-34851339bbda2dba.xml (deflated 34%) 2024-04-03T17:40:01.9673808Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_save_load_api/distributed.checkpoint.test_save_load_api-dada8aac8fba059c.xml (deflated 44%) 2024-04-03T17:40:01.9675649Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_ops/distributed._tensor.test_dtensor_ops-43ad67ab82aab747.xml (deflated 96%) 2024-04-03T17:40:01.9677592Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_overlap/distributed._composable.fsdp.test_fully_shard_overlap-5bbcaaa331fd2f1e.xml (deflated 38%) 2024-04-03T17:40:01.9679551Z adding: test/test-reports/python-pytest/distributed._tensor.test_redistribute/distributed._tensor.test_redistribute-fda9a2dfc9251569.xml (deflated 80%) 2024-04-03T17:40:01.9681574Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_random_state/distributed.tensor.parallel.test_tp_random_state-ce23bcff33e04b4d.xml (deflated 46%) 2024-04-03T17:40:01.9683459Z adding: test/test-reports/python-pytest/distributed._tensor.test_math_ops/distributed._tensor.test_math_ops-ddb7004a6d569866.xml (deflated 77%) 2024-04-03T17:40:01.9685276Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_hsdp_checkpoint/distributed.checkpoint.test_hsdp_checkpoint-f9fece85d894a7f2.xml (deflated 79%) 2024-04-03T17:40:01.9687262Z adding: test/test-reports/python-pytest/distributed.checkpoint.e2e.test_fine_tuning/distributed.checkpoint.e2e.test_fine_tuning-459bf573e7dfbacf.xml (deflated 38%) 2024-04-03T17:40:01.9689220Z adding: test/test-reports/python-pytest/distributed._tensor.test_matrix_ops/distributed._tensor.test_matrix_ops-98d2b210a1a4a3fe.xml (deflated 78%) 2024-04-03T17:40:01.9691231Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_compile/distributed._composable.fully_shard.test_fully_shard_compile-4ed3d0a339de9dd7.xml (deflated 49%) 2024-04-03T17:40:01.9693348Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_model_state/distributed.checkpoint.test_fsdp_model_state-f3b973d3c860a080.xml (deflated 65%) 2024-04-03T17:40:01.9695301Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_state_dict_utils/distributed.checkpoint.test_state_dict_utils-0688d89e85ada232.xml (deflated 71%) 2024-04-03T17:40:01.9697355Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_fsdp_2d_parallel/distributed.tensor.parallel.test_fsdp_2d_parallel-2e8be16d5d66fa65.xml (deflated 87%) 2024-04-03T17:40:01.9699425Z adding: test/test-reports/python-pytest/distributed._tensor.debug.test_comm_mode/distributed._tensor.debug.test_comm_mode-84f63387b1a9a293.xml (deflated 53%) 2024-04-03T17:40:01.9701358Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_examples/distributed.tensor.parallel.test_tp_examples-2fe608d838af89bf.xml (deflated 78%) 2024-04-03T17:40:01.9703106Z adding: test/test-reports/python-pytest/distributed.test_c10d_pypg/distributed.test_c10d_pypg-7dafe86c0fcc68d9.xml (deflated 94%) 2024-04-03T17:40:01.9704957Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_frozen/distributed._composable.fsdp.test_fully_shard_frozen-83232ea82875c7ca.xml (deflated 62%) 2024-04-03T17:40:01.9707107Z adding: test/test-reports/python-pytest/distributed.fsdp.test_distributed_checkpoint/distributed.fsdp.test_distributed_checkpoint-4716112591ab7925.xml (deflated 77%) 2024-04-03T17:40:01.9708985Z adding: test/test-reports/python-pytest/distributed._tensor.test_utils/distributed._tensor.test_utils-62e55ce31f63af79.xml (deflated 63%) 2024-04-03T17:40:01.9711198Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_comm/distributed._composable.fsdp.test_fully_shard_comm-cdd47371231372ed.xml (deflated 69%) 2024-04-03T17:40:01.9713353Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_flatten_params/distributed.fsdp.test_fsdp_flatten_params-15e2d3aa09a06f39.xml (deflated 84%) 2024-04-03T17:40:01.9715725Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_model_checkpoint/distributed._composable.fully_shard.test_fully_shard_model_checkpoint-e4467152ae19186e.xml (deflated 80%) 2024-04-03T17:40:01.9718041Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_utils/distributed.checkpoint.test_utils-195d1e81c061e35e.xml (deflated 69%) 2024-04-03T17:40:01.9719761Z adding: test/test-reports/python-pytest/distributed._tensor.test_view_ops/distributed._tensor.test_view_ops-2c34a758ed0d2bfa.xml (deflated 59%) 2024-04-03T17:40:01.9721784Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_init/distributed._composable.fully_shard.test_fully_shard_init-36ce88b9f79ef7d6.xml (deflated 84%) 2024-04-03T17:40:01.9723753Z adding: test/test-reports/python-pytest/distributed.test_functional_api/distributed.test_functional_api-a18d57e3f5b79ea7.xml (deflated 92%) 2024-04-03T17:40:01.9725546Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_ignored_modules/distributed.fsdp.test_fsdp_ignored_modules-a31050f493174c42.xml (deflated 86%) 2024-04-03T17:40:01.9727445Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_tp_integration/distributed.fsdp.test_fsdp_tp_integration-e0476cf8fc576173.xml (deflated 84%) 2024-04-03T17:40:01.9729320Z adding: test/test-reports/python-pytest/distributed.fsdp.test_checkpoint_wrapper/distributed.fsdp.test_checkpoint_wrapper-49e92a003b21b3a6.xml (deflated 72%) 2024-04-03T17:40:01.9731245Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_grad_acc/distributed.fsdp.test_fsdp_grad_acc-fac94dc39805780d.xml (deflated 84%) 2024-04-03T17:40:01.9733019Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_planner/distributed.checkpoint.test_planner-25d9e3bfc1b5ab33.xml (deflated 62%) 2024-04-03T17:40:01.9734770Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_apply/distributed.fsdp.test_fsdp_apply-0d8e2065acc7b32b.xml (deflated 71%) 2024-04-03T17:40:01.9736622Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_sharded_grad_scaler/distributed.fsdp.test_fsdp_sharded_grad_scaler-bbdfdc42fa0b9e25.xml (deflated 93%) 2024-04-03T17:40:01.9738630Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_pipeline/distributed.pipeline.sync.test_pipeline-0d3fd45a28ed8153.xml (deflated 38%) 2024-04-03T17:40:01.9740740Z adding: test/test-reports/python-pytest/distributed._composable.test_replicate_with_compiler/distributed._composable.test_replicate_with_compiler-ee4788eadc4478b3.xml (deflated 84%) 2024-04-03T17:40:01.9742700Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_state_dict/distributed.fsdp.test_fsdp_state_dict-8c01fc7d0ae8a689.xml (deflated 97%) 2024-04-03T17:40:01.9744485Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-9a4362e23ef9a856.xml (deflated 46%) 2024-04-03T17:40:01.9746407Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-76c5543c814d993e.xml (deflated 64%) 2024-04-03T17:40:01.9748240Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-418f594e98236276.xml (deflated 44%) 2024-04-03T17:40:01.9750382Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_multiple_wrapping/distributed.fsdp.test_fsdp_multiple_wrapping-51bdb31d672c7af9.xml (deflated 48%) 2024-04-03T17:40:01.9752519Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_runtime/distributed._composable.fully_shard.test_fully_shard_runtime-9b2c67cefa0402e1.xml (deflated 54%) 2024-04-03T17:40:01.9754573Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-294f8646e612c60f.xml (deflated 77%) 2024-04-03T17:40:01.9756384Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_exec_order/distributed.fsdp.test_fsdp_exec_order-5ee947a729dec367.xml (deflated 87%) 2024-04-03T17:40:01.9758366Z adding: test/test-reports/python-pytest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks-e71d8f643a6ec7e4.xml (deflated 85%) 2024-04-03T17:40:01.9760205Z adding: test/test-reports/python-pytest/distributed.fsdp.test_wrap/distributed.fsdp.test_wrap-ff4c86ab7c29c514.xml (deflated 94%) 2024-04-03T17:40:01.9762182Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_file_system_checkpoint_cpu/distributed.checkpoint.test_file_system_checkpoint_cpu-4a3c36843b4cf6c1.xml (deflated 79%) 2024-04-03T17:40:01.9764146Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm_hooks/distributed.fsdp.test_fsdp_comm_hooks-12fd41b24e13d051.xml (deflated 94%) 2024-04-03T17:40:01.9765985Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_use_orig_params/distributed.fsdp.test_fsdp_use_orig_params-bb735fe651a05915.xml (deflated 90%) 2024-04-03T17:40:01.9767725Z adding: test/test-reports/python-pytest/distributed.fsdp.test_utils/distributed.fsdp.test_utils-c912099964425f1c.xml (deflated 71%) 2024-04-03T17:40:01.9769449Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_phony/distributed.pipeline.sync.test_phony-52074cae6cebac16.xml (deflated 65%) 2024-04-03T17:40:01.9771279Z adding: test/test-reports/python-pytest/distributed.test_c10d_object_collectives/distributed.test_c10d_object_collectives-e9f6b088111efd0f.xml (deflated 79%) 2024-04-03T17:40:01.9773294Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_multiple_forward/distributed.fsdp.test_fsdp_multiple_forward-d671f988c709cfb2.xml (deflated 47%) 2024-04-03T17:40:01.9775088Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-94442ea724aa4dc0.xml (deflated 88%) 2024-04-03T17:40:01.9776819Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_bugs/distributed.pipeline.sync.test_bugs-c552964880074821.xml (deflated 55%) 2024-04-03T17:40:01.9778679Z adding: test/test-reports/python-pytest/distributed.fsdp.test_hsdp_dtensor_state_dict/distributed.fsdp.test_hsdp_dtensor_state_dict-76c036c5b4636336.xml (deflated 86%) 2024-04-03T17:40:01.9780642Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_gpipe/distributed.pipeline.sync.skip.test_gpipe-da77814524c1ecda.xml (deflated 86%) 2024-04-03T17:40:01.9782578Z adding: test/test-reports/python-pytest/distributed.algorithms.test_join/distributed.algorithms.test_join-e06d00e532409365.xml (deflated 79%) 2024-04-03T17:40:01.9784506Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_embedding/distributed._shard.sharded_tensor.ops.test_embedding-59ff755632392d93.xml (deflated 68%) 2024-04-03T17:40:01.9786636Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_embedding_bag/distributed._shard.sharded_tensor.ops.test_embedding_bag-b767f29a2340792c.xml (deflated 68%) 2024-04-03T17:40:01.9788658Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_inplace/distributed.pipeline.sync.test_inplace-53b123ded4f840fe.xml (deflated 56%) 2024-04-03T17:40:01.9790736Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-be8df62085b7d145.xml (deflated 41%) 2024-04-03T17:40:01.9792676Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5f5964dc6e1e4a27.xml (deflated 42%) 2024-04-03T17:40:01.9794617Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-06615ddf7e89327e.xml (deflated 41%) 2024-04-03T17:40:01.9796521Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-321111edcdeb2afc.xml (deflated 42%) 2024-04-03T17:40:01.9798463Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-00d57f0cb4552584.xml (deflated 41%) 2024-04-03T17:40:01.9800458Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e182809c448c74b7.xml (deflated 41%) 2024-04-03T17:40:01.9802386Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e65e78fbb261475d.xml (deflated 41%) 2024-04-03T17:40:01.9804308Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-645a267814f6d5cb.xml (deflated 41%) 2024-04-03T17:40:01.9806226Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-134160a540b23b45.xml (deflated 43%) 2024-04-03T17:40:01.9808141Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-86dff7ba44641f00.xml (deflated 41%) 2024-04-03T17:40:01.9810055Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-981c2d679bd47bbc.xml (deflated 41%) 2024-04-03T17:40:01.9811952Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0e77293a8b41f1fa.xml (deflated 41%) 2024-04-03T17:40:01.9813999Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2637355ad9965e78.xml (deflated 41%) 2024-04-03T17:40:01.9815916Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-500dad3381820f7a.xml (deflated 42%) 2024-04-03T17:40:01.9817832Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-7367d3b0efda9b5e.xml (deflated 43%) 2024-04-03T17:40:01.9819729Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-36c209249c239830.xml (deflated 43%) 2024-04-03T17:40:01.9821641Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-650940f1437f6336.xml (deflated 42%) 2024-04-03T17:40:01.9823645Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-3edf9829f2e64c52.xml (deflated 42%) 2024-04-03T17:40:01.9825567Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-f03e4fc25439933b.xml (deflated 42%) 2024-04-03T17:40:01.9827486Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-72fa75b6788897fb.xml (deflated 42%) 2024-04-03T17:40:01.9829384Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-6fd69bef7a6d14ff.xml (deflated 42%) 2024-04-03T17:40:01.9831452Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b78fbde2924cc699.xml (deflated 43%) 2024-04-03T17:40:01.9833386Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e23c57f2ce299b9f.xml (deflated 43%) 2024-04-03T17:40:01.9835312Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-3453f4d37b590f85.xml (deflated 43%) 2024-04-03T17:40:01.9837217Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-77e5dc465461ea72.xml (deflated 42%) 2024-04-03T17:40:01.9839127Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-151a63ba3bfe1235.xml (deflated 42%) 2024-04-03T17:40:01.9841126Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c30f2e949b866ddf.xml (deflated 43%) 2024-04-03T17:40:01.9843055Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c34d991d3aa23c2b.xml (deflated 42%) 2024-04-03T17:40:01.9844965Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d41f46f1fbd775dc.xml (deflated 42%) 2024-04-03T17:40:01.9846892Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-7e0bf4df559f8ed9.xml (deflated 42%) 2024-04-03T17:40:01.9848805Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0c2175d1248b88b5.xml (deflated 42%) 2024-04-03T17:40:01.9850721Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-386e73bb230c53a8.xml (deflated 42%) 2024-04-03T17:40:01.9852613Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-690024208973aae7.xml (deflated 42%) 2024-04-03T17:40:01.9854643Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-cd2e4c92ff5e2df0.xml (deflated 42%) 2024-04-03T17:40:01.9856578Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2ba7d4c610bd82d9.xml (deflated 42%) 2024-04-03T17:40:01.9858502Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-6257eafc09977449.xml (deflated 42%) 2024-04-03T17:40:01.9860419Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0eace91b1bd50eef.xml (deflated 42%) 2024-04-03T17:40:01.9862319Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-f85677145989d517.xml (deflated 37%) 2024-04-03T17:40:01.9864314Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-46eda1f41be4e164.xml (deflated 42%) 2024-04-03T17:40:01.9866254Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1978c3dcd1896071.xml (deflated 41%) 2024-04-03T17:40:01.9868169Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-439adef560202094.xml (deflated 42%) 2024-04-03T17:40:01.9870217Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-3b71c9f3d0b64312.xml (deflated 43%) 2024-04-03T17:40:01.9872140Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ed33fe12ed48a750.xml (deflated 43%) 2024-04-03T17:40:01.9874059Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-30bce81ca87374d2.xml (deflated 42%) 2024-04-03T17:40:01.9875991Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-11b5ec9794b8f52a.xml (deflated 42%) 2024-04-03T17:40:01.9877903Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1d192e9678c32f8a.xml (deflated 42%) 2024-04-03T17:40:01.9879820Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-7111c47b6fcf0a05.xml (deflated 42%) 2024-04-03T17:40:01.9881818Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ede1d1f476b45359.xml (deflated 42%) 2024-04-03T17:40:01.9883748Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b435e3e6a4d1cf21.xml (deflated 42%) 2024-04-03T17:40:01.9885666Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-77c399083867571e.xml (deflated 42%) 2024-04-03T17:40:01.9887584Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b78d9663efe5bba5.xml (deflated 42%) 2024-04-03T17:40:01.9889519Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-865b23ae3e2f914f.xml (deflated 42%) 2024-04-03T17:40:01.9891436Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a2c526f579ec500d.xml (deflated 42%) 2024-04-03T17:40:01.9893336Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-222f729a1fd9c008.xml (deflated 42%) 2024-04-03T17:40:01.9895255Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ab78eef7993549f2.xml (deflated 42%) 2024-04-03T17:40:01.9897294Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-195d7af708accd2c.xml (deflated 42%) 2024-04-03T17:40:01.9899209Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5014b29cbd134f7a.xml (deflated 42%) 2024-04-03T17:40:01.9901127Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-7704a0be924176d6.xml (deflated 42%) 2024-04-03T17:40:01.9903029Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5c065409485ac010.xml (deflated 42%) 2024-04-03T17:40:01.9905033Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-af2d69a911cca344.xml (deflated 42%) 2024-04-03T17:40:01.9906965Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-49b201ee6ea87cb6.xml (deflated 42%) 2024-04-03T17:40:01.9908879Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-53bb88e3a3461338.xml (deflated 42%) 2024-04-03T17:40:01.9910920Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0f8630312aef21e6.xml (deflated 42%) 2024-04-03T17:40:01.9912832Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e0a8a64e500c839b.xml (deflated 42%) 2024-04-03T17:40:01.9914745Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-873e0bfcc5c159b5.xml (deflated 43%) 2024-04-03T17:40:01.9916672Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c2728611a6c84338.xml (deflated 42%) 2024-04-03T17:40:01.9918576Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a3ba40f104542523.xml (deflated 43%) 2024-04-03T17:40:01.9920535Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a128cc3d21d95a47.xml (deflated 42%) 2024-04-03T17:40:01.9922748Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-aa6a710d78a78210.xml (deflated 42%) 2024-04-03T17:40:01.9924668Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-242eff3d56cb771a.xml (deflated 42%) 2024-04-03T17:40:01.9926594Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a8d8d594a6c39e5b.xml (deflated 42%) 2024-04-03T17:40:01.9928499Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e839744467a18827.xml (deflated 42%) 2024-04-03T17:40:01.9930408Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4ec0a61d382d204d.xml (deflated 42%) 2024-04-03T17:40:01.9932327Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d2f7477a2c932015.xml (deflated 42%) 2024-04-03T17:40:01.9934231Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-30211f9d4aaa2ae4.xml (deflated 42%) 2024-04-03T17:40:01.9936134Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c0b9d637bd95b12f.xml (deflated 42%) 2024-04-03T17:40:01.9938155Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d963073d2d77c26d.xml (deflated 42%) 2024-04-03T17:40:01.9940067Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c311606cd6aa0a9b.xml (deflated 42%) 2024-04-03T17:40:01.9941985Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4211e78c92cb316a.xml (deflated 42%) 2024-04-03T17:40:01.9943878Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-446efbd3da15e185.xml (deflated 42%) 2024-04-03T17:40:01.9945782Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-7b761098e46162b6.xml (deflated 42%) 2024-04-03T17:40:01.9947794Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d7b82a088576f77b.xml (deflated 42%) 2024-04-03T17:40:01.9949866Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-6a0e5732636df608.xml (deflated 42%) 2024-04-03T17:40:01.9951783Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-46b5f3e589e7ffc8.xml (deflated 42%) 2024-04-03T17:40:01.9953896Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e5bc077089c059a6.xml (deflated 42%) 2024-04-03T17:40:01.9955992Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-60d07212a64c3ca1.xml (deflated 41%) 2024-04-03T17:40:01.9958007Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4b7882f4cd88d9fb.xml (deflated 41%) 2024-04-03T17:40:01.9960015Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-f2528aacc2170fa7.xml (deflated 41%) 2024-04-03T17:40:01.9962022Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b85230f84b114d3a.xml (deflated 41%) 2024-04-03T17:40:01.9964038Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-6467a040d3b14400.xml (deflated 42%) 2024-04-03T17:40:01.9965963Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a6b8425762e1b4c5.xml (deflated 42%) 2024-04-03T17:40:01.9967885Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-28d07e3b016e94b1.xml (deflated 42%) 2024-04-03T17:40:01.9969792Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-97637b342ef2736c.xml (deflated 42%) 2024-04-03T17:40:01.9971717Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5b44bca8dd7d5496.xml (deflated 42%) 2024-04-03T17:40:01.9973638Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-eb4e4139d11a34f7.xml (deflated 42%) 2024-04-03T17:40:01.9975545Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e56975c7679ccffe.xml (deflated 42%) 2024-04-03T17:40:01.9977446Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2e3fb1647c91c2b6.xml (deflated 41%) 2024-04-03T17:40:01.9979487Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1ba186c930fbcf26.xml (deflated 41%) 2024-04-03T17:40:01.9981400Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b19aadc8c7843323.xml (deflated 42%) 2024-04-03T17:40:01.9983321Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0e6c586198fc482f.xml (deflated 42%) 2024-04-03T17:40:01.9985229Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ff3e5d189802a22a.xml (deflated 42%) 2024-04-03T17:40:01.9987148Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-03765a8c004789f8.xml (deflated 42%) 2024-04-03T17:40:01.9989147Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-8a907b0abbd3ceff.xml (deflated 42%) 2024-04-03T17:40:01.9991158Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-611336e178337746.xml (deflated 42%) 2024-04-03T17:40:01.9993070Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-43efedc39af35208.xml (deflated 42%) 2024-04-03T17:40:01.9994977Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-812df3447515b215.xml (deflated 43%) 2024-04-03T17:40:01.9996887Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-39b1d925c4393032.xml (deflated 42%) 2024-04-03T17:40:01.9998734Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-19e2773913213f81.xml (deflated 36%) 2024-04-03T17:40:02.0000793Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c3bba96b97011ac4.xml (deflated 37%) 2024-04-03T17:40:02.0002598Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-809f014b6c0b37ee.xml (deflated 35%) 2024-04-03T17:40:02.0004387Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-58b3f0dbb19ead38.xml (deflated 36%) 2024-04-03T17:40:02.0006184Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-38e0acc53e20b464.xml (deflated 36%) 2024-04-03T17:40:02.0007983Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-cefa8da9ff7d1923.xml (deflated 36%) 2024-04-03T17:40:02.0009801Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-df0de4486acc1595.xml (deflated 37%) 2024-04-03T17:40:02.0011586Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8d686c8a273d2f3c.xml (deflated 36%) 2024-04-03T17:40:02.0013373Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-62634e33c8fa3218.xml (deflated 36%) 2024-04-03T17:40:02.0015178Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-71c11cfbcbe0d211.xml (deflated 36%) 2024-04-03T17:40:02.0016996Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c24ff0e5791da863.xml (deflated 36%) 2024-04-03T17:40:02.0018783Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3647b6fa3e749eed.xml (deflated 36%) 2024-04-03T17:40:02.0020688Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e63d58d0a35ee594.xml (deflated 36%) 2024-04-03T17:40:02.0022499Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-85e89762081dcdeb.xml (deflated 36%) 2024-04-03T17:40:02.0024311Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-bfac3a5169d17828.xml (deflated 35%) 2024-04-03T17:40:02.0026100Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-5ef15b620cc6e7ca.xml (deflated 36%) 2024-04-03T17:40:02.0027898Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-72723f30e751ebe6.xml (deflated 35%) 2024-04-03T17:40:02.0030004Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-11e37ec37ec86002.xml (deflated 36%) 2024-04-03T17:40:02.0031811Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e1969b0b86664603.xml (deflated 36%) 2024-04-03T17:40:02.0033583Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-9734c25e74e89232.xml (deflated 37%) 2024-04-03T17:40:02.0035365Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0d5798e820cf7ccd.xml (deflated 35%) 2024-04-03T17:40:02.0037153Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-82f7337a3a2764c6.xml (deflated 36%) 2024-04-03T17:40:02.0038943Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-49f4fb9f065be414.xml (deflated 35%) 2024-04-03T17:40:02.0040806Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-13d0db5288506358.xml (deflated 35%) 2024-04-03T17:40:02.0042600Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3f334d9cf3b3615e.xml (deflated 36%) 2024-04-03T17:40:02.0044405Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4bed7053b1fdba23.xml (deflated 35%) 2024-04-03T17:40:02.0046201Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-88e3cf3b0fb143a9.xml (deflated 35%) 2024-04-03T17:40:02.0047998Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-5771fe19eef1ec94.xml (deflated 36%) 2024-04-03T17:40:02.0049787Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-329b26d25b1ef052.xml (deflated 36%) 2024-04-03T17:40:02.0051583Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-ed2fda52a87a669c.xml (deflated 36%) 2024-04-03T17:40:02.0053369Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4b830ec943c384cc.xml (deflated 36%) 2024-04-03T17:40:02.0055156Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-529e1397f6e8915c.xml (deflated 36%) 2024-04-03T17:40:02.0056932Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c180d83875bfd9e6.xml (deflated 35%) 2024-04-03T17:40:02.0058729Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d9110c98994858d2.xml (deflated 36%) 2024-04-03T17:40:02.0060691Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1c28a0f0a187c932.xml (deflated 35%) 2024-04-03T17:40:02.0062490Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0a82dcb83e3b4b1f.xml (deflated 36%) 2024-04-03T17:40:02.0064270Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-2ab97a1d4fed17f0.xml (deflated 36%) 2024-04-03T17:40:02.0066060Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0a7b4bb24048a4bd.xml (deflated 36%) 2024-04-03T17:40:02.0067851Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-ff32dfd91b4d7334.xml (deflated 36%) 2024-04-03T17:40:02.0069835Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-67eccec9432a9d99.xml (deflated 36%) 2024-04-03T17:40:02.0071618Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-594bd38c2c0d98a6.xml (deflated 36%) 2024-04-03T17:40:02.0073406Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-09ae891ed5841dde.xml (deflated 35%) 2024-04-03T17:40:02.0075195Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e5ed301775e004ea.xml (deflated 36%) 2024-04-03T17:40:02.0076972Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-57a835b70bf57f3b.xml (deflated 36%) 2024-04-03T17:40:02.0078742Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4d2344729e3a1bbe.xml (deflated 35%) 2024-04-03T17:40:02.0080601Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-50b31da02a5f1a1a.xml (deflated 36%) 2024-04-03T17:40:02.0082396Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6ad09dcd516211a0.xml (deflated 35%) 2024-04-03T17:40:02.0084180Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-45678db115cc3545.xml (deflated 35%) 2024-04-03T17:40:02.0085951Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f8aa1c55f6382497.xml (deflated 35%) 2024-04-03T17:40:02.0087736Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-51c7edd75f469b6c.xml (deflated 35%) 2024-04-03T17:40:02.0089520Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1534d9cd3ae65402.xml (deflated 35%) 2024-04-03T17:40:02.0091305Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6064e05c7861f679.xml (deflated 35%) 2024-04-03T17:40:02.0093064Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b728c19c7fd71d48.xml (deflated 35%) 2024-04-03T17:40:02.0094837Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6e760ee62f2a64b5.xml (deflated 35%) 2024-04-03T17:40:02.0096611Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-800e530a6b9a6608.xml (deflated 35%) 2024-04-03T17:40:02.0098386Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-82db84b336c86234.xml (deflated 35%) 2024-04-03T17:40:02.0100164Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-120ddbf19d2ebd5b.xml (deflated 35%) 2024-04-03T17:40:02.0102075Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-069b153358f47272.xml (deflated 36%) 2024-04-03T17:40:02.0103861Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-2a93785a840fbefd.xml (deflated 35%) 2024-04-03T17:40:02.0105651Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-105383e1eb2a3eff.xml (deflated 35%) 2024-04-03T17:40:02.0107429Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7b7488418db9fa4e.xml (deflated 35%) 2024-04-03T17:40:02.0109218Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-fb939fc50dda2b18.xml (deflated 35%) 2024-04-03T17:40:02.0111316Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-de94cedbc8d4f63d.xml (deflated 35%) 2024-04-03T17:40:02.0113118Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-062c163ef1e71ef6.xml (deflated 36%) 2024-04-03T17:40:02.0114883Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a02106ef6b9d9917.xml (deflated 35%) 2024-04-03T17:40:02.0116664Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c0214d0e76dccc13.xml (deflated 36%) 2024-04-03T17:40:02.0118455Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-5e88a729267cb2a0.xml (deflated 35%) 2024-04-03T17:40:02.0120244Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e34f64f43e923638.xml (deflated 35%) 2024-04-03T17:40:02.0122105Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b27a3d68de7cec66.xml (deflated 35%) 2024-04-03T17:40:02.0123881Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d574d1a9833892ba.xml (deflated 36%) 2024-04-03T17:40:02.0125673Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-76d888dc116315a5.xml (deflated 35%) 2024-04-03T17:40:02.0127457Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-71da0f96458246bc.xml (deflated 36%) 2024-04-03T17:40:02.0129233Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-5adde24125868cc7.xml (deflated 35%) 2024-04-03T17:40:02.0131010Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0db4e9a5701ebb69.xml (deflated 35%) 2024-04-03T17:40:02.0132796Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-9c8eb537542d0b8f.xml (deflated 35%) 2024-04-03T17:40:02.0134583Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4cfb6fa6a56efca4.xml (deflated 35%) 2024-04-03T17:40:02.0136374Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-800dd8ec0a7ad2d2.xml (deflated 36%) 2024-04-03T17:40:02.0138145Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-cbca97a7f5288f14.xml (deflated 35%) 2024-04-03T17:40:02.0139939Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3c0abdbe1bd7a024.xml (deflated 35%) 2024-04-03T17:40:02.0141832Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-cba667605898cd4e.xml (deflated 35%) 2024-04-03T17:40:02.0143626Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1f95d1e6c17005a2.xml (deflated 34%) 2024-04-03T17:40:02.0145395Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-774dd85d6e8518d7.xml (deflated 35%) 2024-04-03T17:40:02.0147174Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-326cb64b50e32f80.xml (deflated 35%) 2024-04-03T17:40:02.0148956Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a097bc39cd176ad9.xml (deflated 37%) 2024-04-03T17:40:02.0150921Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-15891242468a8a46.xml (deflated 36%) 2024-04-03T17:40:02.0152698Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0525cb8275d23c53.xml (deflated 37%) 2024-04-03T17:40:02.0154477Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1cda67037a0fa957.xml (deflated 36%) 2024-04-03T17:40:02.0156262Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-92bc21953cb2fd61.xml (deflated 36%) 2024-04-03T17:40:02.0158028Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-531913435e3c7abc.xml (deflated 56%) 2024-04-03T17:40:02.0159791Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d74aaf4f5687b73c.xml (deflated 35%) 2024-04-03T17:40:02.0161655Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-ca7775db752eea64.xml (deflated 36%) 2024-04-03T17:40:02.0163443Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3cd3e14a9bd09fca.xml (deflated 38%) 2024-04-03T17:40:02.0165232Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-591589e763cb1657.xml (deflated 38%) 2024-04-03T17:40:02.0166999Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-79c5216425012c7b.xml (deflated 38%) 2024-04-03T17:40:02.0168775Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b33e35ad91bf6c4f.xml (deflated 37%) 2024-04-03T17:40:02.0170560Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d0386929e2898037.xml (deflated 38%) 2024-04-03T17:40:02.0172342Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-5cec1d35f8c86367.xml (deflated 38%) 2024-04-03T17:40:02.0174112Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-136d6985f27a4424.xml (deflated 37%) 2024-04-03T17:40:02.0175882Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-73b57567ab80942c.xml (deflated 37%) 2024-04-03T17:40:02.0177670Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c9d9ac6200a865f8.xml (deflated 37%) 2024-04-03T17:40:02.0179447Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b7e2513a0501d590.xml (deflated 37%) 2024-04-03T17:40:02.0181315Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7ad18595528534fb.xml (deflated 36%) 2024-04-03T17:40:02.0183098Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0b4c8cba7cbc74bf.xml (deflated 37%) 2024-04-03T17:40:02.0184894Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-35a55cb29157b595.xml (deflated 39%) 2024-04-03T17:40:02.0186674Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1d37ae4cf38f32bc.xml (deflated 37%) 2024-04-03T17:40:02.0188442Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4d263aec744cb0fd.xml (deflated 37%) 2024-04-03T17:40:02.0190525Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a9f880b9dc466c47.xml (deflated 37%) 2024-04-03T17:40:02.0192336Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7b1d0fcb5dfad646.xml (deflated 37%) 2024-04-03T17:40:02.0194130Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-87cdb363fec6e456.xml (deflated 36%) 2024-04-03T17:40:02.0195901Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e4f3d27fbc14ea5e.xml (deflated 37%) 2024-04-03T17:40:02.0197689Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-71a75e7164cb083e.xml (deflated 37%) 2024-04-03T17:40:02.0199471Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a5f95219dfadb30a.xml (deflated 37%) 2024-04-03T17:40:02.0201328Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-5324a97bd59d7af5.xml (deflated 36%) 2024-04-03T17:40:02.0203110Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-cf81d0dd8c25460d.xml (deflated 37%) 2024-04-03T17:40:02.0204889Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-9bfd58d044687526.xml (deflated 37%) 2024-04-03T17:40:02.0206671Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-38c8b83c69186f43.xml (deflated 37%) 2024-04-03T17:40:02.0208466Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_copy/distributed.pipeline.sync.test_copy-3e8c9fac700707e1.xml (deflated 75%) 2024-04-03T17:40:02.0210327Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_mixed_precision/distributed.fsdp.test_fsdp_mixed_precision-a11749e6b4382e9b.xml (deflated 95%) 2024-04-03T17:40:02.0212351Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_binary_cmp/distributed._shard.sharded_tensor.ops.test_binary_cmp-899ee6fe72ab7b4a.xml (deflated 80%) 2024-04-03T17:40:02.0214184Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-db5574783ba046ab.xml (deflated 44%) 2024-04-03T17:40:02.0215740Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-70469f769eb2de51.xml (deflated 44%) 2024-04-03T17:40:02.0217300Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-6f81b013399df4bf.xml (deflated 44%) 2024-04-03T17:40:02.0218845Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-00c46b4d603bda9f.xml (deflated 43%) 2024-04-03T17:40:02.0220388Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-89a8d8b289838c6e.xml (deflated 44%) 2024-04-03T17:40:02.0222046Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-f564305958d4b4ae.xml (deflated 44%) 2024-04-03T17:40:02.0223586Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-a048f9098ee457dd.xml (deflated 35%) 2024-04-03T17:40:02.0225142Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-6b9bc47c5d08523c.xml (deflated 42%) 2024-04-03T17:40:02.0226697Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-3c775c7e24ad004f.xml (deflated 42%) 2024-04-03T17:40:02.0228247Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-7ae8c296cd4e77a4.xml (deflated 35%) 2024-04-03T17:40:02.0229880Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-bd08c4e53357218f.xml (deflated 35%) 2024-04-03T17:40:02.0231541Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-ba58ba147cb0447b.xml (deflated 35%) 2024-04-03T17:40:02.0233090Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-27500c0ecac7deb9.xml (deflated 43%) 2024-04-03T17:40:02.0234646Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-fd4e1599becbfe44.xml (deflated 43%) 2024-04-03T17:40:02.0236188Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-b8ef26f2442fbac1.xml (deflated 35%) 2024-04-03T17:40:02.0237752Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-355eb2d644ffdace.xml (deflated 37%) 2024-04-03T17:40:02.0239327Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-89039622fcb3815b.xml (deflated 36%) 2024-04-03T17:40:02.0240968Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-5bc1e0d0c9f7c885.xml (deflated 35%) 2024-04-03T17:40:02.0242532Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-e38b9d023d39b380.xml (deflated 35%) 2024-04-03T17:40:02.0244096Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-c09979f2ee3c7119.xml (deflated 37%) 2024-04-03T17:40:02.0245656Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-83b365865b5fa0f9.xml (deflated 36%) 2024-04-03T17:40:02.0247207Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-d5f9082cc752f971.xml (deflated 35%) 2024-04-03T17:40:02.0248746Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-9d404a37a077bb92.xml (deflated 34%) 2024-04-03T17:40:02.0250433Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-0b73e311241c9514.xml (deflated 35%) 2024-04-03T17:40:02.0251999Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-3921b96990feff52.xml (deflated 35%) 2024-04-03T17:40:02.0253549Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-7b27dd5b174e362f.xml (deflated 35%) 2024-04-03T17:40:02.0255099Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-08c2637cd2c6d559.xml (deflated 35%) 2024-04-03T17:40:02.0256716Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-49c694ac98cd8cd8.xml (deflated 34%) 2024-04-03T17:40:02.0258286Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-a0fee5d0e174599d.xml (deflated 34%) 2024-04-03T17:40:02.0259850Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-06abe0f65fa6a706.xml (deflated 34%) 2024-04-03T17:40:02.0261494Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-b853b2b9e8f6e056.xml (deflated 36%) 2024-04-03T17:40:02.0263063Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-d9dce920735c8db4.xml (deflated 37%) 2024-04-03T17:40:02.0264621Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-c7afb645fa4c39db.xml (deflated 35%) 2024-04-03T17:40:02.0266175Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-b3839a33dcb58b0e.xml (deflated 37%) 2024-04-03T17:40:02.0267976Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_tracker/distributed.pipeline.sync.skip.test_tracker-ca1721634e29ed9d.xml (deflated 70%) 2024-04-03T17:40:02.0270216Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_leak/distributed.pipeline.sync.skip.test_leak-364d2b6a9c28d17b.xml (deflated 75%) 2024-04-03T17:40:02.0272226Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_stash_pop/distributed.pipeline.sync.skip.test_stash_pop-00cbf470d8989eba.xml (deflated 77%) 2024-04-03T17:40:02.0274074Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-e645cd80916aaf37.xml (deflated 43%) 2024-04-03T17:40:02.0275722Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-27ceb9ac50289e1c.xml (deflated 43%) 2024-04-03T17:40:02.0277366Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-bcec9f4f79442fc7.xml (deflated 43%) 2024-04-03T17:40:02.0279011Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-231de943657e3a99.xml (deflated 43%) 2024-04-03T17:40:02.0280738Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-135b50ee9f7b3474.xml (deflated 36%) 2024-04-03T17:40:02.0282373Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-f7f047ee837e48a3.xml (deflated 44%) 2024-04-03T17:40:02.0284023Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-bbba29f69ef4d992.xml (deflated 43%) 2024-04-03T17:40:02.0285683Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-4d5b5778e338390a.xml (deflated 44%) 2024-04-03T17:40:02.0287314Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-0b0ad79df1611ba9.xml (deflated 44%) 2024-04-03T17:40:02.0288959Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-8e94401e0f8b9b2f.xml (deflated 44%) 2024-04-03T17:40:02.0290602Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-2681a3eb8d4f90dc.xml (deflated 44%) 2024-04-03T17:40:02.0307867Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-4405c4961638e5a6.xml (deflated 44%) 2024-04-03T17:40:02.0309820Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-f0baa1e6ba77547f.xml (deflated 44%) 2024-04-03T17:40:02.0311497Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-f290afaceabd3b72.xml (deflated 44%) 2024-04-03T17:40:02.0313160Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-74f03a0a644cffc4.xml (deflated 44%) 2024-04-03T17:40:02.0315066Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_tensor_ops/distributed._shard.sharded_tensor.ops.test_tensor_ops-5cfced978a370c45.xml (deflated 83%) 2024-04-03T17:40:02.0317236Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_stream/distributed.pipeline.sync.test_stream-a387f97ad3ece9e6.xml (deflated 89%) 2024-04-03T17:40:02.0318985Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-067a1668639d2b33.xml (deflated 43%) 2024-04-03T17:40:02.0320713Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-9f42a0c1fdb21ae4.xml (deflated 43%) 2024-04-03T17:40:02.0322379Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-e3f62db64c7b9081.xml (deflated 44%) 2024-04-03T17:40:02.0324009Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-16eebd646829e2e4.xml (deflated 44%) 2024-04-03T17:40:02.0325652Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-3b3be59f3c4a8bb9.xml (deflated 45%) 2024-04-03T17:40:02.0327436Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-dbac3d32e2b324c0.xml (deflated 45%) 2024-04-03T17:40:02.0329093Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-94c29fb5617b9cdf.xml (deflated 46%) 2024-04-03T17:40:02.0330723Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-e33bab4bca854560.xml (deflated 46%) 2024-04-03T17:40:02.0332361Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-91f5ecfd2b89a929.xml (deflated 46%) 2024-04-03T17:40:02.0334011Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-d0be8efad8432ce5.xml (deflated 46%) 2024-04-03T17:40:02.0335670Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-8a70eced4d7f5823.xml (deflated 47%) 2024-04-03T17:40:02.0337304Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-6f84a3ec3ddc5d7a.xml (deflated 46%) 2024-04-03T17:40:02.0338944Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-dea7938320effb8e.xml (deflated 45%) 2024-04-03T17:40:02.0340575Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-80c8f14f070882a3.xml (deflated 43%) 2024-04-03T17:40:02.0342200Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-7a82c7c9a481681e.xml (deflated 43%) 2024-04-03T17:40:02.0343801Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-27de879749058baf.xml (deflated 43%) 2024-04-03T17:40:02.0345421Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-a0ccffb991ae01f7.xml (deflated 46%) 2024-04-03T17:40:02.0347054Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-b6727f836c556ef0.xml (deflated 46%) 2024-04-03T17:40:02.0348675Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-63ce254a891922ae.xml (deflated 46%) 2024-04-03T17:40:02.0350503Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-65f807bc0bbf301a.xml (deflated 46%) 2024-04-03T17:40:02.0352131Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-37ac7005741effaf.xml (deflated 46%) 2024-04-03T17:40:02.0353753Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-bae28f03f766e5fd.xml (deflated 46%) 2024-04-03T17:40:02.0355521Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_balance/distributed.pipeline.sync.test_balance-8fa2321eb42fe005.xml (deflated 83%) 2024-04-03T17:40:02.0357698Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/distributed._shard.sharded_tensor.test_sharded_tensor_reshard-fd0776dbc9dc64e5.xml (deflated 70%) 2024-04-03T17:40:02.0359804Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_dependency/distributed.pipeline.sync.test_dependency-d733c3f001162586.xml (deflated 71%) 2024-04-03T17:40:02.0361802Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_api/distributed.pipeline.sync.skip.test_api-142290e27b077bf1.xml (deflated 60%) 2024-04-03T17:40:02.0363827Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_deferred_batch_norm/distributed.pipeline.sync.test_deferred_batch_norm-07e7ff5ae5b75c64.xml (deflated 79%) 2024-04-03T17:40:02.0365872Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_state_dict/distributed.checkpoint.test_state_dict-575f8b719ab0bffc.xml (deflated 87%) 2024-04-03T17:40:02.0367553Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-59fb913f01f2ff29.xml (deflated 35%) 2024-04-03T17:40:02.0369065Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-18c1d29220709034.xml (deflated 37%) 2024-04-03T17:40:02.0370571Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-38049265f9771a53.xml (deflated 36%) 2024-04-03T17:40:02.0372065Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-8dd47565349c8149.xml (deflated 43%) 2024-04-03T17:40:02.0373572Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-028f479d9684b90e.xml (deflated 35%) 2024-04-03T17:40:02.0375078Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-cfe0fb93fd6e5307.xml (deflated 35%) 2024-04-03T17:40:02.0376599Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7fe800a8f25c5307.xml (deflated 36%) 2024-04-03T17:40:02.0378089Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6440bf80503c43bd.xml (deflated 35%) 2024-04-03T17:40:02.0379595Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-562496bee305719c.xml (deflated 36%) 2024-04-03T17:40:02.0381098Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-72fa57cb204a064e.xml (deflated 35%) 2024-04-03T17:40:02.0382614Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-613db8cfec028c91.xml (deflated 43%) 2024-04-03T17:40:02.0384114Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-4e0a58178d742eb0.xml (deflated 36%) 2024-04-03T17:40:02.0385621Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7c14a31676952f11.xml (deflated 43%) 2024-04-03T17:40:02.0387126Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a1f80717b0c99275.xml (deflated 36%) 2024-04-03T17:40:02.0388638Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e8c1e1d6bd2f1d5c.xml (deflated 35%) 2024-04-03T17:40:02.0390226Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-60f067c017599866.xml (deflated 36%) 2024-04-03T17:40:02.0391732Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a4b8a1cda69032a9.xml (deflated 36%) 2024-04-03T17:40:02.0393247Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-df1c6572cfb22cb4.xml (deflated 43%) 2024-04-03T17:40:02.0394745Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-bc5d273f913dcc7d.xml (deflated 36%) 2024-04-03T17:40:02.0396390Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-72fc576e78dec13c.xml (deflated 35%) 2024-04-03T17:40:02.0397903Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-663cfdd9c19acf2f.xml (deflated 43%) 2024-04-03T17:40:02.0399416Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-1bca96393024fdf2.xml (deflated 35%) 2024-04-03T17:40:02.0400985Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5a5f60272106dd73.xml (deflated 36%) 2024-04-03T17:40:02.0402492Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-83a7bfda19113091.xml (deflated 43%) 2024-04-03T17:40:02.0404009Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-8ef06cefdec8c668.xml (deflated 36%) 2024-04-03T17:40:02.0405608Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-ecc5cdccdfdca939.xml (deflated 36%) 2024-04-03T17:40:02.0407120Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-f5f02873b3cdebce.xml (deflated 43%) 2024-04-03T17:40:02.0408632Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-bef8c266fa997593.xml (deflated 36%) 2024-04-03T17:40:02.0410141Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d6f2d93c69499d97.xml (deflated 36%) 2024-04-03T17:40:02.0411660Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-20fa82dbf76d039d.xml (deflated 43%) 2024-04-03T17:40:02.0413172Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-dfb12ed76fbaebb5.xml (deflated 36%) 2024-04-03T17:40:02.0414689Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-0da1a0002156e431.xml (deflated 35%) 2024-04-03T17:40:02.0416194Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-79f5194fbb55f471.xml (deflated 36%) 2024-04-03T17:40:02.0417705Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-bea3a2f0bda35ef7.xml (deflated 43%) 2024-04-03T17:40:02.0419203Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e15b395c65c556bd.xml (deflated 35%) 2024-04-03T17:40:02.0420722Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5eb5b44ddaf3ffda.xml (deflated 36%) 2024-04-03T17:40:02.0422224Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-82f24a4c38832a6d.xml (deflated 43%) 2024-04-03T17:40:02.0423723Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-440369ad3276d848.xml (deflated 36%) 2024-04-03T17:40:02.0425222Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-4e2a3ca4c2a366e2.xml (deflated 35%) 2024-04-03T17:40:02.0426732Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b307a1e3f410f510.xml (deflated 36%) 2024-04-03T17:40:02.0428248Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-cdd5fef4cdbdbe24.xml (deflated 36%) 2024-04-03T17:40:02.0429941Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7463b717fd4e4b5e.xml (deflated 43%) 2024-04-03T17:40:02.0431464Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a4c07bf65cc4fd7b.xml (deflated 36%) 2024-04-03T17:40:02.0432974Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-bc7971570f9bff30.xml (deflated 36%) 2024-04-03T17:40:02.0434485Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-954a2ad2c6e3191e.xml (deflated 36%) 2024-04-03T17:40:02.0436103Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-1a6e8b1f903ac046.xml (deflated 43%) 2024-04-03T17:40:02.0437613Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e407000441cb86f8.xml (deflated 36%) 2024-04-03T17:40:02.0439116Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-ab8418e4f013b4de.xml (deflated 36%) 2024-04-03T17:40:02.0440699Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9027d3e34aa5a7a1.xml (deflated 47%) 2024-04-03T17:40:02.0442204Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7090db44cacc9a30.xml (deflated 36%) 2024-04-03T17:40:02.0443800Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-25aeda8e9be0ffdb.xml (deflated 57%) 2024-04-03T17:40:02.0445331Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d3c44bc66ea4273a.xml (deflated 36%) 2024-04-03T17:40:02.0446839Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5314bc0e09e36a7e.xml (deflated 43%) 2024-04-03T17:40:02.0448338Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-4233e15faa51a1e2.xml (deflated 36%) 2024-04-03T17:40:02.0449853Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9a5e841b1e1cd38f.xml (deflated 43%) 2024-04-03T17:40:02.0451364Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-33c72c0aa48ab656.xml (deflated 43%) 2024-04-03T17:40:02.0452880Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-563d351837ad9f0a.xml (deflated 43%) 2024-04-03T17:40:02.0454387Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-002b3fc7b9bfafd9.xml (deflated 42%) 2024-04-03T17:40:02.0455902Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-42ea453a1583ef76.xml (deflated 42%) 2024-04-03T17:40:02.0457423Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-acfbbdad02acbe05.xml (deflated 42%) 2024-04-03T17:40:02.0458948Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5be7cd8abf10ee5e.xml (deflated 42%) 2024-04-03T17:40:02.0460451Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a685450d80cc5904.xml (deflated 42%) 2024-04-03T17:40:02.0461949Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-0439057a71b13950.xml (deflated 42%) 2024-04-03T17:40:02.0463459Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-96777d5399e7d01f.xml (deflated 42%) 2024-04-03T17:40:02.0464959Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2d78d04b7326e512.xml (deflated 42%) 2024-04-03T17:40:02.0466454Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2a738762a97920f2.xml (deflated 42%) 2024-04-03T17:40:02.0467949Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-fb8b16453822303a.xml (deflated 42%) 2024-04-03T17:40:02.0469463Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-758c2a7fd45ba2c1.xml (deflated 42%) 2024-04-03T17:40:02.0471047Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-0918e66c9afb9418.xml (deflated 42%) 2024-04-03T17:40:02.0472567Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-eddd8f69fcec9520.xml (deflated 42%) 2024-04-03T17:40:02.0474159Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-403199960d805411.xml (deflated 37%) 2024-04-03T17:40:02.0475665Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-56bad8e37d0cc5fe.xml (deflated 43%) 2024-04-03T17:40:02.0477166Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e05e1bd5bfcab265.xml (deflated 36%) 2024-04-03T17:40:02.0478678Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-80267da7f7ccffa7.xml (deflated 36%) 2024-04-03T17:40:02.0480194Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-cf8deb54f85acc68.xml (deflated 36%) 2024-04-03T17:40:02.0481804Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b9e22e9071e081f8.xml (deflated 36%) 2024-04-03T17:40:02.0483391Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d3aeab8a254ce4cb.xml (deflated 45%) 2024-04-03T17:40:02.0484903Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-58f4a22a5473b67f.xml (deflated 44%) 2024-04-03T17:40:02.0486408Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-840522cd26ccd53f.xml (deflated 44%) 2024-04-03T17:40:02.0487917Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d45d10e55f42fd97.xml (deflated 43%) 2024-04-03T17:40:02.0489413Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-15ce119d2140f662.xml (deflated 44%) 2024-04-03T17:40:02.0490922Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-aa1887e150f3f38b.xml (deflated 44%) 2024-04-03T17:40:02.0492432Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-acc5bd8e58753301.xml (deflated 44%) 2024-04-03T17:40:02.0493951Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-1a863cbff33ecf30.xml (deflated 44%) 2024-04-03T17:40:02.0495443Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6d764a9308586631.xml (deflated 36%) 2024-04-03T17:40:02.0496943Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e107b5a342cf3155.xml (deflated 36%) 2024-04-03T17:40:02.0498452Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-f2a688e51fad0e35.xml (deflated 36%) 2024-04-03T17:40:02.0500068Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-945ad71f05ea8129.xml (deflated 36%) 2024-04-03T17:40:02.0501576Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-25085c56722af09c.xml (deflated 44%) 2024-04-03T17:40:02.0503093Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9d2b6dd7b0a8d861.xml (deflated 35%) 2024-04-03T17:40:02.0504602Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5d0569d62ada37b5.xml (deflated 44%) 2024-04-03T17:40:02.0506094Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-c79a459bbaefff3d.xml (deflated 37%) 2024-04-03T17:40:02.0507608Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-4ef2f609df725a71.xml (deflated 36%) 2024-04-03T17:40:02.0509116Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-1fae3f8accd2dd2f.xml (deflated 42%) 2024-04-03T17:40:02.0510772Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-662c877992270747.xml (deflated 42%) 2024-04-03T17:40:02.0512419Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-c759a9866338831d.xml (deflated 37%) 2024-04-03T17:40:02.0514058Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-570ff64c6f9c189c.xml (deflated 36%) 2024-04-03T17:40:02.0515570Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-14f83b669b1f1e2e.xml (deflated 36%) 2024-04-03T17:40:02.0517083Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e1525ddb3f52fde2.xml (deflated 37%) 2024-04-03T17:40:02.0518580Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-ba8274d343246f2b.xml (deflated 35%) 2024-04-03T17:40:02.0520085Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-787b70cb32647a84.xml (deflated 38%) 2024-04-03T17:40:02.0521744Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-76612804fa6fe14c.xml (deflated 37%) 2024-04-03T17:40:02.0523251Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6357fc3e1f45271e.xml (deflated 37%) 2024-04-03T17:40:02.0524745Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-22587d6fbd89b271.xml (deflated 44%) 2024-04-03T17:40:02.0526257Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e4269b1f2faa0712.xml (deflated 44%) 2024-04-03T17:40:02.0527759Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-0cf72efd12ddb447.xml (deflated 44%) 2024-04-03T17:40:02.0529276Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6c5ef847a86c8e3f.xml (deflated 43%) 2024-04-03T17:40:02.0530774Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-94e6f0b1f6393d52.xml (deflated 44%) 2024-04-03T17:40:02.0532281Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-1168659fb8448006.xml (deflated 44%) 2024-04-03T17:40:02.0533788Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-4d823eda28a3cec7.xml (deflated 43%) 2024-04-03T17:40:02.0535286Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-85c55b508f1e5d1f.xml (deflated 43%) 2024-04-03T17:40:02.0536795Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-18335f652f892db3.xml (deflated 43%) 2024-04-03T17:40:02.0538301Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-dcc7fd5c9a9107f0.xml (deflated 34%) 2024-04-03T17:40:02.0539810Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-743dd616504a854c.xml (deflated 34%) 2024-04-03T17:40:02.0541305Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e012797b49f3ce50.xml (deflated 34%) 2024-04-03T17:40:02.0542821Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-02bd1ca9ee3e22e3.xml (deflated 35%) 2024-04-03T17:40:02.0544335Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a032be69105c031a.xml (deflated 35%) 2024-04-03T17:40:02.0545836Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e89cd593fbdd8764.xml (deflated 35%) 2024-04-03T17:40:02.0547329Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-57a1638be33ad6dc.xml (deflated 34%) 2024-04-03T17:40:02.0548829Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-f574224ac168164a.xml (deflated 35%) 2024-04-03T17:40:02.0550457Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-abbb31de720a3c21.xml (deflated 43%) 2024-04-03T17:40:02.0552051Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-ca7fd38574ef8dd8.xml (deflated 36%) 2024-04-03T17:40:02.0553555Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-31670f0be35f3dde.xml (deflated 43%) 2024-04-03T17:40:02.0555067Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-efca9d4f82ccf334.xml (deflated 36%) 2024-04-03T17:40:02.0556574Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-54be8ef7a7ddc454.xml (deflated 43%) 2024-04-03T17:40:02.0558089Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9568915c04086ab0.xml (deflated 35%) 2024-04-03T17:40:02.0559589Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b1a1e4f824de900c.xml (deflated 43%) 2024-04-03T17:40:02.0561241Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-bb62264f65113a7e.xml (deflated 35%) 2024-04-03T17:40:02.0562757Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-3b233185f8c6f16b.xml (deflated 35%) 2024-04-03T17:40:02.0564263Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9d2ba1a386b672ee.xml (deflated 44%) 2024-04-03T17:40:02.0565758Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e273b33092125efa.xml (deflated 34%) 2024-04-03T17:40:02.0567266Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-4fc18c333e7ac881.xml (deflated 35%) 2024-04-03T17:40:02.0568767Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9670b20b2fb7a382.xml (deflated 35%) 2024-04-03T17:40:02.0570464Z adding: test/test-reports/python-pytest/distributed.elastic.utils.logging_test/distributed.elastic.utils.logging_test-e2d0375fdeb8ae3d.xml (deflated 51%) 2024-04-03T17:40:02.0572302Z adding: test/test-reports/python-pytest/distributed.elastic.utils.util_test/distributed.elastic.utils.util_test-616c85d4b0e55985.xml (deflated 75%) 2024-04-03T17:40:02.0574019Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-75dcc1766b051279.xml (deflated 43%) 2024-04-03T17:40:02.0575664Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-df9241cc14793ba2.xml (deflated 43%) 2024-04-03T17:40:02.0577302Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-64f5e6e6d62c9002.xml (deflated 37%) 2024-04-03T17:40:02.0578960Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cd1dbb972a0f3d13.xml (deflated 37%) 2024-04-03T17:40:02.0580618Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a8a44a13bdc5bb6.xml (deflated 41%) 2024-04-03T17:40:02.0582268Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-86ee44b44582a921.xml (deflated 41%) 2024-04-03T17:40:02.0583899Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8c6f3d8c2a5ecf54.xml (deflated 41%) 2024-04-03T17:40:02.0585552Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f0e2bc8eb80abe55.xml (deflated 42%) 2024-04-03T17:40:02.0587203Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9905facdc7b75ca.xml (deflated 37%) 2024-04-03T17:40:02.0588859Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6c71de2b1cbe66ad.xml (deflated 36%) 2024-04-03T17:40:02.0590758Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f228cd202d2e49e6.xml (deflated 36%) 2024-04-03T17:40:02.0592412Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f122eb3e1478bfbe.xml (deflated 36%) 2024-04-03T17:40:02.0594058Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a073f1ee75b640a0.xml (deflated 43%) 2024-04-03T17:40:02.0595696Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a1d8464882bb009.xml (deflated 45%) 2024-04-03T17:40:02.0597323Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ace884724d15efc.xml (deflated 36%) 2024-04-03T17:40:02.0598964Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b8748a5efcbe4e41.xml (deflated 35%) 2024-04-03T17:40:02.0600771Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13caeb3aa69368b4.xml (deflated 36%) 2024-04-03T17:40:02.0602427Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c97b57785f80349d.xml (deflated 36%) 2024-04-03T17:40:02.0604050Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1cf632283293af3c.xml (deflated 36%) 2024-04-03T17:40:02.0605698Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9194b9aa4821b6f5.xml (deflated 36%) 2024-04-03T17:40:02.0607342Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-023d2cf4a197381d.xml (deflated 35%) 2024-04-03T17:40:02.0608986Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fc180d820b0ced86.xml (deflated 36%) 2024-04-03T17:40:02.0610627Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-71446674277cadc7.xml (deflated 36%) 2024-04-03T17:40:02.0612268Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e15dc4c45f57ed9b.xml (deflated 36%) 2024-04-03T17:40:02.0613902Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-26855631557c1062.xml (deflated 36%) 2024-04-03T17:40:02.0615533Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c07b001fb870911.xml (deflated 35%) 2024-04-03T17:40:02.0617162Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-421b3c841589ca3d.xml (deflated 36%) 2024-04-03T17:40:02.0618809Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a3cbf6f3bb334a2.xml (deflated 35%) 2024-04-03T17:40:02.0620451Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13093729e048f100.xml (deflated 36%) 2024-04-03T17:40:02.0622076Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-00515286d9ce4252.xml (deflated 37%) 2024-04-03T17:40:02.0623698Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e1df8d04b7cd643.xml (deflated 36%) 2024-04-03T17:40:02.0625459Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-75e99994a5461a7a.xml (deflated 36%) 2024-04-03T17:40:02.0627098Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89e731bb6fe6b00c.xml (deflated 42%) 2024-04-03T17:40:02.0628741Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0571047722ceb86d.xml (deflated 43%) 2024-04-03T17:40:02.0630539Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5f319a20d844dfa6.xml (deflated 44%) 2024-04-03T17:40:02.0632189Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ad96d1d6f486ae82.xml (deflated 43%) 2024-04-03T17:40:02.0633825Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f957874183669348.xml (deflated 44%) 2024-04-03T17:40:02.0635451Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d090d463d3834c1.xml (deflated 44%) 2024-04-03T17:40:02.0637080Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ae051bb00ff328d1.xml (deflated 44%) 2024-04-03T17:40:02.0638794Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8c855af6b82b5ebb.xml (deflated 45%) 2024-04-03T17:40:02.0640521Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1e704e1067e072f.xml (deflated 44%) 2024-04-03T17:40:02.0642173Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd1c2a3f094b88d6.xml (deflated 44%) 2024-04-03T17:40:02.0643812Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-addee8cd2b401763.xml (deflated 43%) 2024-04-03T17:40:02.0645467Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-52af6c176ffcb99b.xml (deflated 36%) 2024-04-03T17:40:02.0647109Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-79dc6b5d31defd0e.xml (deflated 57%) 2024-04-03T17:40:02.0648751Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-316aed1627019c71.xml (deflated 42%) 2024-04-03T17:40:02.0650376Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2ce177e3f5937136.xml (deflated 36%) 2024-04-03T17:40:02.0652012Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-020d7fabcf0412a1.xml (deflated 36%) 2024-04-03T17:40:02.0653662Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6a29a78f08c4aaaa.xml (deflated 42%) 2024-04-03T17:40:02.0655305Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a1c4284a065c85e6.xml (deflated 42%) 2024-04-03T17:40:02.0656926Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8858872ea62284e9.xml (deflated 36%) 2024-04-03T17:40:02.0658570Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6e9e7ebc1360e8ac.xml (deflated 47%) 2024-04-03T17:40:02.0660218Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e3688c8a4fadb145.xml (deflated 47%) 2024-04-03T17:40:02.0661848Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-075525ac1ae8c529.xml (deflated 48%) 2024-04-03T17:40:02.0663479Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8588b615d0b6fe29.xml (deflated 42%) 2024-04-03T17:40:02.0665131Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8dc12f8b120cb4cb.xml (deflated 42%) 2024-04-03T17:40:02.0666771Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cd0d2c3e86b69a16.xml (deflated 42%) 2024-04-03T17:40:02.0668424Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c3f2ec03f21f5eb.xml (deflated 42%) 2024-04-03T17:40:02.0670279Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3041d238fcfb043f.xml (deflated 36%) 2024-04-03T17:40:02.0671919Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1305f9c539f01d4b.xml (deflated 42%) 2024-04-03T17:40:02.0673558Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fc395c7246b59acf.xml (deflated 42%) 2024-04-03T17:40:02.0675198Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-569584e3462aa4b7.xml (deflated 42%) 2024-04-03T17:40:02.0676824Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-97ba285cd6da0e7d.xml (deflated 35%) 2024-04-03T17:40:02.0678545Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-23e60a074ee414aa.xml (deflated 42%) 2024-04-03T17:40:02.0680180Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e954c5b983941115.xml (deflated 43%) 2024-04-03T17:40:02.0681892Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ef092313f84d24b6.xml (deflated 42%) 2024-04-03T17:40:02.0683521Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5a457df2d7281ac2.xml (deflated 42%) 2024-04-03T17:40:02.0685161Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4902b17a56f75216.xml (deflated 42%) 2024-04-03T17:40:02.0686801Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fe16caecb40572d6.xml (deflated 44%) 2024-04-03T17:40:02.0688456Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-65bbb2bdc3dc9e29.xml (deflated 42%) 2024-04-03T17:40:02.0690092Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e43be00a7c54f289.xml (deflated 44%) 2024-04-03T17:40:02.0691737Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bf9ab1f88223a05b.xml (deflated 36%) 2024-04-03T17:40:02.0693379Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fe9d75244447975c.xml (deflated 37%) 2024-04-03T17:40:02.0695015Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6619b1964770eb77.xml (deflated 43%) 2024-04-03T17:40:02.0696633Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e14c694f51912b91.xml (deflated 37%) 2024-04-03T17:40:02.0698276Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e92bbfb1bf481211.xml (deflated 37%) 2024-04-03T17:40:02.0699923Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d10eb039c0aa8a23.xml (deflated 43%) 2024-04-03T17:40:02.0701573Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a9b8112e0c60ca1.xml (deflated 36%) 2024-04-03T17:40:02.0703200Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-260a57d8d3c6f4a7.xml (deflated 36%) 2024-04-03T17:40:02.0704835Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96945e67b8571382.xml (deflated 37%) 2024-04-03T17:40:02.0706474Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-765b9fcc236ca7a4.xml (deflated 36%) 2024-04-03T17:40:02.0708192Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ede1164c7c61468.xml (deflated 36%) 2024-04-03T17:40:02.0709975Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2fe408d9270e90a3.xml (deflated 45%) 2024-04-03T17:40:02.0711629Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f5b7f7352b040dba.xml (deflated 43%) 2024-04-03T17:40:02.0713283Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1baddfe92c858561.xml (deflated 43%) 2024-04-03T17:40:02.0714921Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5acecd76651b2f7d.xml (deflated 43%) 2024-04-03T17:40:02.0716557Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4bf83d426fd827f9.xml (deflated 42%) 2024-04-03T17:40:02.0718278Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6394e7b048970486.xml (deflated 36%) 2024-04-03T17:40:02.0719912Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-286b57550c23a246.xml (deflated 35%) 2024-04-03T17:40:02.0721624Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5f3d7f215751bd21.xml (deflated 36%) 2024-04-03T17:40:02.0723271Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8be5d47e9b527fd2.xml (deflated 36%) 2024-04-03T17:40:02.0724907Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6e534284e19d55bb.xml (deflated 43%) 2024-04-03T17:40:02.0726537Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b8692469418d679.xml (deflated 43%) 2024-04-03T17:40:02.0728170Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-35a88610ac5cf584.xml (deflated 42%) 2024-04-03T17:40:02.0729804Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8bfb717659487ea3.xml (deflated 41%) 2024-04-03T17:40:02.0731437Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2ba42f4322003acc.xml (deflated 42%) 2024-04-03T17:40:02.0733077Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf46e1f2ed2f7881.xml (deflated 35%) 2024-04-03T17:40:02.0734717Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cbc8d2087c9a0ebd.xml (deflated 36%) 2024-04-03T17:40:02.0736361Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a6332013e0f5e64.xml (deflated 36%) 2024-04-03T17:40:02.0738001Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-88f8809134500bf7.xml (deflated 43%) 2024-04-03T17:40:02.0739633Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-54823124cae894db.xml (deflated 43%) 2024-04-03T17:40:02.0741261Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9ed060aaf7f49d6d.xml (deflated 43%) 2024-04-03T17:40:02.0742907Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-998766df7eb78185.xml (deflated 37%) 2024-04-03T17:40:02.0744536Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8425085372737a17.xml (deflated 37%) 2024-04-03T17:40:02.0746175Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aab8616bde4c4679.xml (deflated 41%) 2024-04-03T17:40:02.0747888Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0104748f7b225252.xml (deflated 42%) 2024-04-03T17:40:02.0749531Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f6994f5a49a2b314.xml (deflated 41%) 2024-04-03T17:40:02.0751278Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e00118522f64276c.xml (deflated 42%) 2024-04-03T17:40:02.0753037Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ac725a646a3b49d1.xml (deflated 37%) 2024-04-03T17:40:02.0754681Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ec23e7f757c6dca.xml (deflated 36%) 2024-04-03T17:40:02.0756401Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f13e8039aec0821f.xml (deflated 36%) 2024-04-03T17:40:02.0758054Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2162ec9e4d123ca.xml (deflated 36%) 2024-04-03T17:40:02.0759706Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dd23c582fc16d706.xml (deflated 43%) 2024-04-03T17:40:02.0761418Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b49c56a1c15bb203.xml (deflated 45%) 2024-04-03T17:40:02.0763056Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f3564007f5ad9a15.xml (deflated 36%) 2024-04-03T17:40:02.0764700Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d27270c8a1f50418.xml (deflated 36%) 2024-04-03T17:40:02.0766346Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-073ab30f92aec694.xml (deflated 36%) 2024-04-03T17:40:02.0767972Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c8299b891b9533fa.xml (deflated 36%) 2024-04-03T17:40:02.0769600Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-78c43ddc7d076863.xml (deflated 36%) 2024-04-03T17:40:02.0771237Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c5728837a948d69c.xml (deflated 36%) 2024-04-03T17:40:02.0772877Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dfd9291b94aa8edb.xml (deflated 35%) 2024-04-03T17:40:02.0774506Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4663d7a77c8055a3.xml (deflated 36%) 2024-04-03T17:40:02.0776147Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1ed6ab0f3b4e70a6.xml (deflated 36%) 2024-04-03T17:40:02.0777789Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5533946402cbb963.xml (deflated 36%) 2024-04-03T17:40:02.0779524Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6bd955f74f1e2522.xml (deflated 36%) 2024-04-03T17:40:02.0781155Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5907736dcb3dfa3f.xml (deflated 36%) 2024-04-03T17:40:02.0782790Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d882dd185462e6a3.xml (deflated 36%) 2024-04-03T17:40:02.0784429Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34a525fe3087e9fd.xml (deflated 35%) 2024-04-03T17:40:02.0786065Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b04bd69b553556d9.xml (deflated 36%) 2024-04-03T17:40:02.0787781Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cde08544a07d0af3.xml (deflated 36%) 2024-04-03T17:40:02.0789415Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ab7c0ad724f47f86.xml (deflated 36%) 2024-04-03T17:40:02.0791142Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-419c7017002a8aa9.xml (deflated 36%) 2024-04-03T17:40:02.0792773Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-605aecc450236025.xml (deflated 42%) 2024-04-03T17:40:02.0794393Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d14ca0db673669b.xml (deflated 44%) 2024-04-03T17:40:02.0796163Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-943031dde1ac6e8b.xml (deflated 44%) 2024-04-03T17:40:02.0797822Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dcff80e90d5507f2.xml (deflated 43%) 2024-04-03T17:40:02.0799457Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-71139e36f14e323c.xml (deflated 44%) 2024-04-03T17:40:02.0801147Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-25641533ac46cbcd.xml (deflated 44%) 2024-04-03T17:40:02.0802785Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b46d9fb17fa13e48.xml (deflated 44%) 2024-04-03T17:40:02.0804432Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aa855fec123f2d60.xml (deflated 45%) 2024-04-03T17:40:02.0806078Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-938463ac68d42026.xml (deflated 44%) 2024-04-03T17:40:02.0807709Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3deef803d9e9f06a.xml (deflated 45%) 2024-04-03T17:40:02.0809354Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e2a64b9f9dc1c33.xml (deflated 43%) 2024-04-03T17:40:02.0810989Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af55b4e329936966.xml (deflated 36%) 2024-04-03T17:40:02.0812627Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0bff83f03a501f12.xml (deflated 57%) 2024-04-03T17:40:02.0814253Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2843df6574e16a3.xml (deflated 42%) 2024-04-03T17:40:02.0815893Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e029c0aab10e553a.xml (deflated 36%) 2024-04-03T17:40:02.0817544Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1cebc41da0a86efc.xml (deflated 36%) 2024-04-03T17:40:02.0819187Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7c6058297f288ad2.xml (deflated 42%) 2024-04-03T17:40:02.0820800Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c5312395b5ee865.xml (deflated 42%) 2024-04-03T17:40:02.0822442Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d46088b0c3316ac8.xml (deflated 37%) 2024-04-03T17:40:02.0824077Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-880cddfe71e25aa1.xml (deflated 47%) 2024-04-03T17:40:02.0825794Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5060590889ac40df.xml (deflated 47%) 2024-04-03T17:40:02.0827421Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f57b1631fac3f3cd.xml (deflated 47%) 2024-04-03T17:40:02.0829061Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-191ce5cbecb54ee7.xml (deflated 42%) 2024-04-03T17:40:02.0830792Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-87e1b1c50b59d0fc.xml (deflated 42%) 2024-04-03T17:40:02.0832430Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6355a6f97113d6da.xml (deflated 42%) 2024-04-03T17:40:02.0834046Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2cc0067483551bfb.xml (deflated 42%) 2024-04-03T17:40:02.0835759Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d08bf44578d307f6.xml (deflated 36%) 2024-04-03T17:40:02.0837402Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ab0108492024d88.xml (deflated 43%) 2024-04-03T17:40:02.0839038Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-994309a45b5a90f2.xml (deflated 42%) 2024-04-03T17:40:02.0840734Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8187a70d2fca5e9a.xml (deflated 42%) 2024-04-03T17:40:02.0842375Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a62ff22239998748.xml (deflated 36%) 2024-04-03T17:40:02.0844015Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-41c6ee80456d48f0.xml (deflated 42%) 2024-04-03T17:40:02.0845656Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e99265b6e76a0097.xml (deflated 43%) 2024-04-03T17:40:02.0847281Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-123bfe270c8c9049.xml (deflated 43%) 2024-04-03T17:40:02.0848918Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e61d121b700cb28.xml (deflated 43%) 2024-04-03T17:40:02.0850561Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ed093ca5e81081b2.xml (deflated 42%) 2024-04-03T17:40:02.0852195Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-082402ddae301311.xml (deflated 44%) 2024-04-03T17:40:02.0853825Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c77c08c01a505a3c.xml (deflated 42%) 2024-04-03T17:40:02.0855473Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-463cc70a4480d163.xml (deflated 43%) 2024-04-03T17:40:02.0857114Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5a2b572e94da0360.xml (deflated 35%) 2024-04-03T17:40:02.0858747Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-248a172dee6c38fd.xml (deflated 36%) 2024-04-03T17:40:02.0860371Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dbdc1e37b4d140b4.xml (deflated 43%) 2024-04-03T17:40:02.0862019Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-14f34cae893fb8dc.xml (deflated 36%) 2024-04-03T17:40:02.0863662Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1784fe3835265fa.xml (deflated 36%) 2024-04-03T17:40:02.0865372Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-54b5a93b102119b7.xml (deflated 43%) 2024-04-03T17:40:02.0867004Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-514232c212f3052a.xml (deflated 36%) 2024-04-03T17:40:02.0868646Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-71aba88a3e87dd30.xml (deflated 36%) 2024-04-03T17:40:02.0870374Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-63dab8496e7959d3.xml (deflated 37%) 2024-04-03T17:40:02.0871999Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3c0390e175acfe0f.xml (deflated 36%) 2024-04-03T17:40:02.0873709Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-610a06eca93c2cca.xml (deflated 35%) 2024-04-03T17:40:02.0875358Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-397e605e69bec369.xml (deflated 45%) 2024-04-03T17:40:02.0877000Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-678e395d1a0cc356.xml (deflated 43%) 2024-04-03T17:40:02.0878628Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f617ecf7646bed1c.xml (deflated 43%) 2024-04-03T17:40:02.0880309Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-588bde2181583972.xml (deflated 43%) 2024-04-03T17:40:02.0881961Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c67b82b1b110124a.xml (deflated 42%) 2024-04-03T17:40:02.0883595Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-11923564dac466cf.xml (deflated 36%) 2024-04-03T17:40:02.0885224Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2db4fc8e05674e1b.xml (deflated 36%) 2024-04-03T17:40:02.0886864Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eff218085ac8f75b.xml (deflated 36%) 2024-04-03T17:40:02.0888506Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f8ecee1827ed085.xml (deflated 36%) 2024-04-03T17:40:02.0890154Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b21dddc25fd387f4.xml (deflated 43%) 2024-04-03T17:40:02.0891785Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-67b31e597a09758e.xml (deflated 43%) 2024-04-03T17:40:02.0893431Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d715875294c54de2.xml (deflated 41%) 2024-04-03T17:40:02.0895071Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3320852887f802b0.xml (deflated 41%) 2024-04-03T17:40:02.0896710Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1e66df71bece1d4.xml (deflated 42%) 2024-04-03T17:40:02.0898507Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70e3214df007b36e.xml (deflated 36%) 2024-04-03T17:40:02.0900148Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-54c08b945e2be6c3.xml (deflated 36%) 2024-04-03T17:40:02.0901780Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-461ed6c65c7519d3.xml (deflated 36%) 2024-04-03T17:40:02.0903418Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3669044040c11f1d.xml (deflated 42%) 2024-04-03T17:40:02.0905139Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-85a0188f2eb05d1c.xml (deflated 36%) 2024-04-03T17:40:02.0906785Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e999ca4e44160561.xml (deflated 41%) 2024-04-03T17:40:02.0908410Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-35ef26fbfbe71cb8.xml (deflated 43%) 2024-04-03T17:40:02.0910256Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-682d89068a2b0730.xml (deflated 42%) 2024-04-03T17:40:02.0911900Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2ba06cab2787ba39.xml (deflated 36%) 2024-04-03T17:40:02.0913618Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c89ab2566b90ccd.xml (deflated 36%) 2024-04-03T17:40:02.0915264Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1927b18e3ec7ea9.xml (deflated 35%) 2024-04-03T17:40:02.0916902Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b2b95b5709c028eb.xml (deflated 36%) 2024-04-03T17:40:02.0918546Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4b3a9c702a89f5e4.xml (deflated 35%) 2024-04-03T17:40:02.0920185Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-02e6b3a1c04f551c.xml (deflated 45%) 2024-04-03T17:40:02.0921880Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4bb090b934840465.xml (deflated 36%) 2024-04-03T17:40:02.0923528Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fa2b4235e2fbcfc1.xml (deflated 36%) 2024-04-03T17:40:02.0925176Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-26c400181adacc41.xml (deflated 36%) 2024-04-03T17:40:02.0926815Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-28366adca8a90b45.xml (deflated 36%) 2024-04-03T17:40:02.0928436Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9959cab5fd2446d5.xml (deflated 36%) 2024-04-03T17:40:02.0930075Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f4553a6ab974b203.xml (deflated 35%) 2024-04-03T17:40:02.0931714Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-47ce7beb8c45e98d.xml (deflated 36%) 2024-04-03T17:40:02.0933368Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f99864ac67a9c273.xml (deflated 42%) 2024-04-03T17:40:02.0934993Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8827fe2753a1fc63.xml (deflated 43%) 2024-04-03T17:40:02.0936638Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cc6f17e2c622fff5.xml (deflated 44%) 2024-04-03T17:40:02.0938269Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c30215652944071.xml (deflated 44%) 2024-04-03T17:40:02.0939902Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-91d553551906472c.xml (deflated 45%) 2024-04-03T17:40:02.0941521Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bdf068d95bdc9704.xml (deflated 45%) 2024-04-03T17:40:02.0943260Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3d35f35b186b7856.xml (deflated 43%) 2024-04-03T17:40:02.0944900Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-615e9a42fc481ec6.xml (deflated 35%) 2024-04-03T17:40:02.0946548Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a4ae8818fb2a2019.xml (deflated 42%) 2024-04-03T17:40:02.0948177Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-01966e7dd9e4c5b5.xml (deflated 36%) 2024-04-03T17:40:02.0949901Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7acf34b442680733.xml (deflated 36%) 2024-04-03T17:40:02.0951541Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-26c3b6bf00d274c8.xml (deflated 41%) 2024-04-03T17:40:02.0953263Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0fbe8569aff0ce2a.xml (deflated 42%) 2024-04-03T17:40:02.0954894Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4b2ccb190dc41921.xml (deflated 42%) 2024-04-03T17:40:02.0956536Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4adfd9b4fb2f950a.xml (deflated 35%) 2024-04-03T17:40:02.0958173Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-52d5b481935ee503.xml (deflated 42%) 2024-04-03T17:40:02.0959817Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f0b0ab10b8c1b612.xml (deflated 42%) 2024-04-03T17:40:02.0961511Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fc3dd76b6e22d77f.xml (deflated 42%) 2024-04-03T17:40:02.0963159Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bf1bc3f149105979.xml (deflated 42%) 2024-04-03T17:40:02.0964799Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1166723b9a780413.xml (deflated 42%) 2024-04-03T17:40:02.0966427Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8809e04e90023874.xml (deflated 42%) 2024-04-03T17:40:02.0968044Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-97ddacc255d3c771.xml (deflated 43%) 2024-04-03T17:40:02.0969687Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f05dbc17cff4b7f8.xml (deflated 42%) 2024-04-03T17:40:02.0971330Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-336e90eaa0d5debd.xml (deflated 42%) 2024-04-03T17:40:02.0972975Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-467197f49b4cf1da.xml (deflated 42%) 2024-04-03T17:40:02.0974598Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a80814eb9e6cd504.xml (deflated 42%) 2024-04-03T17:40:02.0976237Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2d64ed0b4a3c10e1.xml (deflated 43%) 2024-04-03T17:40:02.0977877Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7681e10adb39f5eb.xml (deflated 42%) 2024-04-03T17:40:02.0979505Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b47690dd9f276eb.xml (deflated 43%) 2024-04-03T17:40:02.0981119Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15a5a885e51f6822.xml (deflated 42%) 2024-04-03T17:40:02.0982840Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-53eb24e2bb5c3f70.xml (deflated 42%) 2024-04-03T17:40:02.0984481Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b7ff7bd095bfdd9c.xml (deflated 42%) 2024-04-03T17:40:02.0986131Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-57c0eb4816bbe23b.xml (deflated 42%) 2024-04-03T17:40:02.0987750Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-115f2983235fad77.xml (deflated 42%) 2024-04-03T17:40:02.0989383Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e1509070773aca41.xml (deflated 42%) 2024-04-03T17:40:02.0991186Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-83f2b290a2e29e7b.xml (deflated 43%) 2024-04-03T17:40:02.0992833Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a24acd529390a4a5.xml (deflated 43%) 2024-04-03T17:40:02.0994466Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-abb93bdab701b387.xml (deflated 42%) 2024-04-03T17:40:02.0996114Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-74d01b63d2af339e.xml (deflated 36%) 2024-04-03T17:40:02.0997750Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c83e787a21d582e9.xml (deflated 36%) 2024-04-03T17:40:02.0999388Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1519083fab243ba8.xml (deflated 35%) 2024-04-03T17:40:02.1001095Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1d79465f3b7af2b.xml (deflated 36%) 2024-04-03T17:40:02.1002736Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-10f5342463b43a34.xml (deflated 42%) 2024-04-03T17:40:02.1004369Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d509ac1422b79c72.xml (deflated 35%) 2024-04-03T17:40:02.1006003Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bc447ad2bc5e4c95.xml (deflated 45%) 2024-04-03T17:40:02.1007632Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13fcf3136bb2b847.xml (deflated 45%) 2024-04-03T17:40:02.1009266Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ee89e5321bb47de2.xml (deflated 45%) 2024-04-03T17:40:02.1010917Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-abbf6296d24bac31.xml (deflated 42%) 2024-04-03T17:40:02.1012565Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d9a3e950b5134a6d.xml (deflated 42%) 2024-04-03T17:40:02.1014201Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eac6fb98fdae80b0.xml (deflated 42%) 2024-04-03T17:40:02.1015848Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f82f78976a6496c.xml (deflated 42%) 2024-04-03T17:40:02.1017478Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-263837549ef3e677.xml (deflated 42%) 2024-04-03T17:40:02.1019113Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-caf939493e4d133d.xml (deflated 41%) 2024-04-03T17:40:02.1020731Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-56547570f9f89c84.xml (deflated 36%) 2024-04-03T17:40:02.1022452Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf695b7b800cb0f7.xml (deflated 35%) 2024-04-03T17:40:02.1024096Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b1c95f4c4b79cbc.xml (deflated 35%) 2024-04-03T17:40:02.1025745Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-90bc9f3d60dfbc21.xml (deflated 36%) 2024-04-03T17:40:02.1027368Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2d0180a39c5900aa.xml (deflated 45%) 2024-04-03T17:40:02.1029005Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1788cc0775d47ac0.xml (deflated 43%) 2024-04-03T17:40:02.1030822Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-feec97bae5cbce22.xml (deflated 36%) 2024-04-03T17:40:02.1032475Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9716f1ce7040f281.xml (deflated 36%) 2024-04-03T17:40:02.1034098Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-553cf73879ecabff.xml (deflated 36%) 2024-04-03T17:40:02.1035749Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7018a20ae91e1d05.xml (deflated 42%) 2024-04-03T17:40:02.1037396Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e2838e4c79e00356.xml (deflated 36%) 2024-04-03T17:40:02.1039019Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f2415440611b623.xml (deflated 36%) 2024-04-03T17:40:02.1040709Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fb2ae73dae462603.xml (deflated 35%) 2024-04-03T17:40:02.1042356Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-425d38f5a2188c7f.xml (deflated 36%) 2024-04-03T17:40:02.1044000Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d8964ec236c936a6.xml (deflated 36%) 2024-04-03T17:40:02.1045634Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a68f9d87e1c7389a.xml (deflated 36%) 2024-04-03T17:40:02.1047274Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6f5ea983975a8e2d.xml (deflated 42%) 2024-04-03T17:40:02.1048917Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2629cbdc5d88b5a6.xml (deflated 42%) 2024-04-03T17:40:02.1050569Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e09774c0a45f3050.xml (deflated 43%) 2024-04-03T17:40:02.1052198Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4897e46b804fb88c.xml (deflated 42%) 2024-04-03T17:40:02.1053837Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7b6fe8d16162674a.xml (deflated 36%) 2024-04-03T17:40:02.1055479Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b2e51b96e4a5a17a.xml (deflated 41%) 2024-04-03T17:40:02.1057121Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-37cdd45a16f4ece1.xml (deflated 42%) 2024-04-03T17:40:02.1058754Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-87f50d046ed7794a.xml (deflated 42%) 2024-04-03T17:40:02.1060399Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8d5f6cff4e88fcc7.xml (deflated 36%) 2024-04-03T17:40:02.1062135Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-44c7d4d10a32c3cf.xml (deflated 36%) 2024-04-03T17:40:02.1063785Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9cb0f1ee0d29bc06.xml (deflated 36%) 2024-04-03T17:40:02.1065415Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d110247598414855.xml (deflated 36%) 2024-04-03T17:40:02.1067054Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cc7daeee8ceb7a39.xml (deflated 36%) 2024-04-03T17:40:02.1068709Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1eabdbfa0bc3af63.xml (deflated 45%) 2024-04-03T17:40:02.1070541Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-86ec6ba7f5daa891.xml (deflated 36%) 2024-04-03T17:40:02.1072277Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5a21c61c8f5f47fe.xml (deflated 36%) 2024-04-03T17:40:02.1073921Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3b032d21bac453ad.xml (deflated 36%) 2024-04-03T17:40:02.1075560Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-08f519892558b222.xml (deflated 35%) 2024-04-03T17:40:02.1077198Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-20d92e0d1283c4b9.xml (deflated 36%) 2024-04-03T17:40:02.1078817Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a0661936a41e327e.xml (deflated 36%) 2024-04-03T17:40:02.1080554Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-afc60dfc30c62bcb.xml (deflated 36%) 2024-04-03T17:40:02.1082210Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8bc3e3f2f5576196.xml (deflated 42%) 2024-04-03T17:40:02.1083865Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bded5ca8f16be97b.xml (deflated 43%) 2024-04-03T17:40:02.1085498Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a060ab557508496.xml (deflated 44%) 2024-04-03T17:40:02.1087125Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-847832db02826076.xml (deflated 44%) 2024-04-03T17:40:02.1088767Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6e43fe1603ebba8c.xml (deflated 44%) 2024-04-03T17:40:02.1090417Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0ab4f8017e65541a.xml (deflated 45%) 2024-04-03T17:40:02.1092048Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aef80f5d6af6badc.xml (deflated 43%) 2024-04-03T17:40:02.1093698Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cdde4b0513eff007.xml (deflated 35%) 2024-04-03T17:40:02.1095340Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-29c2990ef8a3ef46.xml (deflated 42%) 2024-04-03T17:40:02.1096973Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c610d957c8a8be0d.xml (deflated 36%) 2024-04-03T17:40:02.1098604Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6394d2b912cbd723.xml (deflated 36%) 2024-04-03T17:40:02.1100366Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-501f952cbb5a34e5.xml (deflated 42%) 2024-04-03T17:40:02.1102006Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51f38417e2f6c5a6.xml (deflated 42%) 2024-04-03T17:40:02.1103646Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e2160f9c6c76850a.xml (deflated 42%) 2024-04-03T17:40:02.1105276Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ecbb72147d01586.xml (deflated 36%) 2024-04-03T17:40:02.1106921Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf764b4422441052.xml (deflated 42%) 2024-04-03T17:40:02.1108625Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b8c934e217fe93f9.xml (deflated 42%) 2024-04-03T17:40:02.1110370Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1d9bf3d2cc9c7134.xml (deflated 42%) 2024-04-03T17:40:02.1112144Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a6323ba4a5bcd1e.xml (deflated 42%) 2024-04-03T17:40:02.1113794Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e0e041fc736a3cf.xml (deflated 42%) 2024-04-03T17:40:02.1115449Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e3ddf8d68528a655.xml (deflated 42%) 2024-04-03T17:40:02.1117094Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c159064d0e20dabf.xml (deflated 43%) 2024-04-03T17:40:02.1118722Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8173643c089c95d2.xml (deflated 42%) 2024-04-03T17:40:02.1120422Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-917ad6ef9721e340.xml (deflated 42%) 2024-04-03T17:40:02.1122073Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-826abfa5da613545.xml (deflated 42%) 2024-04-03T17:40:02.1123714Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c9cc4620b2024ae7.xml (deflated 42%) 2024-04-03T17:40:02.1125353Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2970695b40fe8740.xml (deflated 44%) 2024-04-03T17:40:02.1126999Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e8635a1dcf19bc6d.xml (deflated 42%) 2024-04-03T17:40:02.1128650Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-67a2a86867f71294.xml (deflated 43%) 2024-04-03T17:40:02.1130293Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-054e654b850ea6a1.xml (deflated 42%) 2024-04-03T17:40:02.1131918Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d19e374c10b71982.xml (deflated 42%) 2024-04-03T17:40:02.1133551Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-399173f0a904bdaa.xml (deflated 43%) 2024-04-03T17:40:02.1135189Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e82941c9dc43e8bb.xml (deflated 42%) 2024-04-03T17:40:02.1136827Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-42b9fa16d6839d71.xml (deflated 42%) 2024-04-03T17:40:02.1138454Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-de94e141760bb8a8.xml (deflated 42%) 2024-04-03T17:40:02.1140192Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-844b3e5cf550f3b9.xml (deflated 42%) 2024-04-03T17:40:02.1141831Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-087a1a575a058a1a.xml (deflated 43%) 2024-04-03T17:40:02.1143468Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d4b9044b04e772d6.xml (deflated 42%) 2024-04-03T17:40:02.1145092Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-935b278733a61d17.xml (deflated 36%) 2024-04-03T17:40:02.1146721Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e910622a8c73fd9.xml (deflated 36%) 2024-04-03T17:40:02.1148425Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fd58e598ec45ae13.xml (deflated 35%) 2024-04-03T17:40:02.1150172Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b3d3b8d2d70f4499.xml (deflated 36%) 2024-04-03T17:40:02.1151801Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a96d2a7fe6c6622d.xml (deflated 42%) 2024-04-03T17:40:02.1153453Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fcaeeb8b108011fb.xml (deflated 36%) 2024-04-03T17:40:02.1155106Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ae8856ecaa0b8bd7.xml (deflated 45%) 2024-04-03T17:40:02.1156749Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-45bed91816167a58.xml (deflated 45%) 2024-04-03T17:40:02.1158379Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-faf553af753dffae.xml (deflated 45%) 2024-04-03T17:40:02.1160029Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a54156e9f38d90bd.xml (deflated 42%) 2024-04-03T17:40:02.1161796Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dd181ed7dff894f7.xml (deflated 42%) 2024-04-03T17:40:02.1163550Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-43ab3578fc6625e3.xml (deflated 42%) 2024-04-03T17:40:02.1165184Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2f20466fe7716a2d.xml (deflated 42%) 2024-04-03T17:40:02.1166820Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca041041e6d1fba5.xml (deflated 42%) 2024-04-03T17:40:02.1168473Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-85f01739e8534421.xml (deflated 41%) 2024-04-03T17:40:02.1170109Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd0db2508d1978d9.xml (deflated 36%) 2024-04-03T17:40:02.1171729Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-750079e29f396995.xml (deflated 36%) 2024-04-03T17:40:02.1173359Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-18c314ef254fff00.xml (deflated 36%) 2024-04-03T17:40:02.1174995Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-172da700038f7996.xml (deflated 36%) 2024-04-03T17:40:02.1176630Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fcb12b715baba1b3.xml (deflated 45%) 2024-04-03T17:40:02.1178261Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4d3ba647115e9c4b.xml (deflated 43%) 2024-04-03T17:40:02.1180007Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-31c5872066ffff52.xml (deflated 37%) 2024-04-03T17:40:02.1181653Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-41bc8336abd78090.xml (deflated 36%) 2024-04-03T17:40:02.1183305Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eae33ad220f02a30.xml (deflated 36%) 2024-04-03T17:40:02.1184933Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-986ee94a6be13039.xml (deflated 43%) 2024-04-03T17:40:02.1186574Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-95e90b6266c3093a.xml (deflated 36%) 2024-04-03T17:40:02.1188286Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fddef57f600da4f1.xml (deflated 36%) 2024-04-03T17:40:02.1190019Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2978e422931a6466.xml (deflated 36%) 2024-04-03T17:40:02.1191649Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ccba7e7df2bb2ab3.xml (deflated 35%) 2024-04-03T17:40:02.1193297Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-71f1d152bf8ac715.xml (deflated 36%) 2024-04-03T17:40:02.1194939Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ca606dd79b5862a.xml (deflated 36%) 2024-04-03T17:40:02.1196562Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f72d1e45b04e23d3.xml (deflated 42%) 2024-04-03T17:40:02.1198213Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b2f99c3414c5d71.xml (deflated 42%) 2024-04-03T17:40:02.1199859Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ce53f16a107de6e.xml (deflated 43%) 2024-04-03T17:40:02.1201573Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f715ff39226ad560.xml (deflated 42%) 2024-04-03T17:40:02.1203496Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-7749a459896dad2e.xml (deflated 38%) 2024-04-03T17:40:02.1205713Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-7a0b3c053ea40cbe.xml (deflated 38%) 2024-04-03T17:40:02.1207932Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-28e62c8053cd4ce6.xml (deflated 48%) 2024-04-03T17:40:02.1210148Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-534a7e35e731c5cc.xml (deflated 48%) 2024-04-03T17:40:02.1212360Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-4c004ae7830ab9fa.xml (deflated 48%) 2024-04-03T17:40:02.1214556Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-bf34903e426a70b5.xml (deflated 48%) 2024-04-03T17:40:02.1216763Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-940cac85b7bc3ff1.xml (deflated 38%) 2024-04-03T17:40:02.1218980Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-28be2c5e37c66858.xml (deflated 38%) 2024-04-03T17:40:02.1221292Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-e97e532a2e0edce3.xml (deflated 48%) 2024-04-03T17:40:02.1223488Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-c5021aa006393774.xml (deflated 48%) 2024-04-03T17:40:02.1225692Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-2feb9e4ec50d057b.xml (deflated 48%) 2024-04-03T17:40:02.1227901Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-c99f0d71e42b61df.xml (deflated 48%) 2024-04-03T17:40:02.1254297Z ##[group]Run # Remove any previous test reports if they exist 2024-04-03T17:40:02.1254952Z # Remove any previous test reports if they exist 2024-04-03T17:40:02.1255437Z rm -f logs-*.zip 2024-04-03T17:40:02.1256093Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-04-03T17:40:02.1257019Z # so check to see if the file exists first 2024-04-03T17:40:02.1257491Z if [ -f 'usage_log.txt' ]; then 2024-04-03T17:40:02.1257999Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-04-03T17:40:02.1258468Z fi 2024-04-03T17:40:02.1258791Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-04-03T17:40:02.1259343Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-04-03T17:40:02.1259815Z fi 2024-04-03T17:40:02.1267185Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T17:40:02.1267661Z env: 2024-04-03T17:40:02.1267914Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:40:02.1268527Z DOCKER_CONTAINER_ID: 4c1cfaf4a8d048d787858cf2354817432c864f82c94c156c2f90005f40fddbdd 2024-04-03T17:40:02.1269247Z FILE_SUFFIX: test-distributed-2-2-linux.2xlarge_ 2024-04-03T17:40:02.1269845Z ##[endgroup] 2024-04-03T17:40:02.1557476Z adding: usage_log.txt (deflated 96%) 2024-04-03T17:40:02.1850156Z adding: test/test-reports/distributed.test_distributed_spawn_3.3_c5d5d8d4291dfc47_.log (deflated 96%) 2024-04-03T17:40:02.1852518Z adding: test/test-reports/distributed.checkpoint.test_nested_dict_1.1_6dceaf3748e07c38_.log (deflated 58%) 2024-04-03T17:40:02.1854889Z adding: test/test-reports/distributed.test_distributed_spawn_3.3_8d0b48a9ae1d9492_.log (deflated 12%) 2024-04-03T17:40:02.1857224Z adding: test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_4227686bc69b9cbc_.log (deflated 55%) 2024-04-03T17:40:02.1859495Z adding: test/test-reports/distributed.test_collective_utils_1.1_c5fe5238939fce8e_.log (stored 0%) 2024-04-03T17:40:02.1861733Z adding: test/test-reports/distributed._composable.test_contract_1.1_4695269369407fe2_.log (deflated 64%) 2024-04-03T17:40:02.1863943Z adding: test/test-reports/distributed._tensor.test_math_ops_1.1_d8a43948892d2350_.log (deflated 86%) 2024-04-03T17:40:02.1866100Z adding: test/test-reports/distributed.checkpoint.test_traverse_1.1_b180125bb6150f25_.log (deflated 71%) 2024-04-03T17:40:02.1868289Z adding: test/test-reports/distributed.fsdp.test_fsdp_apply_1.1_56ded0589b8d0e0d_.log (deflated 63%) 2024-04-03T17:40:02.1870772Z adding: test/test-reports/distributed.fsdp.test_fsdp_fx_1.1_ad69edc1ae019671_.log (deflated 53%) 2024-04-03T17:40:02.1872800Z adding: test/test-reports/distributed._spmd.test_tracing_1.1_6fd9dbacc2d7d792_.log (stored 0%) 2024-04-03T17:40:02.1874902Z adding: test/test-reports/distributed._spmd.test_transformation_1.1_81ba4f1dc712fdeb_.log (stored 0%) 2024-04-03T17:40:02.1877044Z adding: test/test-reports/distributed.fsdp.test_fsdp_input_1.1_22b7dbc91fcec567_.log (deflated 58%) 2024-04-03T17:40:02.1879407Z adding: test/test-reports/distributed._tensor.test_init_1.1_702115d1e5b7f6ee_.log (deflated 69%) 2024-04-03T17:40:02.1881701Z adding: test/test-reports/distributed.test_distributed_spawn_3.3_22204db9bce4e8a4_.log (deflated 96%) 2024-04-03T17:40:02.1884108Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_6f2742ae38126607_.log (deflated 72%) 2024-04-03T17:40:02.1886494Z adding: test/test-reports/distributed._tensor.test_matrix_ops_1.1_14155b64e6a156a1_.log (deflated 89%) 2024-04-03T17:40:02.1888856Z adding: test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_382a48f1e7fdd752_.log (deflated 52%) 2024-04-03T17:40:02.1891272Z adding: test/test-reports/distributed._tensor.debug.test_comm_mode_1.1_2718bdb55ceb6146_.log (deflated 56%) 2024-04-03T17:40:02.1893684Z adding: test/test-reports/distributed.checkpoint.test_compatibility_1.1_a7c752493a17bc0d_.log (deflated 59%) 2024-04-03T17:40:02.1896000Z adding: test/test-reports/distributed.test_c10d_pypg_1.1_cc5e0d3e5c55e44b_.log (deflated 90%) 2024-04-03T17:40:02.1898272Z adding: test/test-reports/distributed.checkpoint.fsdp.test_fsdp_dsd_1.1_ce5df97814a5d9f9_.log (deflated 62%) 2024-04-03T17:40:02.1900579Z adding: test/test-reports/distributed._tensor.test_utils_1.1_d0b4df22fb759b6e_.log (deflated 74%) 2024-04-03T17:40:02.1902795Z adding: test/test-reports/distributed._tensor.test_common_rules_1.1_6af0d3df98285343_.log (deflated 76%) 2024-04-03T17:40:02.1905020Z adding: test/test-reports/distributed.fsdp.test_fsdp_flatten_params_1.1_126f9ddf5171f8ad_.log (deflated 80%) 2024-04-03T17:40:02.1907359Z adding: test/test-reports/distributed._composable.test_checkpoint_1.1_cca6209a41e6ec05_.log (deflated 67%) 2024-04-03T17:40:02.1909707Z adding: test/test-reports/distributed.checkpoint.test_utils_1.1_8c2160cf0a44936f_.log (deflated 64%) 2024-04-03T17:40:02.1911965Z adding: test/test-reports/distributed._tensor.test_dtensor_compile_1.1_7f68c87ffc75bd4b_.log (deflated 90%) 2024-04-03T17:40:02.1914213Z adding: test/test-reports/distributed._tensor.test_view_ops_1.1_f8669f1e375bd7c4_.log (deflated 81%) 2024-04-03T17:40:02.1916431Z adding: test/test-reports/distributed.test_c10d_functional_native_1.1_d4ed45433d82ad20_.log (deflated 89%) 2024-04-03T17:40:02.1918570Z adding: test/test-reports/distributed.test_functional_api_1.1_d9807308dbc8baef_.log (deflated 92%) 2024-04-03T17:40:02.1920851Z adding: test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_1d3aa4e78640342b_.log (deflated 51%) 2024-04-03T17:40:02.1923182Z adding: test/test-reports/distributed.fsdp.test_fsdp_ignored_modules_1.1_ebf12540cb17e94a_.log (deflated 85%) 2024-04-03T17:40:02.1925542Z adding: test/test-reports/distributed.checkpoint.test_save_load_api_1.1_0ea29ea45c118ffa_.log (deflated 53%) 2024-04-03T17:40:02.1927900Z adding: test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_1b1e2f78c89a6fb0_.log (deflated 77%) 2024-04-03T17:40:02.3050609Z adding: test/test-reports/distributed._tensor.test_dtensor_ops_1.1_78adfb662063b683_.log (deflated 96%) 2024-04-03T17:40:02.3052840Z adding: test/test-reports/distributed.fsdp.test_fsdp_uneven_1.1_144fd96c7f1d0440_.log (deflated 54%) 2024-04-03T17:40:02.3055031Z adding: test/test-reports/distributed.test_distributed_spawn_1.3_78d84d0ca1565f0d_.log (deflated 12%) 2024-04-03T17:40:02.3057167Z adding: test/test-reports/distributed.test_distributed_spawn_1.3_1760b607f4684e68_.log (deflated 12%) 2024-04-03T17:40:02.3059268Z adding: test/test-reports/distributed.fsdp.test_utils_1.1_8d1c4ce8050de0fa_.log (deflated 64%) 2024-04-03T17:40:02.3072590Z adding: test/test-reports/distributed.test_distributed_spawn_1.3_29ba5b46ee91226d_.log (deflated 95%) 2024-04-03T17:40:02.3074737Z adding: test/test-reports/distributed.fsdp.test_fsdp_comm_1.1_6268768ce900e0e3_.log (deflated 81%) 2024-04-03T17:40:02.3092556Z adding: test/test-reports/distributed.test_distributed_spawn_1.3_b5f415309e51b4bd_.log (deflated 95%) 2024-04-03T17:40:02.3094866Z adding: test/test-reports/distributed.algorithms.test_join_1.1_0b3cede2e48cd9ae_.log (deflated 73%) 2024-04-03T17:40:02.3097354Z adding: test/test-reports/distributed.test_distributed_spawn_3.3_3a6a35090cdb944e_.log (deflated 12%) 2024-04-03T17:40:02.3099637Z adding: test/test-reports/distributed._tensor.test_redistribute_1.1_9c615097b35bdbf4_.log (deflated 91%) 2024-04-03T17:40:02.3102141Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_overlap_1.1_45f7177e77197ac1_.log (deflated 55%) 2024-04-03T17:40:02.3104428Z adding: test/test-reports/distributed.test_c10d_ucc_1.1_bd4ec77c32c58086_.log (stored 0%) 2024-04-03T17:40:02.3106465Z adding: test/test-reports/distributed.fsdp.test_wrap_1.1_598ef885ae7677ce_.log (deflated 90%) 2024-04-03T17:40:02.3108834Z adding: test/test-reports/distributed.tensor.parallel.test_tp_random_state_1.1_9f9a1434487b9b97_.log (deflated 55%) 2024-04-03T17:40:02.3111632Z adding: test/test-reports/distributed.checkpoint.test_hsdp_checkpoint_1.1_fc96828091851eff_.log (deflated 71%) 2024-04-03T17:40:02.3114283Z adding: test/test-reports/distributed.checkpoint.e2e.test_fine_tuning_1.1_c93330a8a6180a83_.log (deflated 53%) 2024-04-03T17:40:02.3116868Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_compile_1.1_df569d378076b0d3_.log (deflated 57%) 2024-04-03T17:40:02.3119428Z adding: test/test-reports/distributed.checkpoint.test_fsdp_model_state_1.1_cd09346898256b10_.log (deflated 62%) 2024-04-03T17:40:02.3121893Z adding: test/test-reports/distributed.checkpoint.test_state_dict_utils_1.1_3ebecad4494b433f_.log (deflated 73%) 2024-04-03T17:40:02.3124394Z adding: test/test-reports/distributed.tensor.parallel.test_fsdp_2d_parallel_1.1_2d41b63a1e23b4c6_.log (deflated 78%) 2024-04-03T17:40:02.3126921Z adding: test/test-reports/distributed.tensor.parallel.test_tp_examples_1.1_33ae65a825fbf68f_.log (deflated 87%) 2024-04-03T17:40:02.3129545Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_f814de743d600ba2_.log (deflated 68%) 2024-04-03T17:40:02.3132130Z adding: test/test-reports/distributed.fsdp.test_distributed_checkpoint_1.1_3a3316f23d073049_.log (deflated 66%) 2024-04-03T17:40:02.3134628Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_comm_1.1_9af651d7a164c21e_.log (deflated 67%) 2024-04-03T17:40:02.3137410Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_model_checkpoint_1.1_f19ae5709be18bf0_.log (deflated 73%) 2024-04-03T17:40:02.3140315Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_init_1.1_edfc122e0068c659_.log (deflated 75%) 2024-04-03T17:40:02.3142845Z adding: test/test-reports/distributed.fsdp.test_checkpoint_wrapper_1.1_6441f6d4343a8d1a_.log (deflated 74%) 2024-04-03T17:40:02.3145180Z adding: test/test-reports/distributed.fsdp.test_fsdp_grad_acc_1.1_df192f7be621c155_.log (deflated 74%) 2024-04-03T17:40:02.3147370Z adding: test/test-reports/distributed.checkpoint.test_planner_1.1_0df65cd45a29d038_.log (deflated 65%) 2024-04-03T17:40:02.3149886Z adding: test/test-reports/distributed.fsdp.test_fsdp_sharded_grad_scaler_1.1_f2b7b15cc3744c35_.log (deflated 89%) 2024-04-03T17:40:02.3152220Z adding: test/test-reports/distributed.pipeline.sync.test_pipeline_1.1_6e37c2b58fbff0b7_.log (deflated 53%) 2024-04-03T17:40:02.3154592Z adding: test/test-reports/distributed._composable.test_replicate_with_compiler_1.1_2974bd799da43978_.log (deflated 77%) 2024-04-03T17:40:02.3156935Z adding: test/test-reports/distributed.fsdp.test_fsdp_state_dict_1.1_7ff3f4d88a04542f_.log (deflated 96%) 2024-04-03T17:40:02.3159115Z adding: test/test-reports/distributed.fsdp.test_fsdp_traversal_1.1_4f5f4356f76e5148_.log (deflated 53%) 2024-04-03T17:40:02.3161505Z adding: test/test-reports/distributed.fsdp.test_fsdp_multiple_wrapping_1.1_aa57bcf528c1d1ab_.log (deflated 56%) 2024-04-03T17:40:02.3164020Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_runtime_1.1_c4aedd956e85e9cf_.log (deflated 63%) 2024-04-03T17:40:02.3166720Z adding: test/test-reports/distributed.fsdp.test_fsdp_clip_grad_norm_1.1_6011e1c571e7fae4_.log (deflated 69%) 2024-04-03T17:40:02.3168956Z adding: test/test-reports/distributed.fsdp.test_fsdp_exec_order_1.1_62f87da35cf9acff_.log (deflated 80%) 2024-04-03T17:40:02.3171376Z adding: test/test-reports/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks_1.1_4284e4eba063f87d_.log (deflated 76%) 2024-04-03T17:40:02.3173812Z adding: test/test-reports/distributed.fsdp.test_fsdp_multiple_forward_1.1_09fd75e637af4be9_.log (deflated 55%) 2024-04-03T17:40:02.3176324Z adding: test/test-reports/distributed.checkpoint.test_file_system_checkpoint_cpu_1.1_15a5985f7e646b42_.log (deflated 94%) 2024-04-03T17:40:02.3178831Z adding: test/test-reports/distributed.pipeline.sync.test_inplace_1.1_2ac9f1e9c8fba849_.log (deflated 58%) 2024-04-03T17:40:02.3181051Z adding: test/test-reports/distributed.fsdp.test_fsdp_comm_hooks_1.1_4505069e43e3c6e1_.log (deflated 89%) 2024-04-03T17:40:02.3183366Z adding: test/test-reports/distributed.pipeline.sync.test_copy_1.1_cee55925a554a231_.log (deflated 63%) 2024-04-03T17:40:02.3185793Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_b5dcf5ab27f7f30b_.log (stored 0%) 2024-04-03T17:40:02.3188277Z adding: test/test-reports/distributed.fsdp.test_hsdp_dtensor_state_dict_1.1_7c15ae9047c4e7a5_.log (deflated 79%) 2024-04-03T17:40:02.3190941Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_d4b79d84255014b8_.log (stored 0%) 2024-04-03T17:40:02.3193395Z adding: test/test-reports/distributed.pipeline.sync.skip.test_gpipe_1.1_70f63913abe31b57_.log (deflated 75%) 2024-04-03T17:40:02.3195903Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_19bd0b51fd36c3bd_.log (deflated 91%) 2024-04-03T17:40:02.3198472Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_embedding_1.1_e31eacf45c3e85ad_.log (deflated 62%) 2024-04-03T17:40:02.3201156Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_8e7b78bc95c25a27_.log (deflated 91%) 2024-04-03T17:40:02.3203691Z adding: test/test-reports/distributed.rpc.cuda.test_tensorpipe_agent_1.1_9106023c6652bd57_.log (deflated 98%) 2024-04-03T17:40:02.3205966Z adding: test/test-reports/distributed.fsdp.test_fsdp_use_orig_params_1.1_62278d94efd278c8_.log (deflated 85%) 2024-04-03T17:40:02.3218309Z adding: test/test-reports/distributed.rpc.test_tensorpipe_agent_2.3_7657fa56eb108670_.log (deflated 97%) 2024-04-03T17:40:02.3219618Z adding: test/test-reports/distributed.pipeline.sync.test_phony_1.1_fb816301847105d5_.log (deflated 60%) 2024-04-03T17:40:02.3221072Z adding: test/test-reports/distributed.test_pg_wrapper_1.1_9e48b0ebd9c2cab4_.log (deflated 93%) 2024-04-03T17:40:02.3222286Z adding: test/test-reports/distributed.test_c10d_object_collectives_1.1_39c4eb4096740940_.log (deflated 75%) 2024-04-03T17:40:02.3223608Z adding: test/test-reports/distributed.pipeline.sync.test_bugs_1.1_0666cd215a5202ad_.log (deflated 58%) 2024-04-03T17:40:02.3224943Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_embedding_bag_1.1_c25966ca0239cc11_.log (deflated 63%) 2024-04-03T17:40:02.3227007Z adding: test/test-reports/distributed.fsdp.test_fsdp_mixed_precision_1.1_93c7c9bfb0078674_.log (deflated 92%) 2024-04-03T17:40:02.3229755Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_binary_cmp_1.1_be12aeec5db0d753_.log (deflated 71%) 2024-04-03T17:40:02.3232147Z adding: test/test-reports/distributed.test_c10d_common_1.1_83a4b6dc0c98832d_.log (deflated 94%) 2024-04-03T17:40:02.3234405Z adding: test/test-reports/distributed.pipeline.sync.skip.test_tracker_1.1_f437e5b526a84669_.log (deflated 65%) 2024-04-03T17:40:02.3236916Z adding: test/test-reports/distributed.pipeline.sync.skip.test_leak_1.1_2e724b94d551734d_.log (deflated 70%) 2024-04-03T17:40:02.3239419Z adding: test/test-reports/distributed.pipeline.sync.skip.test_stash_pop_1.1_07bd87ac4f9d6309_.log (deflated 67%) 2024-04-03T17:40:02.3242032Z adding: test/test-reports/distributed.test_c10d_spawn_gloo_1.1_523842e9c3091119_.log (deflated 94%) 2024-04-03T17:40:02.3244495Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_tensor_ops_1.1_fae1d865887ce4d2_.log (deflated 71%) 2024-04-03T17:40:02.3247004Z adding: test/test-reports/distributed.pipeline.sync.test_stream_1.1_948332f82422634d_.log (deflated 79%) 2024-04-03T17:40:02.3249160Z adding: test/test-reports/distributed.test_c10d_spawn_nccl_1.1_ae651aff4a3b4061_.log (deflated 93%) 2024-04-03T17:40:02.3251333Z adding: test/test-reports/distributed.test_c10d_spawn_ucc_1.1_2141fd603bdecac5_.log (deflated 92%) 2024-04-03T17:40:02.3253513Z adding: test/test-reports/distributed.pipeline.sync.test_balance_1.1_e53cafbf15c3fd1f_.log (deflated 74%) 2024-04-03T17:40:02.3256026Z adding: test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_reshard_1.1_20eb55a5d8820481_.log (deflated 64%) 2024-04-03T17:40:02.3258671Z adding: test/test-reports/distributed.pipeline.sync.test_dependency_1.1_c12249169cdcd565_.log (deflated 65%) 2024-04-03T17:40:02.3261073Z adding: test/test-reports/distributed.pipeline.sync.skip.test_api_1.1_0602854db6b362d4_.log (deflated 58%) 2024-04-03T17:40:02.3263546Z adding: test/test-reports/distributed.pipeline.sync.test_deferred_batch_norm_1.1_f8598ef2576e2b60_.log (deflated 72%) 2024-04-03T17:40:02.3265919Z adding: test/test-reports/distributed.checkpoint.test_state_dict_1.1_2c650f09024bb5e6_.log (deflated 76%) 2024-04-03T17:40:02.3270058Z adding: test/test-reports/distributed.test_c10d_gloo_1.1_c69adfe41352467b_.log (deflated 96%) 2024-04-03T17:40:02.3271912Z adding: test/test-reports/distributed.test_c10d_nccl_1.1_5c67086c104f5154_.log (stored 0%) 2024-04-03T17:40:02.3273714Z adding: test/test-reports/distributed.elastic.utils.logging_test_1.1_ff3e28ebe613a78a_.log (deflated 58%) 2024-04-03T17:40:02.3275672Z adding: test/test-reports/distributed.elastic.utils.util_test_1.1_823d7ecff9c83a88_.log (deflated 69%) 2024-04-03T17:40:02.3340734Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-04-03T17:40:02.3341147Z with: 2024-04-03T17:40:02.3341406Z s3-bucket: gha-artifacts 2024-04-03T17:40:02.3341813Z s3-prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T17:40:02.3342265Z retention-days: 14 2024-04-03T17:40:02.3342570Z if-no-files-found: warn 2024-04-03T17:40:02.3342901Z path: test-jsons-*.zip 2024-04-03T17:40:02.3343213Z name: artifact 2024-04-03T17:40:02.3343478Z region: us-east-1 2024-04-03T17:40:02.3343750Z env: 2024-04-03T17:40:02.3343985Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:40:02.3344646Z DOCKER_CONTAINER_ID: 4c1cfaf4a8d048d787858cf2354817432c864f82c94c156c2f90005f40fddbdd 2024-04-03T17:40:02.3345309Z ##[endgroup] 2024-04-03T17:40:02.7027873Z NOTE: s3-prefix specified, ignoring name parameter 2024-04-03T17:40:02.7028977Z With the provided path, there will be 1 file uploaded 2024-04-03T17:40:02.7029799Z Uploading to s3 prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T17:40:02.8514076Z Starting upload of test-jsons-test-distributed-2-2-linux.2xlarge_.zip 2024-04-03T17:40:03.0088650Z Finished upload of test-jsons-test-distributed-2-2-linux.2xlarge_.zip 2024-04-03T17:40:03.0224290Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-04-03T17:40:03.0224706Z with: 2024-04-03T17:40:03.0224968Z s3-bucket: gha-artifacts 2024-04-03T17:40:03.0225380Z s3-prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T17:40:03.0225825Z retention-days: 14 2024-04-03T17:40:03.0226148Z if-no-files-found: error 2024-04-03T17:40:03.0226487Z path: test-reports-*.zip 2024-04-03T17:40:03.0226799Z name: artifact 2024-04-03T17:40:03.0227076Z region: us-east-1 2024-04-03T17:40:03.0227349Z env: 2024-04-03T17:40:03.0227585Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:40:03.0228177Z DOCKER_CONTAINER_ID: 4c1cfaf4a8d048d787858cf2354817432c864f82c94c156c2f90005f40fddbdd 2024-04-03T17:40:03.0228810Z ##[endgroup] 2024-04-03T17:40:03.3592879Z NOTE: s3-prefix specified, ignoring name parameter 2024-04-03T17:40:03.3594211Z With the provided path, there will be 1 file uploaded 2024-04-03T17:40:03.3594818Z Uploading to s3 prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T17:40:03.3628915Z Starting upload of test-reports-test-distributed-2-2-linux.2xlarge_.zip 2024-04-03T17:40:03.6776820Z Finished upload of test-reports-test-distributed-2-2-linux.2xlarge_.zip 2024-04-03T17:40:03.6933330Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-04-03T17:40:03.6933750Z with: 2024-04-03T17:40:03.6934021Z s3-bucket: gha-artifacts 2024-04-03T17:40:03.6934434Z s3-prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T17:40:03.6934879Z retention-days: 14 2024-04-03T17:40:03.6935202Z if-no-files-found: ignore 2024-04-03T17:40:03.6935542Z path: logs-*.zip 2024-04-03T17:40:03.6935821Z name: artifact 2024-04-03T17:40:03.6936104Z region: us-east-1 2024-04-03T17:40:03.6936383Z env: 2024-04-03T17:40:03.6936628Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:40:03.6937343Z DOCKER_CONTAINER_ID: 4c1cfaf4a8d048d787858cf2354817432c864f82c94c156c2f90005f40fddbdd 2024-04-03T17:40:03.6938009Z ##[endgroup] 2024-04-03T17:40:04.0354682Z NOTE: s3-prefix specified, ignoring name parameter 2024-04-03T17:40:04.0355382Z With the provided path, there will be 1 file uploaded 2024-04-03T17:40:04.0356002Z Uploading to s3 prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T17:40:04.0390132Z Starting upload of logs-test-distributed-2-2-linux.2xlarge_.zip 2024-04-03T17:40:04.2463491Z Finished upload of logs-test-distributed-2-2-linux.2xlarge_.zip 2024-04-03T17:40:04.2601196Z ##[group]Run # shellcheck disable=SC2156 2024-04-03T17:40:04.2601662Z # shellcheck disable=SC2156 2024-04-03T17:40:04.2602453Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-04-03T17:40:04.2610104Z shell: /usr/bin/bash -e {0} 2024-04-03T17:40:04.2610425Z env: 2024-04-03T17:40:04.2610679Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:40:04.2611273Z DOCKER_CONTAINER_ID: 4c1cfaf4a8d048d787858cf2354817432c864f82c94c156c2f90005f40fddbdd 2024-04-03T17:40:04.2611918Z ##[endgroup] 2024-04-03T17:40:04.7933026Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-04-03T17:40:04.7933712Z with: 2024-04-03T17:40:04.7933953Z env: 2024-04-03T17:40:04.7934205Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:40:04.7934791Z DOCKER_CONTAINER_ID: 4c1cfaf4a8d048d787858cf2354817432c864f82c94c156c2f90005f40fddbdd 2024-04-03T17:40:04.7935434Z ##[endgroup] 2024-04-03T17:40:04.7955877Z ##[group]Run set -eou pipefail 2024-04-03T17:40:04.7956281Z set -eou pipefail 2024-04-03T17:40:04.7956624Z  2024-04-03T17:40:04.7957123Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-04-03T17:40:04.7957752Z for _ in $(seq 1440); do 2024-04-03T17:40:04.7958209Z  # Break if no ssh session exists anymore 2024-04-03T17:40:04.7958697Z  if [ "$(who)" = "" ]; then 2024-04-03T17:40:04.7959099Z  break 2024-04-03T17:40:04.7959393Z  fi 2024-04-03T17:40:04.7959704Z  echo "." 2024-04-03T17:40:04.7960017Z  sleep 5 2024-04-03T17:40:04.7960402Z done 2024-04-03T17:40:04.7967804Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T17:40:04.7968285Z env: 2024-04-03T17:40:04.7968548Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:40:04.7969145Z DOCKER_CONTAINER_ID: 4c1cfaf4a8d048d787858cf2354817432c864f82c94c156c2f90005f40fddbdd 2024-04-03T17:40:04.7969779Z ##[endgroup] 2024-04-03T17:40:04.7991293Z Holding runner for 2 hours until all ssh sessions have logged out 2024-04-03T17:40:04.8054410Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T17:40:04.8055186Z # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T17:40:04.8055776Z # shellcheck disable=SC2046 2024-04-03T17:40:04.8056227Z docker stop $(docker ps -q) || true 2024-04-03T17:40:04.8056817Z # Prune all of the docker images 2024-04-03T17:40:04.8057259Z docker system prune -af 2024-04-03T17:40:04.8064656Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T17:40:04.8065150Z env: 2024-04-03T17:40:04.8065416Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:40:04.8066022Z DOCKER_CONTAINER_ID: 4c1cfaf4a8d048d787858cf2354817432c864f82c94c156c2f90005f40fddbdd 2024-04-03T17:40:04.8066658Z ##[endgroup] 2024-04-03T17:40:15.1558542Z 4c1cfaf4a8d0 2024-04-03T17:40:15.7637162Z Deleted Containers: 2024-04-03T17:40:15.7637826Z 4c1cfaf4a8d048d787858cf2354817432c864f82c94c156c2f90005f40fddbdd 2024-04-03T17:40:15.7638276Z 2024-04-03T17:40:20.5288167Z Deleted Images: 2024-04-03T17:40:20.5290075Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T17:40:20.5292853Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11@sha256:66934e8a2aa7cad8e06a1fbc5a6f0e239ee3d124c7929e4418f76d11beb80c7b 2024-04-03T17:40:20.5294887Z deleted: sha256:edab37a6fecd099844f4888cdb58ab57de8cc1ffb0964933664a10a6b9fa73a8 2024-04-03T17:40:20.5295754Z deleted: sha256:905eb3165d72389fe2a77b6b0b9b42979fec6d4762ba5d6372b004485a7887f2 2024-04-03T17:40:20.5296607Z deleted: sha256:b52147393ef7954d559c54e5e8bdf5c9362923c6bc5e5c1ac7b416ae3bc50f1b 2024-04-03T17:40:20.5297469Z deleted: sha256:33129cdc36b6b17d0d59f4913c861e2ab55cec6b1f0bcf729807fced51ee458e 2024-04-03T17:40:20.5298323Z deleted: sha256:905ee334605b68ef01e63105bb18f2ea2b4fc1b7cb7feabfb7a0eb65023cfbaa 2024-04-03T17:40:20.5299188Z deleted: sha256:538bde79919a681e153c980201fba870a6f7ceb5f0cf38c531bdb7d8808da5fe 2024-04-03T17:40:20.5300320Z deleted: sha256:db649c126d92a59d7f3777fa9cda59a19fc09346e3f2da46637f769b001da08d 2024-04-03T17:40:20.5301164Z deleted: sha256:401aa540d7013a2fec18292b710a4b7b704ff0e5191a6409bfa7f5105c03bf0f 2024-04-03T17:40:20.5302005Z deleted: sha256:3b53de43223804a7efad8f916999035e775d90036803f86521e9be97539f40ad 2024-04-03T17:40:20.5303029Z deleted: sha256:b2f44ae198a8e733062bc965e9f8f0b237e493f74113f343e75e9f11c1ffd82d 2024-04-03T17:40:20.5303885Z deleted: sha256:512d4e1ee86372c20c627cfcbd49474ff2417e1cbe160861e19b0a6b612858b7 2024-04-03T17:40:20.5304747Z deleted: sha256:40f2d30dfb73d70ffeaccc94d76abf117254af30c7ae8d05b869cd1e02072c56 2024-04-03T17:40:20.5305619Z deleted: sha256:30cbd3d349b325ad324c7222b09f06461d7d5ba9c2f0d2d47902f95e4eb735e9 2024-04-03T17:40:20.5306884Z deleted: sha256:84e3a88d823a68472d445767c8677811a579b76d421ddc260d06de3a3d20254a 2024-04-03T17:40:20.5307795Z deleted: sha256:74be4e310354a4d7f3b6f81aa1701ef3f988e96e7c8ce0d1accc035bcebb7903 2024-04-03T17:40:20.5309398Z deleted: sha256:6a4cb11588f457a23f797cb6c3c63ff04ca813557f1a5199a4b56bb96f5a400f 2024-04-03T17:40:20.5310975Z deleted: sha256:154b4926ace46a1bc9dd4859a007e72e05c53455362fa4d1b9c6a7a88658e2ad 2024-04-03T17:40:20.5311820Z deleted: sha256:83cca981aeb36a816e3f7961b4db68f6d1083a1a485359e27002dd05551a0faf 2024-04-03T17:40:20.5312678Z deleted: sha256:bd99ecf15c76d11726541058b9988b00b0448f5572e29581f8fe4d1beb6f2f04 2024-04-03T17:40:20.5313530Z deleted: sha256:c2306f98283138f1e8a0eb86821c56a874a3a0e64f3c5bd78af77e3d4dc4cea2 2024-04-03T17:40:20.5329279Z deleted: sha256:8d3cbc73dd42c894ad477153964abbd1db5ff1a85639c576f3dd15aa49fc1e32 2024-04-03T17:40:20.5330541Z deleted: sha256:93bca5d18fc95e7b538f7b011f199ea47d4c4a24d92e37befdb55c65315daef3 2024-04-03T17:40:20.5331433Z deleted: sha256:ca03cea0e25997d85fc876c81c3ec9a2d5280b5b9efa04b3f5dde9482894735b 2024-04-03T17:40:20.5332317Z deleted: sha256:2868da78fde9fe36f01a3bdf87ff3af36ec30cf36787d0a5cbad1573f9eb72a7 2024-04-03T17:40:20.5333190Z deleted: sha256:861a2e61cf7edf4a2d567f8f91a0a1d6720b2804696e455566653519def5eed6 2024-04-03T17:40:20.5334061Z deleted: sha256:ae20e25a200dfe3331cb1623aaa4d94fda1da70ca1457bece6ee0736406c477b 2024-04-03T17:40:20.5334916Z deleted: sha256:b2c2049f6254580f321ddb8e344a6f30920295714920d542f50895964494bcaa 2024-04-03T17:40:20.5335744Z deleted: sha256:6528c514c7f21f80242cf6a7bdc1c8d915649cfafdb940bc43a730b92c055395 2024-04-03T17:40:20.5336768Z deleted: sha256:661270485835c1ad00ddc00a41d55afcdf2f15326873b6564685568e51502f0b 2024-04-03T17:40:20.5337622Z deleted: sha256:c620865dc9b9190a1d5020fa279930e88aea7cc0fba3fce9a23526d59fb91b8b 2024-04-03T17:40:20.5338494Z deleted: sha256:6bbb1aecdf7e92a8f542cf0d72e09fb3ebb5d3e54e026c45fc6005479c973812 2024-04-03T17:40:20.5339369Z deleted: sha256:c38992a57858cfe5482c0f6a222dd2ec1365febf3634116368e0224766b8eabf 2024-04-03T17:40:20.5340219Z deleted: sha256:a806e3a5b978f751b2d24fb75b0ec1531609aa5438e969b553e35eae9420291d 2024-04-03T17:40:20.5341059Z deleted: sha256:f977093d1313dfd0c96542aec48775e1c6153e6d4da0e44e4019a1f33bc7ccbc 2024-04-03T17:40:20.5341901Z deleted: sha256:d87f19303d93852e7dec32bda379838586b85592647ae5fb83e9145395922c52 2024-04-03T17:40:20.5342746Z deleted: sha256:30fab3365ed7c6036db2bd494146182269ab08e5f076a9f05cc782ae2a1be6f2 2024-04-03T17:40:20.5343622Z deleted: sha256:dc7d5459e2773dac5bd936b3e0f2712f59e2f2efb6fdd9fa58b1dad13e7ba709 2024-04-03T17:40:20.5344484Z deleted: sha256:3c9770b6c5083d2b47eb7cb18e40f73377ad8c85193c48d10dc53453942a58ca 2024-04-03T17:40:20.5345345Z deleted: sha256:6037d0815881f7ca8e604b2cd0ba9c02a7b51d1dfaf9b45f72a9690593f4522e 2024-04-03T17:40:20.5346203Z deleted: sha256:4c5ec0d9046384b56009a78bd08ea8c4d53566560bfc1a5f3cbbf29fb5d28e78 2024-04-03T17:40:20.5347036Z deleted: sha256:831fd71d6c6e601a049d95d96f3b86602c60014f66c4a8ba995274930ec6360d 2024-04-03T17:40:20.5347893Z deleted: sha256:7f853fb34a14d00c3cc35edb4c6c2d139853c00531d5c0e947aaf0af32fc8c6d 2024-04-03T17:40:20.5348754Z deleted: sha256:6216695cd3743e80cf171dc625b35bdd8c875eb38ccb1c74de9ecaf70ef12cf7 2024-04-03T17:40:20.5349745Z deleted: sha256:a3987ee214e96c8a4c394439202fdebc1742b5c9b7c49c795eb5b02873caf5c0 2024-04-03T17:40:20.5350619Z deleted: sha256:ec36b2dfaf0780fc77239a6442a03d3fe12344a62fd9f160706ea8fb31827c31 2024-04-03T17:40:20.5351488Z deleted: sha256:e74fdfc92a654609e8a0e58038f355ac20ca6e48f9740e1f8eeb4258745664e6 2024-04-03T17:40:20.5352379Z deleted: sha256:213a5ea0dfcd1c6d1a1b663bfb48666e21fa4ccfb4e50814fe80178c59877bf8 2024-04-03T17:40:20.5353373Z deleted: sha256:2eeecd229060f5607471a8b1ae0eb47c1ef360d0bde7b0b918fe58a471d07632 2024-04-03T17:40:20.5354243Z deleted: sha256:e06952350caa9d97c27e0cf877ec331c95ccd08f9d101338b3f2430048435795 2024-04-03T17:40:20.5355103Z deleted: sha256:3f092d21eb8319fde82a9b1457866cfe4b510bf67241420ca209e8bf7102059e 2024-04-03T17:40:20.5355939Z deleted: sha256:2e04267fe055da334722fb21dbc887d8f07a04fdf09a51938de789a08b291cfe 2024-04-03T17:40:20.5356793Z deleted: sha256:a8917fb72475f126ca60bfa82d4cf4f13f68faf76b35caca0a08278683a28bf8 2024-04-03T17:40:20.5357658Z deleted: sha256:90cc1b8908359659cd6d3452ac6ebe7a6c136fea9a08dbce8895e7dbdb6207a5 2024-04-03T17:40:20.5358531Z deleted: sha256:0fff0f3eae6cbddf08197e2d51fdba3a40ec153315bb94e5bf8cfd4b46f98419 2024-04-03T17:40:20.5359405Z deleted: sha256:24ab507a199fbc79871ceb0443245e24c0458d10faade9c24593b0623f78cb03 2024-04-03T17:40:20.5360251Z deleted: sha256:cd88b61c700f6c253b46448dff97698516945970d56776e20e894d80d190b904 2024-04-03T17:40:20.5361193Z deleted: sha256:8d4170f13f8a43f7f191b1c5a5f5b1bdd6530ddd457a80a7b69de301801550e9 2024-04-03T17:40:20.5362049Z deleted: sha256:eeb6d5b1e91efc779ec4a6422b4d87e49967833e726c2108c9fdfff3a9febf43 2024-04-03T17:40:20.5362913Z deleted: sha256:c93041a9e83bfb126e457b0603544f90579dda9c6a13a915849645eb85a68502 2024-04-03T17:40:20.5363763Z deleted: sha256:2fb3495a30e7eb0864d26a8a93d8c78b86b8a64cd41619a1361a08b799d11685 2024-04-03T17:40:20.5364628Z deleted: sha256:b66ee3c97bcad9d55cb39a67ccd6b3beb626b4187bedceff060d6df941cc837d 2024-04-03T17:40:20.5365516Z deleted: sha256:392ced3ad906438bdead3c86295eb143f2fb85264af147e6f1f35d01c1ced362 2024-04-03T17:40:20.5366390Z deleted: sha256:87c81bac51502ae15d854b5a0e091a15ee10bc2b6d418a5705ca4a3abfcd0d28 2024-04-03T17:40:20.5367256Z deleted: sha256:d4f429e424bdec09e61d24c39f65c9c3933c4dfeb4b6310f4987a1799657ad77 2024-04-03T17:40:20.5368105Z deleted: sha256:f5efbd022127e1175e36f2c5fd792ac1f7c47418c93f79846df2412bdb7430bd 2024-04-03T17:40:20.5369062Z deleted: sha256:e2fad19928d02016347bc999f793c1912a2a4c979511f4e87b49db49233eed88 2024-04-03T17:40:20.5369909Z deleted: sha256:ee0f573586ebb0580982f974c3101c73d10d53c9e5523cb283bb5fd9cfb45017 2024-04-03T17:40:20.5370742Z deleted: sha256:aec57c0a36a59d850a778188d86004b271d7ea7c7f3099b69cd0964c23df31f3 2024-04-03T17:40:20.5371613Z deleted: sha256:97e7daacc3e32cc73f0a634ac185f46b236145f7adc160db674efdf892c1766f 2024-04-03T17:40:20.5372480Z deleted: sha256:e5ff4155a5c530732299da68cf28a5a52028fa5a25899de16a686b43d6c04e63 2024-04-03T17:40:20.5373330Z deleted: sha256:6145ed59ce48c64268ca5c98a5fbe2fbff17157e38af09e12bff68b74c79e74c 2024-04-03T17:40:20.5374198Z deleted: sha256:8ecde50858cbc91165551bcd309f8cd00e49b8ae1f7f864070b5551fc199ec46 2024-04-03T17:40:20.5375063Z deleted: sha256:c0c28dfb686a210566daab4fafef59cb4720c8036d3011179056cd059e034bce 2024-04-03T17:40:20.5375909Z deleted: sha256:d5d2024b69813e69f912057b0af512c77952d0e780a84868328829a7c3b1a5b7 2024-04-03T17:40:20.5376746Z deleted: sha256:1d15b0497ee6d16bddf0294f7a72aea8a230ef252d8a736c3084de3bd448f007 2024-04-03T17:40:20.5377617Z deleted: sha256:6f9321552d3cd0357443910af819fc1080aae134db4db2cebc9fbe8950f5970e 2024-04-03T17:40:20.5378452Z deleted: sha256:10406cf5503ff7215e2fa6a948121f26252776c8a77d59c03ea960d181953e33 2024-04-03T17:40:20.5379296Z deleted: sha256:edacda4f3bed0017ee8e30644e3ca826770e4fa9594617f67f67d8b04566c4d6 2024-04-03T17:40:20.5380149Z deleted: sha256:2734737e3cd948c91fd4003c5ee35820d4115a112c0478e0f031365431c1a2f6 2024-04-03T17:40:20.5380986Z deleted: sha256:eb1b20984d5deb9d3d1743e16a10683d787ce12f6b88133174ec4999344f362c 2024-04-03T17:40:20.5381822Z deleted: sha256:549d11624b0cbbb68984f5d6a33498bfb076fc901e4f23ea53b32971c7aaed02 2024-04-03T17:40:20.5382661Z deleted: sha256:26f2869e3ab075b907969de9aa179979f4fc9f8cc864025a646497ed71e2f0bc 2024-04-03T17:40:20.5383504Z deleted: sha256:222a48fb23cb13e5340360dee15485a1e547c730e8b0f79be58b38b443f96a32 2024-04-03T17:40:20.5384360Z deleted: sha256:81ef77bb993ba4fb3978141df789aba6d6754bb86c51c859d717367d9c37a65f 2024-04-03T17:40:20.5385290Z deleted: sha256:7bb82f4b0c6491c4b4cafaadd5377931be37d828ac2899475ac8abe9d0e520af 2024-04-03T17:40:20.5386164Z deleted: sha256:5498e8c22f6996f25ef193ee58617d5b37e2a96decf22e72de13c3b34e147591 2024-04-03T17:40:20.5386669Z 2024-04-03T17:40:20.5386821Z Total reclaimed space: 14.29GB 2024-04-03T17:40:20.5474571Z Post job cleanup. 2024-04-03T17:40:20.5533313Z Post job cleanup. 2024-04-03T17:40:20.6397544Z [command]/usr/bin/git version 2024-04-03T17:40:20.6439571Z git version 2.40.1 2024-04-03T17:40:20.6476886Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/74f9a2ee-34c7-40cd-a788-81102af7b2c7' before making global git config changes 2024-04-03T17:40:20.6478272Z Adding repository directory to the temporary git global config as a safe directory 2024-04-03T17:40:20.6481605Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T17:40:20.6528482Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-04-03T17:40:20.6559444Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-04-03T17:40:20.6900295Z Entering 'android/libs/fbjni' 2024-04-03T17:40:20.6949447Z Entering 'third_party/FP16' 2024-04-03T17:40:20.7000012Z Entering 'third_party/FXdiv' 2024-04-03T17:40:20.7047789Z Entering 'third_party/NNPACK' 2024-04-03T17:40:20.7096270Z Entering 'third_party/QNNPACK' 2024-04-03T17:40:20.7145849Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T17:40:20.7187320Z Entering 'third_party/XNNPACK' 2024-04-03T17:40:20.7353294Z Entering 'third_party/benchmark' 2024-04-03T17:40:20.7400963Z Entering 'third_party/cpuinfo' 2024-04-03T17:40:20.7451127Z Entering 'third_party/cub' 2024-04-03T17:40:20.7498452Z Entering 'third_party/cudnn_frontend' 2024-04-03T17:40:20.7539485Z Entering 'third_party/cutlass' 2024-04-03T17:40:20.7587571Z Entering 'third_party/eigen' 2024-04-03T17:40:20.7652169Z Entering 'third_party/fbgemm' 2024-04-03T17:40:20.7727914Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T17:40:20.7805160Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T17:40:20.7873056Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T17:40:20.7975115Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T17:40:20.8032475Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T17:40:20.8077359Z Entering 'third_party/flatbuffers' 2024-04-03T17:40:20.8120977Z Entering 'third_party/fmt' 2024-04-03T17:40:20.8169679Z Entering 'third_party/foxi' 2024-04-03T17:40:20.8216395Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T17:40:20.8263676Z Entering 'third_party/gloo' 2024-04-03T17:40:20.8311779Z Entering 'third_party/googletest' 2024-04-03T17:40:20.8362522Z Entering 'third_party/ideep' 2024-04-03T17:40:20.8422439Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T17:40:20.8507695Z Entering 'third_party/ios-cmake' 2024-04-03T17:40:20.8556119Z Entering 'third_party/ittapi' 2024-04-03T17:40:20.8596825Z Entering 'third_party/kineto' 2024-04-03T17:40:20.8637306Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T17:40:20.8677332Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T17:40:20.8718224Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T17:40:20.8757742Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T17:40:20.8797731Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T17:40:20.8838394Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T17:40:20.8879553Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T17:40:20.8919669Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T17:40:20.8961188Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T17:40:20.9001813Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T17:40:20.9042605Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T17:40:20.9083338Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T17:40:20.9124434Z Entering 'third_party/mimalloc' 2024-04-03T17:40:20.9165238Z Entering 'third_party/nccl/nccl' 2024-04-03T17:40:20.9216015Z Entering 'third_party/neon2sse' 2024-04-03T17:40:20.9262260Z Entering 'third_party/nlohmann' 2024-04-03T17:40:20.9304435Z Entering 'third_party/onnx' 2024-04-03T17:40:20.9451596Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T17:40:20.9513723Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T17:40:20.9564243Z Entering 'third_party/onnx-tensorrt' 2024-04-03T17:40:20.9629704Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T17:40:20.9723085Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T17:40:20.9780856Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T17:40:20.9843701Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T17:40:20.9892551Z Entering 'third_party/pocketfft' 2024-04-03T17:40:20.9933239Z Entering 'third_party/protobuf' 2024-04-03T17:40:21.0029208Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T17:40:21.0088901Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T17:40:21.0138419Z Entering 'third_party/psimd' 2024-04-03T17:40:21.0183400Z Entering 'third_party/pthreadpool' 2024-04-03T17:40:21.0232001Z Entering 'third_party/pybind11' 2024-04-03T17:40:21.0285982Z Entering 'third_party/python-peachpy' 2024-04-03T17:40:21.0327861Z Entering 'third_party/sleef' 2024-04-03T17:40:21.0370752Z Entering 'third_party/tbb' 2024-04-03T17:40:21.0414383Z Entering 'third_party/tensorpipe' 2024-04-03T17:40:21.0455828Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T17:40:21.0495774Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T17:40:21.0537209Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T17:40:21.0578247Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T17:40:21.0618115Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T17:40:21.0661572Z Entering 'third_party/zstd' 2024-04-03T17:40:21.0717807Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-04-03T17:40:21.0740420Z http.https://github.com/.extraheader 2024-04-03T17:40:21.0748354Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-04-03T17:40:21.0779551Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-04-03T17:40:21.1025327Z Entering 'android/libs/fbjni' 2024-04-03T17:40:21.1051760Z http.https://github.com/.extraheader 2024-04-03T17:40:21.1078938Z Entering 'third_party/FP16' 2024-04-03T17:40:21.1105165Z http.https://github.com/.extraheader 2024-04-03T17:40:21.1132436Z Entering 'third_party/FXdiv' 2024-04-03T17:40:21.1158849Z http.https://github.com/.extraheader 2024-04-03T17:40:21.1185240Z Entering 'third_party/NNPACK' 2024-04-03T17:40:21.1211367Z http.https://github.com/.extraheader 2024-04-03T17:40:21.1239688Z Entering 'third_party/QNNPACK' 2024-04-03T17:40:21.1265612Z http.https://github.com/.extraheader 2024-04-03T17:40:21.1293063Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T17:40:21.1319898Z http.https://github.com/.extraheader 2024-04-03T17:40:21.1346596Z Entering 'third_party/XNNPACK' 2024-04-03T17:40:21.1373346Z http.https://github.com/.extraheader 2024-04-03T17:40:21.1416665Z Entering 'third_party/benchmark' 2024-04-03T17:40:21.1443027Z http.https://github.com/.extraheader 2024-04-03T17:40:21.1469019Z Entering 'third_party/cpuinfo' 2024-04-03T17:40:21.1495984Z http.https://github.com/.extraheader 2024-04-03T17:40:21.1522806Z Entering 'third_party/cub' 2024-04-03T17:40:21.1548523Z http.https://github.com/.extraheader 2024-04-03T17:40:21.1575808Z Entering 'third_party/cudnn_frontend' 2024-04-03T17:40:21.1602161Z http.https://github.com/.extraheader 2024-04-03T17:40:21.1628620Z Entering 'third_party/cutlass' 2024-04-03T17:40:21.1654819Z http.https://github.com/.extraheader 2024-04-03T17:40:21.1689569Z Entering 'third_party/eigen' 2024-04-03T17:40:21.1715802Z http.https://github.com/.extraheader 2024-04-03T17:40:21.1744072Z Entering 'third_party/fbgemm' 2024-04-03T17:40:21.1771288Z http.https://github.com/.extraheader 2024-04-03T17:40:21.1798499Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T17:40:21.1823447Z http.https://github.com/.extraheader 2024-04-03T17:40:21.1851968Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T17:40:21.1877252Z http.https://github.com/.extraheader 2024-04-03T17:40:21.1903793Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T17:40:21.1928621Z http.https://github.com/.extraheader 2024-04-03T17:40:21.1962544Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T17:40:21.1987480Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2014950Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T17:40:21.2040854Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2068611Z Entering 'third_party/flatbuffers' 2024-04-03T17:40:21.2094552Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2124015Z Entering 'third_party/fmt' 2024-04-03T17:40:21.2150425Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2177310Z Entering 'third_party/foxi' 2024-04-03T17:40:21.2203047Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2229823Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T17:40:21.2255307Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2282372Z Entering 'third_party/gloo' 2024-04-03T17:40:21.2308080Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2335307Z Entering 'third_party/googletest' 2024-04-03T17:40:21.2362184Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2389340Z Entering 'third_party/ideep' 2024-04-03T17:40:21.2415181Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2441447Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T17:40:21.2467065Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2502081Z Entering 'third_party/ios-cmake' 2024-04-03T17:40:21.2527688Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2554607Z Entering 'third_party/ittapi' 2024-04-03T17:40:21.2580930Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2608164Z Entering 'third_party/kineto' 2024-04-03T17:40:21.2633902Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2660562Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T17:40:21.2685951Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2712972Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T17:40:21.2739040Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2767740Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T17:40:21.2793147Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2819679Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T17:40:21.2845702Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2873200Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T17:40:21.2899017Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2925586Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T17:40:21.2953009Z http.https://github.com/.extraheader 2024-04-03T17:40:21.2981459Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T17:40:21.3008351Z http.https://github.com/.extraheader 2024-04-03T17:40:21.3035826Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T17:40:21.3060955Z http.https://github.com/.extraheader 2024-04-03T17:40:21.3088548Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T17:40:21.3114806Z http.https://github.com/.extraheader 2024-04-03T17:40:21.3143807Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T17:40:21.3168915Z http.https://github.com/.extraheader 2024-04-03T17:40:21.3197709Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T17:40:21.3223457Z http.https://github.com/.extraheader 2024-04-03T17:40:21.3251100Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T17:40:21.3276606Z http.https://github.com/.extraheader 2024-04-03T17:40:21.3305610Z Entering 'third_party/mimalloc' 2024-04-03T17:40:21.3332386Z http.https://github.com/.extraheader 2024-04-03T17:40:21.3360060Z Entering 'third_party/nccl/nccl' 2024-04-03T17:40:21.3386467Z http.https://github.com/.extraheader 2024-04-03T17:40:21.3414016Z Entering 'third_party/neon2sse' 2024-04-03T17:40:21.3440844Z http.https://github.com/.extraheader 2024-04-03T17:40:21.3467617Z Entering 'third_party/nlohmann' 2024-04-03T17:40:21.3493914Z http.https://github.com/.extraheader 2024-04-03T17:40:21.3523600Z Entering 'third_party/onnx' 2024-04-03T17:40:21.3550746Z http.https://github.com/.extraheader 2024-04-03T17:40:21.3594998Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T17:40:21.3620798Z http.https://github.com/.extraheader 2024-04-03T17:40:21.3648865Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T17:40:21.3675787Z http.https://github.com/.extraheader 2024-04-03T17:40:21.3705412Z Entering 'third_party/onnx-tensorrt' 2024-04-03T17:40:21.3733288Z http.https://github.com/.extraheader 2024-04-03T17:40:21.3759393Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T17:40:21.3785169Z http.https://github.com/.extraheader 2024-04-03T17:40:21.3817382Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T17:40:21.3843561Z http.https://github.com/.extraheader 2024-04-03T17:40:21.3872736Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T17:40:21.3899391Z http.https://github.com/.extraheader 2024-04-03T17:40:21.3926048Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T17:40:21.3952089Z http.https://github.com/.extraheader 2024-04-03T17:40:21.3982417Z Entering 'third_party/pocketfft' 2024-04-03T17:40:21.4009557Z http.https://github.com/.extraheader 2024-04-03T17:40:21.4036705Z Entering 'third_party/protobuf' 2024-04-03T17:40:21.4063654Z http.https://github.com/.extraheader 2024-04-03T17:40:21.4094787Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T17:40:21.4121621Z http.https://github.com/.extraheader 2024-04-03T17:40:21.4148365Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T17:40:21.4174037Z http.https://github.com/.extraheader 2024-04-03T17:40:21.4202784Z Entering 'third_party/psimd' 2024-04-03T17:40:21.4229303Z http.https://github.com/.extraheader 2024-04-03T17:40:21.4256803Z Entering 'third_party/pthreadpool' 2024-04-03T17:40:21.4284929Z http.https://github.com/.extraheader 2024-04-03T17:40:21.4312025Z Entering 'third_party/pybind11' 2024-04-03T17:40:21.4337531Z http.https://github.com/.extraheader 2024-04-03T17:40:21.4365248Z Entering 'third_party/python-peachpy' 2024-04-03T17:40:21.4392712Z http.https://github.com/.extraheader 2024-04-03T17:40:21.4420003Z Entering 'third_party/sleef' 2024-04-03T17:40:21.4446610Z http.https://github.com/.extraheader 2024-04-03T17:40:21.4473159Z Entering 'third_party/tbb' 2024-04-03T17:40:21.4500384Z http.https://github.com/.extraheader 2024-04-03T17:40:21.4529518Z Entering 'third_party/tensorpipe' 2024-04-03T17:40:21.4557458Z http.https://github.com/.extraheader 2024-04-03T17:40:21.4584269Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T17:40:21.4609873Z http.https://github.com/.extraheader 2024-04-03T17:40:21.4637626Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T17:40:21.4663319Z http.https://github.com/.extraheader 2024-04-03T17:40:21.4690499Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T17:40:21.4716328Z http.https://github.com/.extraheader 2024-04-03T17:40:21.4743498Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T17:40:21.4768902Z http.https://github.com/.extraheader 2024-04-03T17:40:21.4794714Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T17:40:21.4820123Z http.https://github.com/.extraheader 2024-04-03T17:40:21.4849366Z Entering 'third_party/zstd' 2024-04-03T17:40:21.4877528Z http.https://github.com/.extraheader 2024-04-03T17:40:21.5172032Z Cleaning up orphan processes