2024-04-03T16:13:28.8572132Z Current runner version: '2.315.0' 2024-04-03T16:13:28.8581960Z Runner name: 'i-00c317a4521719f93' 2024-04-03T16:13:28.8583173Z Runner group name: 'Default' 2024-04-03T16:13:28.8584466Z Machine name: 'ip-10-0-56-183' 2024-04-03T16:13:28.8591256Z ##[group]GITHUB_TOKEN Permissions 2024-04-03T16:13:28.8594578Z Actions: read 2024-04-03T16:13:28.8595414Z Checks: read 2024-04-03T16:13:28.8596102Z Contents: read 2024-04-03T16:13:28.8596966Z Deployments: read 2024-04-03T16:13:28.8597761Z Discussions: read 2024-04-03T16:13:28.8598462Z Issues: read 2024-04-03T16:13:28.8599233Z Metadata: read 2024-04-03T16:13:28.8599987Z Packages: read 2024-04-03T16:13:28.8600703Z Pages: read 2024-04-03T16:13:28.8601475Z PullRequests: read 2024-04-03T16:13:28.8602342Z RepositoryProjects: read 2024-04-03T16:13:28.8603166Z SecurityEvents: read 2024-04-03T16:13:28.8604015Z Statuses: read 2024-04-03T16:13:28.8604770Z ##[endgroup] 2024-04-03T16:13:28.8609671Z Secret source: None 2024-04-03T16:13:28.8611076Z Prepare workflow directory 2024-04-03T16:13:29.0110120Z Prepare all required actions 2024-04-03T16:13:29.0375061Z Getting action download info 2024-04-03T16:13:29.2377454Z Download action repository 'pytorch/test-infra@main' (SHA:f5543b9b2d7b7c7b8eccd6576b68fc0472b1fbde) 2024-04-03T16:13:29.6621233Z Download action repository 'pytorch/pytorch@main' (SHA:d895192e87f9a2c9ab3d6fab60f08855a08c3900) 2024-04-03T16:13:34.5682241Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-04-03T16:13:34.7382050Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-04-03T16:13:35.1030310Z Getting action download info 2024-04-03T16:13:35.2056202Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-04-03T16:13:35.4233071Z Getting action download info 2024-04-03T16:13:35.5155256Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-04-03T16:13:35.6990007Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/123154/merge (4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0) 2024-04-03T16:13:35.6994430Z ##[group] Inputs 2024-04-03T16:13:35.6995145Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-04-03T16:13:35.6997596Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-04-03T16:13:35.7000797Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:13:35.7002323Z sync-tag: 2024-04-03T16:13:35.7003698Z timeout-minutes: 360 2024-04-03T16:13:35.7004189Z use-gha: 2024-04-03T16:13:35.7004609Z dashboard-tag: 2024-04-03T16:13:35.7005073Z s3-bucket: gha-artifacts 2024-04-03T16:13:35.7005561Z aws-role-to-assume: 2024-04-03T16:13:35.7006036Z ##[endgroup] 2024-04-03T16:13:35.7007498Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:13:35.8454587Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-04-03T16:13:35.8455422Z with: 2024-04-03T16:13:35.8456294Z github-secret: *** 2024-04-03T16:13:35.8457691Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-04-03T16:13:35.8459156Z activate-with-label: false 2024-04-03T16:13:35.8459675Z label: with-ssh 2024-04-03T16:13:35.8460135Z remove-existing-keys: true 2024-04-03T16:13:35.8460618Z env: 2024-04-03T16:13:35.8461015Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:13:35.8461546Z ##[endgroup] 2024-04-03T16:13:35.9934614Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-04-03T16:13:36.3901077Z Grabbing public ssh keys from https://github.com/caic99.keys 2024-04-03T16:13:36.4505186Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-04-03T16:13:36.4531247Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-04-03T16:13:36.4580876Z Login using: ssh ec2-user@ec2-54-90-114-50.compute-1.amazonaws.com 2024-04-03T16:13:36.4582015Z All testing is done inside the container, to start an interactive session run: 2024-04-03T16:13:36.4583203Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-04-03T16:13:36.4951961Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-04-03T16:13:36.4952775Z with: 2024-04-03T16:13:36.4953163Z submodules: recursive 2024-04-03T16:13:36.4953641Z fetch-depth: 0 2024-04-03T16:13:36.4954027Z env: 2024-04-03T16:13:36.4954745Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:13:36.4955236Z ##[endgroup] 2024-04-03T16:13:36.5294453Z ##[group]Run retry () { 2024-04-03T16:13:36.5295032Z retry () { 2024-04-03T16:13:36.5295761Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-04-03T16:13:36.5296544Z } 2024-04-03T16:13:36.5296979Z echo "${GITHUB_WORKSPACE}" 2024-04-03T16:13:36.5297579Z if [ -z "${NO_SUDO}" ]; then 2024-04-03T16:13:36.5298245Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-04-03T16:13:36.5298900Z else 2024-04-03T16:13:36.5299391Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-04-03T16:13:36.5299981Z fi 2024-04-03T16:13:36.5300482Z mkdir "${GITHUB_WORKSPACE}" 2024-04-03T16:13:36.5334283Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:13:36.5335030Z env: 2024-04-03T16:13:36.5335404Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:13:36.5335894Z NO_SUDO: 2024-04-03T16:13:36.5336287Z ##[endgroup] 2024-04-03T16:13:36.5484029Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:13:36.5748545Z ##[group]Run malfet/checkout@silent-checkout 2024-04-03T16:13:36.5749175Z with: 2024-04-03T16:13:36.5749623Z ref: d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:13:36.5750267Z fetch-depth: 0 2024-04-03T16:13:36.5750709Z submodules: recursive 2024-04-03T16:13:36.5751175Z quiet-checkout: true 2024-04-03T16:13:36.5751670Z repository: pytorch/pytorch 2024-04-03T16:13:36.5752416Z token: *** 2024-04-03T16:13:36.5752816Z ssh-strict: true 2024-04-03T16:13:36.5753282Z persist-credentials: true 2024-04-03T16:13:36.5753789Z clean: true 2024-04-03T16:13:36.5754665Z sparse-checkout-cone-mode: true 2024-04-03T16:13:36.5755245Z lfs: false 2024-04-03T16:13:36.5755670Z set-safe-directory: true 2024-04-03T16:13:36.5756125Z env: 2024-04-03T16:13:36.5756500Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:13:36.5756983Z ##[endgroup] 2024-04-03T16:13:36.7245411Z Syncing repository: pytorch/pytorch 2024-04-03T16:13:36.7248106Z ##[group]Getting Git version info 2024-04-03T16:13:36.7249197Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-04-03T16:13:36.7250456Z [command]/usr/bin/git version 2024-04-03T16:13:36.7250983Z git version 2.40.1 2024-04-03T16:13:36.7252884Z ##[endgroup] 2024-04-03T16:13:36.7273428Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/321baacf-c5b4-4dbb-83e2-f8aa60df5cdb' before making global git config changes 2024-04-03T16:13:36.7275673Z Adding repository directory to the temporary git global config as a safe directory 2024-04-03T16:13:36.7281652Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:13:36.7330089Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-04-03T16:13:36.7336867Z ##[group]Initializing the repository 2024-04-03T16:13:36.7341506Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:13:36.7374654Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-04-03T16:13:36.7376220Z hint: is subject to change. To configure the initial branch name to use in all 2024-04-03T16:13:36.7377388Z hint: of your new repositories, which will suppress this warning, call: 2024-04-03T16:13:36.7378387Z hint: 2024-04-03T16:13:36.7379000Z hint: git config --global init.defaultBranch 2024-04-03T16:13:36.7379675Z hint: 2024-04-03T16:13:36.7380410Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-04-03T16:13:36.7381628Z hint: 'development'. The just-created branch can be renamed via this command: 2024-04-03T16:13:36.7382655Z hint: 2024-04-03T16:13:36.7383127Z hint: git branch -m 2024-04-03T16:13:36.7384191Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-04-03T16:13:36.7390248Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-04-03T16:13:36.7426947Z ##[endgroup] 2024-04-03T16:13:36.7427775Z ##[group]Disabling automatic garbage collection 2024-04-03T16:13:36.7432185Z [command]/usr/bin/git config --local gc.auto 0 2024-04-03T16:13:36.7467679Z ##[endgroup] 2024-04-03T16:13:36.7468402Z ##[group]Setting up auth 2024-04-03T16:13:36.7479558Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-04-03T16:13:36.7517338Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-04-03T16:13:36.7871013Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-04-03T16:13:36.7904612Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-04-03T16:13:36.8227979Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-04-03T16:13:36.8282473Z ##[endgroup] 2024-04-03T16:13:36.8283262Z ##[group]Fetching the repository 2024-04-03T16:13:36.8292376Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-04-03T16:13:42.1192849Z remote: Enumerating objects: 1123747 2024-04-03T16:13:42.1195000Z remote: Enumerating objects: 1124411, done. 2024-04-03T16:13:42.1196733Z remote: Counting objects: 0% (1/664) 2024-04-03T16:13:42.1198311Z remote: Counting objects: 1% (7/664) 2024-04-03T16:13:42.1199931Z remote: Counting objects: 2% (14/664) 2024-04-03T16:13:42.1201240Z remote: Counting objects: 3% (20/664) 2024-04-03T16:13:42.1202820Z remote: Counting objects: 4% (27/664) 2024-04-03T16:13:42.1204469Z remote: Counting objects: 5% (34/664) 2024-04-03T16:13:42.1206014Z remote: Counting objects: 6% (40/664) 2024-04-03T16:13:42.1207584Z remote: Counting objects: 7% (47/664) 2024-04-03T16:13:42.1208597Z remote: Counting objects: 8% (54/664) 2024-04-03T16:13:42.1209409Z remote: Counting objects: 9% (60/664) 2024-04-03T16:13:42.1210195Z remote: Counting objects: 10% (67/664) 2024-04-03T16:13:42.1211001Z remote: Counting objects: 11% (74/664) 2024-04-03T16:13:42.1211867Z remote: Counting objects: 12% (80/664) 2024-04-03T16:13:42.1212667Z remote: Counting objects: 13% (87/664) 2024-04-03T16:13:42.1213449Z remote: Counting objects: 14% (93/664) 2024-04-03T16:13:42.1214274Z remote: Counting objects: 15% (100/664) 2024-04-03T16:13:42.1215097Z remote: Counting objects: 16% (107/664) 2024-04-03T16:13:42.1215893Z remote: Counting objects: 17% (113/664) 2024-04-03T16:13:42.1216707Z remote: Counting objects: 18% (120/664) 2024-04-03T16:13:42.1217522Z remote: Counting objects: 19% (127/664) 2024-04-03T16:13:42.1273394Z remote: Counting objects: 20% (133/664) 2024-04-03T16:13:42.1274557Z remote: Counting objects: 21% (140/664) 2024-04-03T16:13:42.1275779Z remote: Counting objects: 22% (147/664) 2024-04-03T16:13:42.1276734Z remote: Counting objects: 23% (153/664) 2024-04-03T16:13:42.1277544Z remote: Counting objects: 24% (160/664) 2024-04-03T16:13:42.1278370Z remote: Counting objects: 25% (166/664) 2024-04-03T16:13:42.1279185Z remote: Counting objects: 26% (173/664) 2024-04-03T16:13:42.1279980Z remote: Counting objects: 27% (180/664) 2024-04-03T16:13:42.1281060Z remote: Counting objects: 28% (186/664) 2024-04-03T16:13:42.1281922Z remote: Counting objects: 29% (193/664) 2024-04-03T16:13:42.1282722Z remote: Counting objects: 30% (200/664) 2024-04-03T16:13:42.1283542Z remote: Counting objects: 31% (206/664) 2024-04-03T16:13:42.1284354Z remote: Counting objects: 32% (213/664) 2024-04-03T16:13:42.1285162Z remote: Counting objects: 33% (220/664) 2024-04-03T16:13:42.1285977Z remote: Counting objects: 34% (226/664) 2024-04-03T16:13:42.1286800Z remote: Counting objects: 35% (233/664) 2024-04-03T16:13:42.1287615Z remote: Counting objects: 36% (240/664) 2024-04-03T16:13:42.1288410Z remote: Counting objects: 37% (246/664) 2024-04-03T16:13:42.1289218Z remote: Counting objects: 38% (253/664) 2024-04-03T16:13:42.1290027Z remote: Counting objects: 39% (259/664) 2024-04-03T16:13:42.1291003Z remote: Counting objects: 40% (266/664) 2024-04-03T16:13:42.1291821Z remote: Counting objects: 41% (273/664) 2024-04-03T16:13:42.1292635Z remote: Counting objects: 42% (279/664) 2024-04-03T16:13:42.1293421Z remote: Counting objects: 43% (286/664) 2024-04-03T16:13:42.1294227Z remote: Counting objects: 44% (293/664) 2024-04-03T16:13:42.1295035Z remote: Counting objects: 45% (299/664) 2024-04-03T16:13:42.1295822Z remote: Counting objects: 46% (306/664) 2024-04-03T16:13:42.1296642Z remote: Counting objects: 47% (313/664) 2024-04-03T16:13:42.1297456Z remote: Counting objects: 48% (319/664) 2024-04-03T16:13:42.1298246Z remote: Counting objects: 49% (326/664) 2024-04-03T16:13:42.1299064Z remote: Counting objects: 50% (332/664) 2024-04-03T16:13:42.1299871Z remote: Counting objects: 51% (339/664) 2024-04-03T16:13:42.1300659Z remote: Counting objects: 52% (346/664) 2024-04-03T16:13:42.1301469Z remote: Counting objects: 53% (352/664) 2024-04-03T16:13:42.1302288Z remote: Counting objects: 54% (359/664) 2024-04-03T16:13:42.1303078Z remote: Counting objects: 55% (366/664) 2024-04-03T16:13:42.1303886Z remote: Counting objects: 56% (372/664) 2024-04-03T16:13:42.1304696Z remote: Counting objects: 57% (379/664) 2024-04-03T16:13:42.1305487Z remote: Counting objects: 58% (386/664) 2024-04-03T16:13:42.1307005Z remote: Counting objects: 59% (392/664) 2024-04-03T16:13:42.1308629Z remote: Counting objects: 60% (399/664) 2024-04-03T16:13:42.1309807Z remote: Counting objects: 61% (406/664) 2024-04-03T16:13:42.1310614Z remote: Counting objects: 62% (412/664) 2024-04-03T16:13:42.1311443Z remote: Counting objects: 63% (419/664) 2024-04-03T16:13:42.1312261Z remote: Counting objects: 64% (425/664) 2024-04-03T16:13:42.1313072Z remote: Counting objects: 65% (432/664) 2024-04-03T16:13:42.1313862Z remote: Counting objects: 66% (439/664) 2024-04-03T16:13:42.1314953Z remote: Counting objects: 67% (445/664) 2024-04-03T16:13:42.1315776Z remote: Counting objects: 68% (452/664) 2024-04-03T16:13:42.1316567Z remote: Counting objects: 69% (459/664) 2024-04-03T16:13:42.1317377Z remote: Counting objects: 70% (465/664) 2024-04-03T16:13:42.1318184Z remote: Counting objects: 71% (472/664) 2024-04-03T16:13:42.1318973Z remote: Counting objects: 72% (479/664) 2024-04-03T16:13:42.1320066Z remote: Counting objects: 73% (485/664) 2024-04-03T16:13:42.1320880Z remote: Counting objects: 74% (492/664) 2024-04-03T16:13:42.1321675Z remote: Counting objects: 75% (498/664) 2024-04-03T16:13:42.1322484Z remote: Counting objects: 76% (505/664) 2024-04-03T16:13:42.1323294Z remote: Counting objects: 77% (512/664) 2024-04-03T16:13:42.1324083Z remote: Counting objects: 78% (518/664) 2024-04-03T16:13:42.1324893Z remote: Counting objects: 79% (525/664) 2024-04-03T16:13:42.1325707Z remote: Counting objects: 80% (532/664) 2024-04-03T16:13:42.1326623Z remote: Counting objects: 81% (538/664) 2024-04-03T16:13:42.1327464Z remote: Counting objects: 82% (545/664) 2024-04-03T16:13:42.1328277Z remote: Counting objects: 83% (552/664) 2024-04-03T16:13:42.1329067Z remote: Counting objects: 84% (558/664) 2024-04-03T16:13:42.1329879Z remote: Counting objects: 85% (565/664) 2024-04-03T16:13:42.1330697Z remote: Counting objects: 86% (572/664) 2024-04-03T16:13:42.1331500Z remote: Counting objects: 87% (578/664) 2024-04-03T16:13:42.1332294Z remote: Counting objects: 88% (585/664) 2024-04-03T16:13:42.1333101Z remote: Counting objects: 89% (591/664) 2024-04-03T16:13:42.1333913Z remote: Counting objects: 90% (598/664) 2024-04-03T16:13:42.1334704Z remote: Counting objects: 91% (605/664) 2024-04-03T16:13:42.1335512Z remote: Counting objects: 92% (611/664) 2024-04-03T16:13:42.1336321Z remote: Counting objects: 93% (618/664) 2024-04-03T16:13:42.1337123Z remote: Counting objects: 94% (625/664) 2024-04-03T16:13:42.1337939Z remote: Counting objects: 95% (631/664) 2024-04-03T16:13:42.1338751Z remote: Counting objects: 96% (638/664) 2024-04-03T16:13:42.1339602Z remote: Counting objects: 97% (645/664) 2024-04-03T16:13:42.1340409Z remote: Counting objects: 98% (651/664) 2024-04-03T16:13:42.1341232Z remote: Counting objects: 99% (658/664) 2024-04-03T16:13:42.1342026Z remote: Counting objects: 100% (664/664) 2024-04-03T16:13:42.1342886Z remote: Counting objects: 100% (664/664), done. 2024-04-03T16:13:42.1343789Z remote: Compressing objects: 0% (1/369) 2024-04-03T16:13:42.1344615Z remote: Compressing objects: 1% (4/369) 2024-04-03T16:13:42.1374058Z remote: Compressing objects: 2% (8/369) 2024-04-03T16:13:42.1532848Z remote: Compressing objects: 3% (12/369) 2024-04-03T16:13:42.1600298Z remote: Compressing objects: 4% (15/369) 2024-04-03T16:13:42.1962228Z remote: Compressing objects: 5% (19/369) 2024-04-03T16:13:42.2055677Z remote: Compressing objects: 6% (23/369) 2024-04-03T16:13:42.2174827Z remote: Compressing objects: 7% (26/369) 2024-04-03T16:13:42.2309588Z remote: Compressing objects: 8% (30/369) 2024-04-03T16:13:42.2371102Z remote: Compressing objects: 9% (34/369) 2024-04-03T16:13:42.2396842Z remote: Compressing objects: 10% (37/369) 2024-04-03T16:13:42.2486625Z remote: Compressing objects: 11% (41/369) 2024-04-03T16:13:42.2503958Z remote: Compressing objects: 12% (45/369) 2024-04-03T16:13:42.2573456Z remote: Compressing objects: 13% (48/369) 2024-04-03T16:13:42.2574343Z remote: Compressing objects: 14% (52/369) 2024-04-03T16:13:42.2575200Z remote: Compressing objects: 15% (56/369) 2024-04-03T16:13:42.2603849Z remote: Compressing objects: 16% (60/369) 2024-04-03T16:13:42.2616260Z remote: Compressing objects: 17% (63/369) 2024-04-03T16:13:42.2628337Z remote: Compressing objects: 18% (67/369) 2024-04-03T16:13:42.2633539Z remote: Compressing objects: 19% (71/369) 2024-04-03T16:13:42.2638575Z remote: Compressing objects: 20% (74/369) 2024-04-03T16:13:42.2642845Z remote: Compressing objects: 21% (78/369) 2024-04-03T16:13:42.2646679Z remote: Compressing objects: 22% (82/369) 2024-04-03T16:13:42.2649949Z remote: Compressing objects: 23% (85/369) 2024-04-03T16:13:42.2656991Z remote: Compressing objects: 24% (89/369) 2024-04-03T16:13:42.2657878Z remote: Compressing objects: 25% (93/369) 2024-04-03T16:13:42.2664181Z remote: Compressing objects: 26% (96/369) 2024-04-03T16:13:42.2665070Z remote: Compressing objects: 27% (100/369) 2024-04-03T16:13:42.2673859Z remote: Compressing objects: 28% (104/369) 2024-04-03T16:13:42.2678845Z remote: Compressing objects: 29% (108/369) 2024-04-03T16:13:42.2683908Z remote: Compressing objects: 30% (111/369) 2024-04-03T16:13:42.2687230Z remote: Compressing objects: 31% (115/369) 2024-04-03T16:13:42.2688117Z remote: Compressing objects: 32% (119/369) 2024-04-03T16:13:42.2705111Z remote: Compressing objects: 33% (122/369) 2024-04-03T16:13:42.2705976Z remote: Compressing objects: 34% (126/369) 2024-04-03T16:13:42.2712616Z remote: Compressing objects: 35% (130/369) 2024-04-03T16:13:42.2722949Z remote: Compressing objects: 36% (133/369) 2024-04-03T16:13:42.2723817Z remote: Compressing objects: 37% (137/369) 2024-04-03T16:13:42.2724684Z remote: Compressing objects: 38% (141/369) 2024-04-03T16:13:42.2731464Z remote: Compressing objects: 39% (144/369) 2024-04-03T16:13:42.2732352Z remote: Compressing objects: 40% (148/369) 2024-04-03T16:13:42.2744013Z remote: Compressing objects: 41% (152/369) 2024-04-03T16:13:42.2745128Z remote: Compressing objects: 42% (155/369) 2024-04-03T16:13:42.2746368Z remote: Compressing objects: 43% (159/369) 2024-04-03T16:13:42.2747850Z remote: Compressing objects: 44% (163/369) 2024-04-03T16:13:42.2749319Z remote: Compressing objects: 45% (167/369) 2024-04-03T16:13:42.2750208Z remote: Compressing objects: 46% (170/369) 2024-04-03T16:13:42.2754256Z remote: Compressing objects: 47% (174/369) 2024-04-03T16:13:42.2756478Z remote: Compressing objects: 48% (178/369) 2024-04-03T16:13:42.2757619Z remote: Compressing objects: 49% (181/369) 2024-04-03T16:13:42.2759049Z remote: Compressing objects: 50% (185/369) 2024-04-03T16:13:42.2761196Z remote: Compressing objects: 51% (189/369) 2024-04-03T16:13:42.2763021Z remote: Compressing objects: 52% (192/369) 2024-04-03T16:13:42.2764602Z remote: Compressing objects: 53% (196/369) 2024-04-03T16:13:42.2766174Z remote: Compressing objects: 54% (200/369) 2024-04-03T16:13:42.2767397Z remote: Compressing objects: 55% (203/369) 2024-04-03T16:13:42.2768742Z remote: Compressing objects: 56% (207/369) 2024-04-03T16:13:42.2769617Z remote: Compressing objects: 57% (211/369) 2024-04-03T16:13:42.2770455Z remote: Compressing objects: 58% (215/369) 2024-04-03T16:13:42.2771995Z remote: Compressing objects: 59% (218/369) 2024-04-03T16:13:42.2773020Z remote: Compressing objects: 60% (222/369) 2024-04-03T16:13:42.2773897Z remote: Compressing objects: 61% (226/369) 2024-04-03T16:13:42.2774756Z remote: Compressing objects: 62% (229/369) 2024-04-03T16:13:42.2775615Z remote: Compressing objects: 63% (233/369) 2024-04-03T16:13:42.2777080Z remote: Compressing objects: 64% (237/369) 2024-04-03T16:13:42.2778512Z remote: Compressing objects: 65% (240/369) 2024-04-03T16:13:42.2779389Z remote: Compressing objects: 66% (244/369) 2024-04-03T16:13:42.2780226Z remote: Compressing objects: 67% (248/369) 2024-04-03T16:13:42.2781343Z remote: Compressing objects: 68% (251/369) 2024-04-03T16:13:42.2782779Z remote: Compressing objects: 69% (255/369) 2024-04-03T16:13:42.2783644Z remote: Compressing objects: 70% (259/369) 2024-04-03T16:13:42.2785161Z remote: Compressing objects: 71% (262/369) 2024-04-03T16:13:42.2786651Z remote: Compressing objects: 72% (266/369) 2024-04-03T16:13:42.2788014Z remote: Compressing objects: 73% (270/369) 2024-04-03T16:13:42.2789537Z remote: Compressing objects: 74% (274/369) 2024-04-03T16:13:42.2790423Z remote: Compressing objects: 75% (277/369) 2024-04-03T16:13:42.2792042Z remote: Compressing objects: 76% (281/369) 2024-04-03T16:13:42.2792917Z remote: Compressing objects: 77% (285/369) 2024-04-03T16:13:42.2794788Z remote: Compressing objects: 78% (288/369) 2024-04-03T16:13:42.2795983Z remote: Compressing objects: 79% (292/369) 2024-04-03T16:13:42.2797371Z remote: Compressing objects: 80% (296/369) 2024-04-03T16:13:42.2798506Z remote: Compressing objects: 81% (299/369) 2024-04-03T16:13:42.2800063Z remote: Compressing objects: 82% (303/369) 2024-04-03T16:13:42.2801250Z remote: Compressing objects: 83% (307/369) 2024-04-03T16:13:42.2802626Z remote: Compressing objects: 84% (310/369) 2024-04-03T16:13:42.2803877Z remote: Compressing objects: 85% (314/369) 2024-04-03T16:13:42.2805572Z remote: Compressing objects: 86% (318/369) 2024-04-03T16:13:42.2806455Z remote: Compressing objects: 87% (322/369) 2024-04-03T16:13:42.2807310Z remote: Compressing objects: 88% (325/369) 2024-04-03T16:13:42.2808194Z remote: Compressing objects: 89% (329/369) 2024-04-03T16:13:42.2809045Z remote: Compressing objects: 90% (333/369) 2024-04-03T16:13:42.2809898Z remote: Compressing objects: 91% (336/369) 2024-04-03T16:13:42.2810736Z remote: Compressing objects: 92% (340/369) 2024-04-03T16:13:42.2811597Z remote: Compressing objects: 93% (344/369) 2024-04-03T16:13:42.2812480Z remote: Compressing objects: 94% (347/369) 2024-04-03T16:13:42.2813313Z remote: Compressing objects: 95% (351/369) 2024-04-03T16:13:42.2814165Z remote: Compressing objects: 96% (355/369) 2024-04-03T16:13:42.2815018Z remote: Compressing objects: 97% (358/369) 2024-04-03T16:13:42.2815855Z remote: Compressing objects: 98% (362/369) 2024-04-03T16:13:42.2816722Z remote: Compressing objects: 99% (366/369) 2024-04-03T16:13:42.2817571Z remote: Compressing objects: 100% (369/369) 2024-04-03T16:13:42.2818461Z remote: Compressing objects: 100% (369/369), done. 2024-04-03T16:14:14.3871566Z remote: Total 1124411 (delta 364), reused 532 (delta 295), pack-reused 1123747 2024-04-03T16:14:43.3618469Z [command]/usr/bin/git rev-parse --verify --quiet d253bd7befc64137de913dbb8727b511f1052411^{object} 2024-04-03T16:14:43.3648154Z d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:14:43.3654479Z ##[endgroup] 2024-04-03T16:14:43.3655272Z ##[group]Determining the checkout info 2024-04-03T16:14:43.3656986Z ##[endgroup] 2024-04-03T16:14:43.3657910Z ##[group]Checking out the ref 2024-04-03T16:14:43.3662355Z [command]/usr/bin/git checkout --quiet --force d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:14:45.5966164Z ##[endgroup] 2024-04-03T16:14:45.5967016Z ##[group]Setting up auth for fetching submodules 2024-04-03T16:14:45.5973169Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-04-03T16:14:45.6028931Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-04-03T16:14:45.6063805Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-04-03T16:14:45.6099017Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-04-03T16:14:45.6131083Z ##[endgroup] 2024-04-03T16:14:45.6131814Z ##[group]Fetching submodules 2024-04-03T16:14:45.6137147Z [command]/usr/bin/git submodule sync --recursive 2024-04-03T16:14:45.6498737Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-04-03T16:14:45.6845262Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-04-03T16:14:45.6847810Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-04-03T16:14:45.6851800Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-04-03T16:14:45.6855295Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-04-03T16:14:45.6858989Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2024-04-03T16:14:45.6863786Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-04-03T16:14:45.6866883Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-04-03T16:14:45.6871666Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-04-03T16:14:45.6876072Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-04-03T16:14:45.6880941Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2024-04-03T16:14:45.6885924Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-04-03T16:14:45.6890719Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-04-03T16:14:45.6895578Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-04-03T16:14:45.6900792Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-04-03T16:14:45.6906286Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-04-03T16:14:45.6911585Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-04-03T16:14:45.6918002Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-04-03T16:14:45.6924220Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:14:45.6929862Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-04-03T16:14:45.6936108Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-04-03T16:14:45.6942275Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-04-03T16:14:45.6948894Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2024-04-03T16:14:45.6955783Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-04-03T16:14:45.6963116Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-04-03T16:14:45.6970002Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-04-03T16:14:45.6976923Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-04-03T16:14:45.6984136Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2024-04-03T16:14:45.6991362Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-04-03T16:14:45.6999266Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-04-03T16:14:45.7006770Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2024-04-03T16:14:45.7014504Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-04-03T16:14:45.7022341Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-04-03T16:14:45.7030367Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-04-03T16:14:45.7039396Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-04-03T16:14:45.7047405Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-04-03T16:14:45.7055822Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-04-03T16:14:45.7064118Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-04-03T16:14:45.7072790Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2024-04-03T16:14:45.7082509Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-04-03T16:14:45.7091546Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2024-04-03T16:14:45.7124259Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-04-03T16:14:46.1226709Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-04-03T16:14:46.3633188Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-04-03T16:14:46.5913018Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-04-03T16:14:46.9400895Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2024-04-03T16:14:47.2637703Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-04-03T16:14:49.6745999Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-04-03T16:15:01.4564704Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-04-03T16:15:01.9931073Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-04-03T16:15:02.8305483Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2024-04-03T16:15:04.5914581Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-04-03T16:15:06.1650712Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-04-03T16:15:09.2805443Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-04-03T16:15:17.3701750Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-04-03T16:15:18.7567508Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-04-03T16:15:20.6946017Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-04-03T16:15:22.1244470Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-04-03T16:15:22.2989402Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-04-03T16:15:22.8585047Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-04-03T16:15:23.2104164Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-04-03T16:15:24.5277299Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-04-03T16:15:24.8941044Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2024-04-03T16:15:25.0773530Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-04-03T16:15:25.3471308Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-04-03T16:15:27.3111223Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-04-03T16:15:28.0303997Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-04-03T16:15:28.6971225Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2024-04-03T16:15:28.9977017Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-04-03T16:15:37.1776300Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-04-03T16:15:39.9369907Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2024-04-03T16:15:40.4732202Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-04-03T16:15:40.7004545Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-04-03T16:15:51.0971097Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-04-03T16:15:51.3102118Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-04-03T16:15:51.5265506Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-04-03T16:15:52.6765565Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-04-03T16:15:53.0019887Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-04-03T16:15:53.6943272Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2024-04-03T16:15:56.3869240Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-04-03T16:15:56.8646384Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2024-04-03T16:15:59.8386640Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-04-03T16:15:59.8527646Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-04-03T16:15:59.8633554Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-04-03T16:15:59.8943984Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-04-03T16:15:59.9246451Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2024-04-03T16:15:59.9754993Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-04-03T16:16:01.2907512Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-04-03T16:16:01.3192948Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-04-03T16:16:01.4505131Z Submodule path 'third_party/cpuinfo': checked out 'd6860c477c99f1fce9e28eb206891af3c0e1a1d7' 2024-04-03T16:16:01.4962588Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2024-04-03T16:16:01.5325774Z Submodule path 'third_party/cudnn_frontend': checked out '150798fe976556078f443fdb059a1ff0361f58a2' 2024-04-03T16:16:02.2650374Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-04-03T16:16:02.6202651Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-04-03T16:16:02.7139767Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-04-03T16:16:02.7159963Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:16:02.7163083Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:16:02.7166960Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:16:02.7170657Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:16:02.7175082Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:16:02.7205117Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-04-03T16:16:03.8577609Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-04-03T16:16:04.5662460Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-04-03T16:16:07.7337305Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-04-03T16:16:09.0732602Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-04-03T16:16:09.5004987Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-04-03T16:16:09.6355195Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-04-03T16:16:10.2265216Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-04-03T16:16:10.3123039Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-04-03T16:16:10.3264743Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-04-03T16:16:10.4790883Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-04-03T16:16:10.5267452Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-04-03T16:16:10.5381237Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-04-03T16:16:10.5915145Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-04-03T16:16:10.6241102Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-04-03T16:16:10.6838731Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-04-03T16:16:10.6995858Z Submodule path 'third_party/ideep': checked out '8a6cc4e09dc509f04f83c085e38786b1fb44e14d' 2024-04-03T16:16:10.7014885Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-04-03T16:16:10.7043946Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-04-03T16:16:24.6634084Z Submodule path 'third_party/ideep/mkl-dnn': checked out '86e6af5974177e513fd3fee58425e1063e7f1361' 2024-04-03T16:16:24.6761897Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2024-04-03T16:16:24.6974554Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-04-03T16:16:24.8325724Z Submodule path 'third_party/kineto': checked out '8466a8b111b36dc725e6855d52a0b133d925a8e0' 2024-04-03T16:16:24.8345695Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:16:24.8348690Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:16:24.8353015Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:16:24.8381977Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-04-03T16:16:25.8236451Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-04-03T16:16:27.2604365Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-04-03T16:16:28.7161175Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-04-03T16:16:28.7180007Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:16:28.7183181Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:16:28.7187214Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:16:28.7190809Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:16:28.7195377Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:16:28.7199895Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:16:28.7203592Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:16:28.7208350Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:16:28.7239212Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-04-03T16:16:30.0073263Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-04-03T16:16:30.4052550Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-04-03T16:16:33.3523880Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-04-03T16:16:34.2051515Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-04-03T16:16:34.6103645Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-04-03T16:16:35.9378263Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-04-03T16:16:44.0741381Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-04-03T16:16:44.5433116Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-04-03T16:16:44.5666016Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-04-03T16:16:44.6167764Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-04-03T16:16:44.6331767Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-04-03T16:16:44.6350316Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:16:44.6380349Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-04-03T16:16:45.3927410Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-04-03T16:16:45.4161482Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-04-03T16:16:45.4705290Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-04-03T16:16:45.6157200Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-04-03T16:16:45.6365926Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-04-03T16:16:45.6867892Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-04-03T16:16:45.7651690Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-04-03T16:16:45.8134301Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-04-03T16:16:45.8433922Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-04-03T16:16:45.8609477Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2024-04-03T16:16:46.0105255Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-04-03T16:16:46.4658922Z Submodule path 'third_party/onnx': checked out 'ccde5da81388ffa770ca98b64e07f803ad089414' 2024-04-03T16:16:46.4699317Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-04-03T16:16:46.4702234Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-04-03T16:16:46.4732808Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-04-03T16:16:46.9055113Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-04-03T16:16:48.0669255Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-04-03T16:16:48.1124798Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-04-03T16:16:48.1360942Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2024-04-03T16:16:48.1380101Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:16:48.1408753Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2024-04-03T16:16:50.9641542Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2024-04-03T16:16:50.9666494Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:16:50.9669544Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:16:50.9699935Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2024-04-03T16:16:51.4052955Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2024-04-03T16:16:52.5385969Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2024-04-03T16:16:52.6376669Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2024-04-03T16:16:52.6394645Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:16:52.6422964Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2024-04-03T16:16:52.8337115Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-04-03T16:16:52.8452262Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-04-03T16:16:53.2135868Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-04-03T16:16:53.2161448Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:16:53.2164447Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-04-03T16:16:53.2194401Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-04-03T16:16:53.6708586Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-04-03T16:16:54.9992097Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-04-03T16:16:55.0964518Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-04-03T16:16:55.1068834Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-04-03T16:16:55.1217403Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-04-03T16:16:55.1655091Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-04-03T16:16:55.2022798Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-04-03T16:16:55.2575983Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-04-03T16:16:55.4108584Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2024-04-03T16:16:55.4446009Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-04-03T16:16:55.4465482Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:16:55.4468082Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:16:55.4471941Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:16:55.4475888Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:16:55.4505360Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-04-03T16:16:56.7584251Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-04-03T16:16:57.0044623Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-04-03T16:16:58.5217796Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-04-03T16:16:59.7049364Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-04-03T16:16:59.7237704Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-04-03T16:16:59.8164583Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-04-03T16:16:59.8536418Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-04-03T16:16:59.8555185Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:16:59.8584653Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-04-03T16:17:00.0698131Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-04-03T16:17:00.2716841Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2024-04-03T16:17:00.2759373Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-04-03T16:17:00.3121788Z Entering 'android/libs/fbjni' 2024-04-03T16:17:00.3169119Z Entering 'third_party/FP16' 2024-04-03T16:17:00.3216275Z Entering 'third_party/FXdiv' 2024-04-03T16:17:00.3263335Z Entering 'third_party/NNPACK' 2024-04-03T16:17:00.3310090Z Entering 'third_party/QNNPACK' 2024-04-03T16:17:00.3358049Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:17:00.3406078Z Entering 'third_party/XNNPACK' 2024-04-03T16:17:00.3471605Z Entering 'third_party/benchmark' 2024-04-03T16:17:00.3519031Z Entering 'third_party/cpuinfo' 2024-04-03T16:17:00.3565936Z Entering 'third_party/cub' 2024-04-03T16:17:00.3612771Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:17:00.3660308Z Entering 'third_party/cutlass' 2024-04-03T16:17:00.3715876Z Entering 'third_party/eigen' 2024-04-03T16:17:00.3765280Z Entering 'third_party/fbgemm' 2024-04-03T16:17:00.3812564Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:17:00.3858294Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:17:00.3904353Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:17:00.3959303Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:17:00.4004547Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:17:00.4050816Z Entering 'third_party/flatbuffers' 2024-04-03T16:17:00.4101867Z Entering 'third_party/fmt' 2024-04-03T16:17:00.4149327Z Entering 'third_party/foxi' 2024-04-03T16:17:00.4197192Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:17:00.4244265Z Entering 'third_party/gloo' 2024-04-03T16:17:00.4291673Z Entering 'third_party/googletest' 2024-04-03T16:17:00.4338342Z Entering 'third_party/ideep' 2024-04-03T16:17:00.4384327Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:17:00.4438750Z Entering 'third_party/ios-cmake' 2024-04-03T16:17:00.4485681Z Entering 'third_party/ittapi' 2024-04-03T16:17:00.4531227Z Entering 'third_party/kineto' 2024-04-03T16:17:00.4577665Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:17:00.4623371Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:17:00.4670780Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:17:00.4715944Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:17:00.4763027Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:17:00.4807032Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:17:00.4853848Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:17:00.4899307Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:17:00.4945472Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:17:00.4992060Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:17:00.5039473Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:17:00.5085478Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:17:00.5131347Z Entering 'third_party/mimalloc' 2024-04-03T16:17:00.5179558Z Entering 'third_party/nccl/nccl' 2024-04-03T16:17:00.5225818Z Entering 'third_party/neon2sse' 2024-04-03T16:17:00.5271769Z Entering 'third_party/nlohmann' 2024-04-03T16:17:00.5320689Z Entering 'third_party/onnx' 2024-04-03T16:17:00.5385450Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:17:00.5431089Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:17:00.5478692Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:17:00.5524225Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:17:00.5575483Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:17:00.5621600Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:17:00.5666356Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:17:00.5718585Z Entering 'third_party/pocketfft' 2024-04-03T16:17:00.5766181Z Entering 'third_party/protobuf' 2024-04-03T16:17:00.5815991Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:17:00.5860770Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:17:00.5908874Z Entering 'third_party/psimd' 2024-04-03T16:17:00.5955529Z Entering 'third_party/pthreadpool' 2024-04-03T16:17:00.6001018Z Entering 'third_party/pybind11' 2024-04-03T16:17:00.6047942Z Entering 'third_party/python-peachpy' 2024-04-03T16:17:00.6094157Z Entering 'third_party/sleef' 2024-04-03T16:17:00.6140760Z Entering 'third_party/tbb' 2024-04-03T16:17:00.6188815Z Entering 'third_party/tensorpipe' 2024-04-03T16:17:00.6235022Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:17:00.6281302Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:17:00.6326109Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:17:00.6371212Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:17:00.6416219Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:17:00.6464003Z Entering 'third_party/zstd' 2024-04-03T16:17:00.6526974Z ##[endgroup] 2024-04-03T16:17:00.6529842Z ##[group]Persisting credentials for submodules 2024-04-03T16:17:00.6533429Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-04-03T16:17:00.6883886Z Entering 'android/libs/fbjni' 2024-04-03T16:17:00.6941394Z Entering 'third_party/FP16' 2024-04-03T16:17:00.6999861Z Entering 'third_party/FXdiv' 2024-04-03T16:17:00.7056599Z Entering 'third_party/NNPACK' 2024-04-03T16:17:00.7114533Z Entering 'third_party/QNNPACK' 2024-04-03T16:17:00.7174297Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:17:00.7231844Z Entering 'third_party/XNNPACK' 2024-04-03T16:17:00.7309360Z Entering 'third_party/benchmark' 2024-04-03T16:17:00.7367441Z Entering 'third_party/cpuinfo' 2024-04-03T16:17:00.7426795Z Entering 'third_party/cub' 2024-04-03T16:17:00.7484477Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:17:00.7541913Z Entering 'third_party/cutlass' 2024-04-03T16:17:00.7608592Z Entering 'third_party/eigen' 2024-04-03T16:17:00.7668236Z Entering 'third_party/fbgemm' 2024-04-03T16:17:00.7725996Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:17:00.7782155Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:17:00.7838204Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:17:00.7902515Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:17:00.7958289Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:17:00.8015096Z Entering 'third_party/flatbuffers' 2024-04-03T16:17:00.8076341Z Entering 'third_party/fmt' 2024-04-03T16:17:00.8134429Z Entering 'third_party/foxi' 2024-04-03T16:17:00.8193354Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:17:00.8251142Z Entering 'third_party/gloo' 2024-04-03T16:17:00.8309192Z Entering 'third_party/googletest' 2024-04-03T16:17:00.8368539Z Entering 'third_party/ideep' 2024-04-03T16:17:00.8425791Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:17:00.8492704Z Entering 'third_party/ios-cmake' 2024-04-03T16:17:00.8550554Z Entering 'third_party/ittapi' 2024-04-03T16:17:00.8608557Z Entering 'third_party/kineto' 2024-04-03T16:17:00.8665364Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:17:00.8724559Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:17:00.8784518Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:17:00.8843124Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:17:00.8900288Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:17:00.8955566Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:17:00.9015694Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:17:00.9073453Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:17:00.9132501Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:17:00.9191005Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:17:00.9251147Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:17:00.9308128Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:17:00.9367145Z Entering 'third_party/mimalloc' 2024-04-03T16:17:00.9425241Z Entering 'third_party/nccl/nccl' 2024-04-03T16:17:00.9483387Z Entering 'third_party/neon2sse' 2024-04-03T16:17:00.9541201Z Entering 'third_party/nlohmann' 2024-04-03T16:17:00.9600145Z Entering 'third_party/onnx' 2024-04-03T16:17:00.9677364Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:17:00.9734048Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:17:00.9794336Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:17:00.9851926Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:17:00.9915165Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:17:00.9973173Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:17:01.0030865Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:17:01.0095837Z Entering 'third_party/pocketfft' 2024-04-03T16:17:01.0155350Z Entering 'third_party/protobuf' 2024-04-03T16:17:01.0218249Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:17:01.0276405Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:17:01.0335267Z Entering 'third_party/psimd' 2024-04-03T16:17:01.0393090Z Entering 'third_party/pthreadpool' 2024-04-03T16:17:01.0452279Z Entering 'third_party/pybind11' 2024-04-03T16:17:01.0511252Z Entering 'third_party/python-peachpy' 2024-04-03T16:17:01.0569087Z Entering 'third_party/sleef' 2024-04-03T16:17:01.0626336Z Entering 'third_party/tbb' 2024-04-03T16:17:01.0687514Z Entering 'third_party/tensorpipe' 2024-04-03T16:17:01.0746127Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:17:01.0803233Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:17:01.0860718Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:17:01.0917815Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:17:01.0972576Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:17:01.1031379Z Entering 'third_party/zstd' 2024-04-03T16:17:01.1106432Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-04-03T16:17:01.1455811Z Entering 'android/libs/fbjni' 2024-04-03T16:17:01.1508088Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-04-03T16:17:01.1528635Z Entering 'third_party/FP16' 2024-04-03T16:17:01.1581766Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-04-03T16:17:01.1602508Z Entering 'third_party/FXdiv' 2024-04-03T16:17:01.1655247Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-04-03T16:17:01.1676052Z Entering 'third_party/NNPACK' 2024-04-03T16:17:01.1729137Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-04-03T16:17:01.1750334Z Entering 'third_party/QNNPACK' 2024-04-03T16:17:01.1803502Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2024-04-03T16:17:01.1824140Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:17:01.1877514Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-04-03T16:17:01.1897959Z Entering 'third_party/XNNPACK' 2024-04-03T16:17:01.1951179Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-04-03T16:17:01.1991664Z Entering 'third_party/benchmark' 2024-04-03T16:17:01.2043955Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:17:01.2064267Z Entering 'third_party/cpuinfo' 2024-04-03T16:17:01.2117899Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-04-03T16:17:01.2138964Z Entering 'third_party/cub' 2024-04-03T16:17:01.2191138Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2024-04-03T16:17:01.2211860Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:17:01.2265741Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-04-03T16:17:01.2286520Z Entering 'third_party/cutlass' 2024-04-03T16:17:01.2339044Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-04-03T16:17:01.2369833Z Entering 'third_party/eigen' 2024-04-03T16:17:01.2422038Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-04-03T16:17:01.2446381Z Entering 'third_party/fbgemm' 2024-04-03T16:17:01.2499350Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-04-03T16:17:01.2520619Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:17:01.2573666Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-04-03T16:17:01.2594002Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:17:01.2647963Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-04-03T16:17:01.2668881Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:17:01.2721469Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-04-03T16:17:01.2749603Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:17:01.2803413Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:17:01.2823901Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:17:01.2876551Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-04-03T16:17:01.2898220Z Entering 'third_party/flatbuffers' 2024-04-03T16:17:01.2951477Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-04-03T16:17:01.2975331Z Entering 'third_party/fmt' 2024-04-03T16:17:01.3028824Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-04-03T16:17:01.3050103Z Entering 'third_party/foxi' 2024-04-03T16:17:01.3102755Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-04-03T16:17:01.3123732Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:17:01.3177155Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-04-03T16:17:01.3197786Z Entering 'third_party/gloo' 2024-04-03T16:17:01.3250312Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-04-03T16:17:01.3270965Z Entering 'third_party/googletest' 2024-04-03T16:17:01.3324270Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:17:01.3345131Z Entering 'third_party/ideep' 2024-04-03T16:17:01.3398309Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-04-03T16:17:01.3417374Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:17:01.3469716Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-04-03T16:17:01.3498627Z Entering 'third_party/ios-cmake' 2024-04-03T16:17:01.3550531Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2024-04-03T16:17:01.3571034Z Entering 'third_party/ittapi' 2024-04-03T16:17:01.3623855Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-04-03T16:17:01.3646504Z Entering 'third_party/kineto' 2024-04-03T16:17:01.3700646Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-04-03T16:17:01.3721494Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:17:01.3774956Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-04-03T16:17:01.3794825Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:17:01.3848360Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-04-03T16:17:01.3869960Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:17:01.3924715Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-04-03T16:17:01.3944479Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:17:01.3998373Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-04-03T16:17:01.4018567Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:17:01.4073036Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-04-03T16:17:01.4092245Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:17:01.4146588Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-04-03T16:17:01.4168877Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:17:01.4222347Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-04-03T16:17:01.4243772Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:17:01.4296791Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:17:01.4317499Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:17:01.4370545Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-04-03T16:17:01.4391693Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:17:01.4445742Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-04-03T16:17:01.4467319Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:17:01.4521198Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-04-03T16:17:01.4541310Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:17:01.4594667Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-04-03T16:17:01.4616035Z Entering 'third_party/mimalloc' 2024-04-03T16:17:01.4669959Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-04-03T16:17:01.4690934Z Entering 'third_party/nccl/nccl' 2024-04-03T16:17:01.4743943Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-04-03T16:17:01.4765371Z Entering 'third_party/neon2sse' 2024-04-03T16:17:01.4818570Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2024-04-03T16:17:01.4839259Z Entering 'third_party/nlohmann' 2024-04-03T16:17:01.4892629Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-04-03T16:17:01.4914135Z Entering 'third_party/onnx' 2024-04-03T16:17:01.4967377Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-04-03T16:17:01.5007216Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:17:01.5060064Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:17:01.5080956Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:17:01.5133101Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:17:01.5156320Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:17:01.5209876Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2024-04-03T16:17:01.5228719Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:17:01.5282244Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2024-04-03T16:17:01.5307343Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:17:01.5362053Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:17:01.5382023Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:17:01.5435737Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:17:01.5455107Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:17:01.5508777Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-04-03T16:17:01.5534170Z Entering 'third_party/pocketfft' 2024-04-03T16:17:01.5587201Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-04-03T16:17:01.5607949Z Entering 'third_party/protobuf' 2024-04-03T16:17:01.5661366Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-04-03T16:17:01.5686237Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:17:01.5738597Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:17:01.5758847Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:17:01.5811424Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:17:01.5833130Z Entering 'third_party/psimd' 2024-04-03T16:17:01.5887140Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-04-03T16:17:01.5907385Z Entering 'third_party/pthreadpool' 2024-04-03T16:17:01.5961603Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-04-03T16:17:01.5981914Z Entering 'third_party/pybind11' 2024-04-03T16:17:01.6035501Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:17:01.6057372Z Entering 'third_party/python-peachpy' 2024-04-03T16:17:01.6111231Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-04-03T16:17:01.6132015Z Entering 'third_party/sleef' 2024-04-03T16:17:01.6184475Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-04-03T16:17:01.6205678Z Entering 'third_party/tbb' 2024-04-03T16:17:01.6258324Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2024-04-03T16:17:01.6281916Z Entering 'third_party/tensorpipe' 2024-04-03T16:17:01.6336298Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-04-03T16:17:01.6356931Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:17:01.6410275Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:17:01.6430948Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:17:01.6484729Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-04-03T16:17:01.6504467Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:17:01.6557897Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-04-03T16:17:01.6578361Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:17:01.6631366Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:17:01.6650456Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:17:01.6703408Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-04-03T16:17:01.6727968Z Entering 'third_party/zstd' 2024-04-03T16:17:01.6780875Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2024-04-03T16:17:01.7943283Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-04-03T16:17:01.8293101Z Entering 'android/libs/fbjni' 2024-04-03T16:17:01.8339525Z Entering 'third_party/FP16' 2024-04-03T16:17:01.8386351Z Entering 'third_party/FXdiv' 2024-04-03T16:17:01.8433344Z Entering 'third_party/NNPACK' 2024-04-03T16:17:01.8481089Z Entering 'third_party/QNNPACK' 2024-04-03T16:17:01.8528194Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:17:01.8574775Z Entering 'third_party/XNNPACK' 2024-04-03T16:17:01.8641256Z Entering 'third_party/benchmark' 2024-04-03T16:17:01.8687671Z Entering 'third_party/cpuinfo' 2024-04-03T16:17:01.8733712Z Entering 'third_party/cub' 2024-04-03T16:17:01.8780270Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:17:01.8827629Z Entering 'third_party/cutlass' 2024-04-03T16:17:01.8884163Z Entering 'third_party/eigen' 2024-04-03T16:17:01.8932725Z Entering 'third_party/fbgemm' 2024-04-03T16:17:01.8979909Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:17:01.9025012Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:17:01.9071351Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:17:01.9125357Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:17:01.9170876Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:17:01.9217818Z Entering 'third_party/flatbuffers' 2024-04-03T16:17:01.9268530Z Entering 'third_party/fmt' 2024-04-03T16:17:01.9315240Z Entering 'third_party/foxi' 2024-04-03T16:17:01.9362235Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:17:01.9411289Z Entering 'third_party/gloo' 2024-04-03T16:17:01.9459846Z Entering 'third_party/googletest' 2024-04-03T16:17:01.9508149Z Entering 'third_party/ideep' 2024-04-03T16:17:01.9554273Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:17:01.9609224Z Entering 'third_party/ios-cmake' 2024-04-03T16:17:01.9656002Z Entering 'third_party/ittapi' 2024-04-03T16:17:01.9703682Z Entering 'third_party/kineto' 2024-04-03T16:17:01.9749904Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:17:01.9795556Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:17:01.9844061Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:17:01.9889380Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:17:01.9935323Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:17:01.9980780Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:17:02.0028260Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:17:02.0075139Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:17:02.0122013Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:17:02.0169572Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:17:02.0216759Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:17:02.0262843Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:17:02.0311524Z Entering 'third_party/mimalloc' 2024-04-03T16:17:02.0359751Z Entering 'third_party/nccl/nccl' 2024-04-03T16:17:02.0406803Z Entering 'third_party/neon2sse' 2024-04-03T16:17:02.0454939Z Entering 'third_party/nlohmann' 2024-04-03T16:17:02.0542976Z Entering 'third_party/onnx' 2024-04-03T16:17:02.0610163Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:17:02.0658364Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:17:02.0708525Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:17:02.0754999Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:17:02.0807905Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:17:02.0855204Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:17:02.0903427Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:17:02.0955746Z Entering 'third_party/pocketfft' 2024-04-03T16:17:02.1002956Z Entering 'third_party/protobuf' 2024-04-03T16:17:02.1054155Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:17:02.1100311Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:17:02.1148231Z Entering 'third_party/psimd' 2024-04-03T16:17:02.1196991Z Entering 'third_party/pthreadpool' 2024-04-03T16:17:02.1244245Z Entering 'third_party/pybind11' 2024-04-03T16:17:02.1291010Z Entering 'third_party/python-peachpy' 2024-04-03T16:17:02.1339961Z Entering 'third_party/sleef' 2024-04-03T16:17:02.1388223Z Entering 'third_party/tbb' 2024-04-03T16:17:02.1439365Z Entering 'third_party/tensorpipe' 2024-04-03T16:17:02.1487263Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:17:02.1533198Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:17:02.1578767Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:17:02.1624709Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:17:02.1669590Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:17:02.1719562Z Entering 'third_party/zstd' 2024-04-03T16:17:02.1787669Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-04-03T16:17:02.2144597Z Entering 'android/libs/fbjni' 2024-04-03T16:17:02.2192747Z Entering 'third_party/FP16' 2024-04-03T16:17:02.2241295Z Entering 'third_party/FXdiv' 2024-04-03T16:17:02.2289229Z Entering 'third_party/NNPACK' 2024-04-03T16:17:02.2337161Z Entering 'third_party/QNNPACK' 2024-04-03T16:17:02.2385472Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:17:02.2432729Z Entering 'third_party/XNNPACK' 2024-04-03T16:17:02.2501733Z Entering 'third_party/benchmark' 2024-04-03T16:17:02.2549036Z Entering 'third_party/cpuinfo' 2024-04-03T16:17:02.2598457Z Entering 'third_party/cub' 2024-04-03T16:17:02.2645279Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:17:02.2692873Z Entering 'third_party/cutlass' 2024-04-03T16:17:02.2749166Z Entering 'third_party/eigen' 2024-04-03T16:17:02.2800016Z Entering 'third_party/fbgemm' 2024-04-03T16:17:02.2848036Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:17:02.2893445Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:17:02.2941682Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:17:02.2995555Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:17:02.3041586Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:17:02.3088625Z Entering 'third_party/flatbuffers' 2024-04-03T16:17:02.3138443Z Entering 'third_party/fmt' 2024-04-03T16:17:02.3185633Z Entering 'third_party/foxi' 2024-04-03T16:17:02.3232846Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:17:02.3281466Z Entering 'third_party/gloo' 2024-04-03T16:17:02.3327629Z Entering 'third_party/googletest' 2024-04-03T16:17:02.3374317Z Entering 'third_party/ideep' 2024-04-03T16:17:02.3420203Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:17:02.3473636Z Entering 'third_party/ios-cmake' 2024-04-03T16:17:02.3520640Z Entering 'third_party/ittapi' 2024-04-03T16:17:02.3567078Z Entering 'third_party/kineto' 2024-04-03T16:17:02.3614236Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:17:02.3659787Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:17:02.3707302Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:17:02.3754754Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:17:02.3802145Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:17:02.3847426Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:17:02.3895833Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:17:02.3941433Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:17:02.3987279Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:17:02.4035150Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:17:02.4083417Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:17:02.4128803Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:17:02.4175755Z Entering 'third_party/mimalloc' 2024-04-03T16:17:02.4223672Z Entering 'third_party/nccl/nccl' 2024-04-03T16:17:02.4270660Z Entering 'third_party/neon2sse' 2024-04-03T16:17:02.4317970Z Entering 'third_party/nlohmann' 2024-04-03T16:17:02.4367909Z Entering 'third_party/onnx' 2024-04-03T16:17:02.4433131Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:17:02.4480488Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:17:02.4528906Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:17:02.4575914Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:17:02.4626849Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:17:02.4673120Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:17:02.4719179Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:17:02.4769910Z Entering 'third_party/pocketfft' 2024-04-03T16:17:02.4816246Z Entering 'third_party/protobuf' 2024-04-03T16:17:02.4887150Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:17:02.4912984Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:17:02.4961179Z Entering 'third_party/psimd' 2024-04-03T16:17:02.5008365Z Entering 'third_party/pthreadpool' 2024-04-03T16:17:02.5054779Z Entering 'third_party/pybind11' 2024-04-03T16:17:02.5102062Z Entering 'third_party/python-peachpy' 2024-04-03T16:17:02.5149116Z Entering 'third_party/sleef' 2024-04-03T16:17:02.5197098Z Entering 'third_party/tbb' 2024-04-03T16:17:02.5246300Z Entering 'third_party/tensorpipe' 2024-04-03T16:17:02.5292264Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:17:02.5337362Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:17:02.5381751Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:17:02.5427120Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:17:02.5471323Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:17:02.5519580Z Entering 'third_party/zstd' 2024-04-03T16:17:02.5582497Z ##[endgroup] 2024-04-03T16:17:02.5631379Z [command]/usr/bin/git log -1 --format='%H' 2024-04-03T16:17:02.5663708Z 'd253bd7befc64137de913dbb8727b511f1052411' 2024-04-03T16:17:02.5950646Z Prepare all required actions 2024-04-03T16:17:02.5951500Z Getting action download info 2024-04-03T16:17:02.7396104Z ##[group]Run ./.github/actions/setup-linux 2024-04-03T16:17:02.7396692Z env: 2024-04-03T16:17:02.7397081Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:17:02.7397557Z ##[endgroup] 2024-04-03T16:17:02.7490072Z ##[group]Run set -euo pipefail 2024-04-03T16:17:02.7490674Z set -euo pipefail 2024-04-03T16:17:02.7491231Z function get_ec2_metadata() { 2024-04-03T16:17:02.7492001Z  # Pulled from instance metadata endpoint for EC2 2024-04-03T16:17:02.7493287Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-04-03T16:17:02.7494413Z  category=$1 2024-04-03T16:17:02.7495168Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-04-03T16:17:02.7496080Z  runner_name_str=i-00c317a4521719f93 2024-04-03T16:17:02.7496868Z  if [[ $runner_name_str != *"gcp"* ]]; then 2024-04-03T16:17:02.7497822Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-04-03T16:17:02.7498652Z  else 2024-04-03T16:17:02.7499392Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-04-03T16:17:02.7500245Z  fi 2024-04-03T16:17:02.7500643Z } 2024-04-03T16:17:02.7501148Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-04-03T16:17:02.7501965Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-04-03T16:17:02.7502904Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-04-03T16:17:02.7503720Z echo "system info $(uname -a)" 2024-04-03T16:17:02.7513964Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:17:02.7514925Z env: 2024-04-03T16:17:02.7515312Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:17:02.7515800Z ##[endgroup] 2024-04-03T16:17:02.7630126Z ami-id: ami-0ce0c36d7a00b20e2 2024-04-03T16:17:02.7704541Z instance-id: i-00c317a4521719f93 2024-04-03T16:17:02.7778279Z instance-type: g3.8xlarge 2024-04-03T16:17:02.7788456Z system info Linux ip-10-0-56-183.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-04-03T16:17:02.7819543Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-04-03T16:17:02.7820877Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-04-03T16:17:02.7830119Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:17:02.7830831Z env: 2024-04-03T16:17:02.7831212Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:17:02.7831683Z ##[endgroup] 2024-04-03T16:17:02.7925508Z ##[group]Run if systemctl is-active --quiet docker; then 2024-04-03T16:17:02.7926364Z if systemctl is-active --quiet docker; then 2024-04-03T16:17:02.7927123Z  echo "Docker daemon is running..."; 2024-04-03T16:17:02.7927793Z else 2024-04-03T16:17:02.7928503Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-04-03T16:17:02.7929324Z fi 2024-04-03T16:17:02.7938418Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:17:02.7939196Z env: 2024-04-03T16:17:02.7939582Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:17:02.7940055Z ##[endgroup] 2024-04-03T16:17:02.7993977Z Docker daemon is running... 2024-04-03T16:17:02.8069185Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-04-03T16:17:02.8069993Z with: 2024-04-03T16:17:02.8070367Z shell: bash 2024-04-03T16:17:02.8070760Z timeout_minutes: 5 2024-04-03T16:17:02.8071213Z max_attempts: 3 2024-04-03T16:17:02.8071651Z retry_wait_seconds: 30 2024-04-03T16:17:02.8073890Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-04-03T16:17:02.8076679Z polling_interval_seconds: 1 2024-04-03T16:17:02.8077223Z warning_on_retry: true 2024-04-03T16:17:02.8077718Z continue_on_error: false 2024-04-03T16:17:02.8078168Z env: 2024-04-03T16:17:02.8078544Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:17:02.8079045Z AWS_RETRY_MODE: standard 2024-04-03T16:17:02.8079516Z AWS_MAX_ATTEMPTS: 5 2024-04-03T16:17:02.8079995Z AWS_DEFAULT_REGION: us-east-1 2024-04-03T16:17:02.8080524Z ##[endgroup] 2024-04-03T16:17:03.9497361Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-04-03T16:17:03.9498575Z Configure a credential helper to remove this warning. See 2024-04-03T16:17:03.9499813Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-04-03T16:17:03.9500593Z 2024-04-03T16:17:03.9500781Z Login Succeeded 2024-04-03T16:17:04.8910241Z Command completed after 1 attempt(s). 2024-04-03T16:17:04.8976710Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-04-03T16:17:04.8977793Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-04-03T16:17:04.8978750Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-04-03T16:17:04.8988840Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:17:04.8989574Z env: 2024-04-03T16:17:04.8989947Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:17:04.8990442Z ##[endgroup] 2024-04-03T16:17:04.9080458Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T16:17:04.9081572Z # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T16:17:04.9082411Z # shellcheck disable=SC2046 2024-04-03T16:17:04.9083071Z docker stop $(docker ps -q) || true 2024-04-03T16:17:04.9083758Z # Prune all of the docker images 2024-04-03T16:17:04.9084381Z docker system prune -af 2024-04-03T16:17:04.9093701Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:17:04.9094426Z env: 2024-04-03T16:17:04.9094822Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:17:04.9095300Z ##[endgroup] 2024-04-03T16:17:04.9572026Z "docker stop" requires at least 1 argument. 2024-04-03T16:17:04.9572900Z See 'docker stop --help'. 2024-04-03T16:17:04.9573255Z 2024-04-03T16:17:04.9573600Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-04-03T16:17:04.9574149Z 2024-04-03T16:17:04.9576570Z Stop one or more running containers 2024-04-03T16:17:04.9858694Z Total reclaimed space: 0B 2024-04-03T16:17:04.9903473Z ##[group]Run set +e 2024-04-03T16:17:04.9903986Z set +e 2024-04-03T16:17:04.9904393Z set -x 2024-04-03T16:17:04.9904802Z  2024-04-03T16:17:04.9905232Z PT_DOMAIN=download.pytorch.org 2024-04-03T16:17:04.9906375Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-04-03T16:17:04.9907968Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-04-03T16:17:04.9909081Z # one is returned at random 2024-04-03T16:17:04.9909833Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-04-03T16:17:04.9910570Z  2024-04-03T16:17:04.9911009Z if [ -z "${RESOLVED_IP}" ]; then 2024-04-03T16:17:04.9911878Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-04-03T16:17:04.9912981Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-04-03T16:17:04.9913780Z  2024-04-03T16:17:04.9914681Z  if [ -z "${RESOLVED_IP}" ]; then 2024-04-03T16:17:04.9915533Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-04-03T16:17:04.9916268Z  exit 1 2024-04-03T16:17:04.9916711Z  fi 2024-04-03T16:17:04.9917092Z fi 2024-04-03T16:17:04.9917475Z  2024-04-03T16:17:04.9917979Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-04-03T16:17:04.9919161Z  # Clean up any old records first 2024-04-03T16:17:04.9919919Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-04-03T16:17:04.9920569Z fi 2024-04-03T16:17:04.9920938Z  2024-04-03T16:17:04.9921545Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-04-03T16:17:04.9922330Z cat /etc/hosts 2024-04-03T16:17:04.9931728Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:17:04.9932427Z env: 2024-04-03T16:17:04.9932815Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:17:04.9933311Z ##[endgroup] 2024-04-03T16:17:04.9961277Z + PT_DOMAIN=download.pytorch.org 2024-04-03T16:17:04.9967103Z ++ dig -4 +short download.pytorch.org 2024-04-03T16:17:04.9968237Z ++ tail -n1 2024-04-03T16:17:05.0296459Z + RESOLVED_IP=108.138.64.125 2024-04-03T16:17:05.0297313Z + '[' -z 108.138.64.125 ']' 2024-04-03T16:17:05.0297952Z + grep -r download.pytorch.org /etc/hosts 2024-04-03T16:17:05.0310004Z + echo '108.138.64.125 download.pytorch.org' 2024-04-03T16:17:05.0310984Z + sudo tee -a /etc/hosts 2024-04-03T16:17:05.0421276Z 108.138.64.125 download.pytorch.org 2024-04-03T16:17:05.0440937Z + cat /etc/hosts 2024-04-03T16:17:05.0447463Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-04-03T16:17:05.0465225Z ::1 localhost6 localhost6.localdomain6 2024-04-03T16:17:05.0465982Z 108.138.64.125 download.pytorch.org 2024-04-03T16:17:05.0686367Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-04-03T16:17:05.0687271Z with: 2024-04-03T16:17:05.0688677Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:17:05.0690303Z docker-build-dir: .ci/docker 2024-04-03T16:17:05.0690842Z working-directory: . 2024-04-03T16:17:05.0691501Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:17:05.0692272Z force-push: false 2024-04-03T16:17:05.0692711Z env: 2024-04-03T16:17:05.0693073Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:17:05.0693566Z ##[endgroup] 2024-04-03T16:17:05.0721831Z ##[group]Run set -ex 2024-04-03T16:17:05.0722335Z set -ex 2024-04-03T16:17:05.0722749Z  2024-04-03T16:17:05.0723537Z # If the docker build directory or the build script doesn't exist, the action will 2024-04-03T16:17:05.0724982Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-04-03T16:17:05.0726140Z # job could then download the pre-built image as usual 2024-04-03T16:17:05.0727212Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-04-03T16:17:05.0728188Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-04-03T16:17:05.0729109Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:17:05.0729898Z  2024-04-03T16:17:05.0730637Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-04-03T16:17:05.0731551Z  exit 0 2024-04-03T16:17:05.0731957Z else 2024-04-03T16:17:05.0732474Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-04-03T16:17:05.0733118Z fi 2024-04-03T16:17:05.0733485Z  2024-04-03T16:17:05.0734151Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-04-03T16:17:05.0735402Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-04-03T16:17:05.0736525Z  # use it as it is, but first let's extract the tag 2024-04-03T16:17:05.0737549Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-04-03T16:17:05.0738655Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:17:05.0739672Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:17:05.0740466Z else 2024-04-03T16:17:05.0741251Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-04-03T16:17:05.0742212Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:17:05.0743504Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:17:05.0744597Z fi 2024-04-03T16:17:05.0753596Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:17:05.0754557Z env: 2024-04-03T16:17:05.0754942Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:17:05.0755432Z REPO_NAME: pytorch 2024-04-03T16:17:05.0756884Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:17:05.0758456Z DOCKER_BUILD_DIR: .ci/docker 2024-04-03T16:17:05.0759170Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:17:05.0759899Z ##[endgroup] 2024-04-03T16:17:05.0788183Z + [[ ! -d .ci/docker ]] 2024-04-03T16:17:05.0788764Z + [[ ! -f .ci/docker/build.sh ]] 2024-04-03T16:17:05.0789317Z + echo skip=false 2024-04-03T16:17:05.0791763Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-04-03T16:17:05.0796930Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:17:05.0798527Z ++ awk -F '[:,]' '{print $2}' 2024-04-03T16:17:05.0808134Z + DOCKER_TAG=16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:17:05.0809067Z + echo docker-tag=16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:17:05.0811225Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:17:05.0933239Z ##[group]Run set +e 2024-04-03T16:17:05.0933737Z set +e 2024-04-03T16:17:05.0934132Z set -x 2024-04-03T16:17:05.0934518Z  2024-04-03T16:17:05.0934866Z login() { 2024-04-03T16:17:05.0935787Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-04-03T16:17:05.0936806Z } 2024-04-03T16:17:05.0937156Z  2024-04-03T16:17:05.0937529Z retry () { 2024-04-03T16:17:05.0938056Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-04-03T16:17:05.0938729Z } 2024-04-03T16:17:05.0939090Z  2024-04-03T16:17:05.0939515Z retry login "${DOCKER_REGISTRY}" 2024-04-03T16:17:05.0940073Z  2024-04-03T16:17:05.0940718Z # Check if image already exists, if it does then skip building it 2024-04-03T16:17:05.0941719Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-04-03T16:17:05.0942404Z  exit 0 2024-04-03T16:17:05.0942813Z fi 2024-04-03T16:17:05.0943174Z  2024-04-03T16:17:05.0943855Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-04-03T16:17:05.0945033Z # be empty. The default action would be to continue rebuild the image 2024-04-03T16:17:05.0946073Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-04-03T16:17:05.0947037Z  # if we're on the base branch then use the parent commit 2024-04-03T16:17:05.0947855Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-04-03T16:17:05.0948444Z else 2024-04-03T16:17:05.0949088Z  # otherwise we're on a PR, so use the most recent base commit 2024-04-03T16:17:05.0950051Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-04-03T16:17:05.0950747Z fi 2024-04-03T16:17:05.0951109Z  2024-04-03T16:17:05.0951515Z if [[ -z "${MERGE_BASE}" ]]; then 2024-04-03T16:17:05.0952214Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-04-03T16:17:05.0953040Z  2024-04-03T16:17:05.0953960Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-04-03T16:17:05.0955342Z  exit 0 2024-04-03T16:17:05.0955747Z fi 2024-04-03T16:17:05.0956101Z  2024-04-03T16:17:05.0956705Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-04-03T16:17:05.0958111Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-04-03T16:17:05.0959261Z  exit 1 2024-04-03T16:17:05.0959655Z fi 2024-04-03T16:17:05.0960026Z  2024-04-03T16:17:05.0960698Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-04-03T16:17:05.0962046Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-04-03T16:17:05.0963287Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-04-03T16:17:05.0964676Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-04-03T16:17:05.0966246Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-04-03T16:17:05.0967162Z fi 2024-04-03T16:17:05.0967524Z  2024-04-03T16:17:05.0968144Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-04-03T16:17:05.0977129Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:17:05.0977826Z env: 2024-04-03T16:17:05.0978183Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:17:05.0978668Z DOCKER_BUILD_DIR: .ci/docker 2024-04-03T16:17:05.0979298Z BASE_REVISION: 1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:17:05.0980919Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:17:05.0982524Z DOCKER_TAG: 16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:17:05.0983347Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:17:05.0984062Z ##[endgroup] 2024-04-03T16:17:05.1012099Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:17:05.1013002Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:17:05.1015176Z + aws ecr get-login-password --region us-east-1 2024-04-03T16:17:05.1016463Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:17:05.6385149Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-04-03T16:17:05.6386352Z Configure a credential helper to remove this warning. See 2024-04-03T16:17:05.6387562Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-04-03T16:17:05.6388350Z 2024-04-03T16:17:05.6388518Z Login Succeeded 2024-04-03T16:17:05.6402860Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:17:05.8841408Z { 2024-04-03T16:17:05.8841821Z "schemaVersion": 2, 2024-04-03T16:17:05.8842582Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-04-03T16:17:05.8843447Z "config": { 2024-04-03T16:17:05.8844111Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-04-03T16:17:05.8844881Z "size": 44693, 2024-04-03T16:17:05.8845690Z "digest": "sha256:f0aa5113e491b8f8ef3b51fc066bce798422ecc2bdeb4be54a8a4efa6914052e" 2024-04-03T16:17:05.8846607Z }, 2024-04-03T16:17:05.8846949Z "layers": [ 2024-04-03T16:17:05.8847817Z { 2024-04-03T16:17:05.8848575Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8850303Z "size": 28580681, 2024-04-03T16:17:05.8851493Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-04-03T16:17:05.8852405Z }, 2024-04-03T16:17:05.8853024Z { 2024-04-03T16:17:05.8853653Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8854794Z "size": 7943353, 2024-04-03T16:17:05.8855701Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-04-03T16:17:05.8856868Z }, 2024-04-03T16:17:05.8857555Z { 2024-04-03T16:17:05.8858811Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8860360Z "size": 56226389, 2024-04-03T16:17:05.8861716Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-04-03T16:17:05.8863228Z }, 2024-04-03T16:17:05.8863789Z { 2024-04-03T16:17:05.8865132Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8866170Z "size": 187, 2024-04-03T16:17:05.8866924Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-04-03T16:17:05.8867814Z }, 2024-04-03T16:17:05.8868159Z { 2024-04-03T16:17:05.8868770Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8869575Z "size": 6885, 2024-04-03T16:17:05.8870356Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-04-03T16:17:05.8871239Z }, 2024-04-03T16:17:05.8871577Z { 2024-04-03T16:17:05.8872192Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8873189Z "size": 1383662475, 2024-04-03T16:17:05.8874002Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-04-03T16:17:05.8875299Z }, 2024-04-03T16:17:05.8875627Z { 2024-04-03T16:17:05.8876246Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8877050Z "size": 62282, 2024-04-03T16:17:05.8877800Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-04-03T16:17:05.8878683Z }, 2024-04-03T16:17:05.8879024Z { 2024-04-03T16:17:05.8879618Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8880437Z "size": 1684, 2024-04-03T16:17:05.8881231Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-04-03T16:17:05.8882116Z }, 2024-04-03T16:17:05.8882462Z { 2024-04-03T16:17:05.8883077Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8883860Z "size": 1522, 2024-04-03T16:17:05.8884648Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-04-03T16:17:05.8885542Z }, 2024-04-03T16:17:05.8885863Z { 2024-04-03T16:17:05.8886481Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8887289Z "size": 2445884909, 2024-04-03T16:17:05.8888061Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-04-03T16:17:05.8889107Z + exit 0 2024-04-03T16:17:05.8889473Z }, 2024-04-03T16:17:05.8889797Z { 2024-04-03T16:17:05.8890420Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8891230Z "size": 85882, 2024-04-03T16:17:05.8891989Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-04-03T16:17:05.8892872Z }, 2024-04-03T16:17:05.8893214Z { 2024-04-03T16:17:05.8893811Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8894613Z "size": 1827, 2024-04-03T16:17:05.8895424Z "digest": "sha256:020f47d0fa1be3a2df8dd2e863841d351ab72d00c7df4a4a16879ceea1239f4b" 2024-04-03T16:17:05.8896316Z }, 2024-04-03T16:17:05.8896658Z { 2024-04-03T16:17:05.8897278Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8898061Z "size": 654809190, 2024-04-03T16:17:05.8898887Z "digest": "sha256:8b29859251bec1f67480cc3935ceb13aeb54305ec53bcb4ef1d4977fcd62e2b0" 2024-04-03T16:17:05.8899785Z }, 2024-04-03T16:17:05.8900108Z { 2024-04-03T16:17:05.8900727Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8901698Z "size": 544, 2024-04-03T16:17:05.8902435Z "digest": "sha256:0e1508d7c9ab1501f0c741f4729a53eb3758856d8761c5258f732506d73c594e" 2024-04-03T16:17:05.8903307Z }, 2024-04-03T16:17:05.8903654Z { 2024-04-03T16:17:05.8904249Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8905049Z "size": 1282, 2024-04-03T16:17:05.8905841Z "digest": "sha256:89f208ed177f5a57c0375b5e3eabab8772559309b532c759b1ce8ea6cf1b3e4b" 2024-04-03T16:17:05.8906718Z }, 2024-04-03T16:17:05.8907060Z { 2024-04-03T16:17:05.8907680Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8908460Z "size": 483, 2024-04-03T16:17:05.8909250Z "digest": "sha256:ddecf600b7ac2f21cd805972b7cb9b1305ec1846970aca6b50db2c833e5b38cb" 2024-04-03T16:17:05.8910161Z }, 2024-04-03T16:17:05.8910485Z { 2024-04-03T16:17:05.8911104Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8911912Z "size": 91705074, 2024-04-03T16:17:05.8912718Z "digest": "sha256:7965ac4a7eb7caf381dde0f1bb10190588c124b84a8b1e9c77adf5143e9dc4f9" 2024-04-03T16:17:05.8913624Z }, 2024-04-03T16:17:05.8913968Z { 2024-04-03T16:17:05.8914902Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8915701Z "size": 3255, 2024-04-03T16:17:05.8916627Z "digest": "sha256:6b9458cb1b822d8bda5cc039459fec8b5d9e46187880eddb9e529319467f189b" 2024-04-03T16:17:05.8917531Z }, 2024-04-03T16:17:05.8917873Z { 2024-04-03T16:17:05.8918488Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8919265Z "size": 1904, 2024-04-03T16:17:05.8920057Z "digest": "sha256:a40a390f581ad70eca936ae2fe3aeadb7e19642cd6211232d77be76ba9fa7dd2" 2024-04-03T16:17:05.8920966Z }, 2024-04-03T16:17:05.8921287Z { 2024-04-03T16:17:05.8921900Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8922698Z "size": 699, 2024-04-03T16:17:05.8923476Z "digest": "sha256:01d4a84baddbd86b6da32efdf2e97a1088a27be6c25e024af3e1bc20f1db320e" 2024-04-03T16:17:05.8924409Z }, 2024-04-03T16:17:05.8924752Z { 2024-04-03T16:17:05.8925346Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8926152Z "size": 2803779640, 2024-04-03T16:17:05.8926971Z "digest": "sha256:db8dcb001b22779a63f46677dafc9340d330e77f96e6dcb21cd1b1210a88c0ae" 2024-04-03T16:17:05.8927856Z }, 2024-04-03T16:17:05.8928200Z { 2024-04-03T16:17:05.8928812Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8929592Z "size": 380, 2024-04-03T16:17:05.8930356Z "digest": "sha256:3eaa6199e98399217ecd8fe5727b666c8273744f0518bc3e5b1ac140ce3d123c" 2024-04-03T16:17:05.8931241Z }, 2024-04-03T16:17:05.8931563Z { 2024-04-03T16:17:05.8932181Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8932986Z "size": 11856, 2024-04-03T16:17:05.8933751Z "digest": "sha256:dd253632212f47f8e65518f3f9c4a80baeddd5f04f7bf0eb3840a976c93b5276" 2024-04-03T16:17:05.8934655Z }, 2024-04-03T16:17:05.8934996Z { 2024-04-03T16:17:05.8935589Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8936382Z "size": 802, 2024-04-03T16:17:05.8937132Z "digest": "sha256:5e6597056347c056681d9213c22a37f96446cb7b0d9b6838b5e6101ac98c7e8a" 2024-04-03T16:17:05.8937992Z }, 2024-04-03T16:17:05.8938314Z { 2024-04-03T16:17:05.8938990Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8939773Z "size": 106, 2024-04-03T16:17:05.8940548Z "digest": "sha256:858ba61782a0452f8aa8a1cef9b4fd1f301e540347dff3ed187bf69086efb8cc" 2024-04-03T16:17:05.8941443Z }, 2024-04-03T16:17:05.8941764Z { 2024-04-03T16:17:05.8942376Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8943177Z "size": 503, 2024-04-03T16:17:05.8943931Z "digest": "sha256:9cb381d49846ff5212226715691aa2c6abc9987a95b92335e81a6d03d9c4f944" 2024-04-03T16:17:05.8944972Z }, 2024-04-03T16:17:05.8945315Z { 2024-04-03T16:17:05.8945908Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8946708Z "size": 121477305, 2024-04-03T16:17:05.8947505Z "digest": "sha256:610c26b5e9a03476b891eae042a708b3462a40bc118e1e5e472f936448896531" 2024-04-03T16:17:05.8948382Z }, 2024-04-03T16:17:05.8948707Z { 2024-04-03T16:17:05.8949333Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8950133Z "size": 109, 2024-04-03T16:17:05.8950876Z "digest": "sha256:03a7067427cf481c50c8195f331a42f0ec54f72e472ea1fdf170ad664c44530a" 2024-04-03T16:17:05.8951751Z }, 2024-04-03T16:17:05.8952093Z { 2024-04-03T16:17:05.8952690Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8953485Z "size": 557, 2024-04-03T16:17:05.8954576Z "digest": "sha256:b4f17b88d23ef498cfc2839ad0358588aaca3004f277c84f3ec7d68336482c1e" 2024-04-03T16:17:05.8955464Z }, 2024-04-03T16:17:05.8955820Z { 2024-04-03T16:17:05.8956434Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8957220Z "size": 1175558, 2024-04-03T16:17:05.8958017Z "digest": "sha256:bc6bfcbe5530b217c0792229e143c80718d420f57f5e5db203ecf7be083b4d29" 2024-04-03T16:17:05.8958938Z }, 2024-04-03T16:17:05.8959259Z { 2024-04-03T16:17:05.8960010Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8960831Z "size": 103, 2024-04-03T16:17:05.8961810Z "digest": "sha256:df877d3106b553016cfe184088c5aa43dc0e564cb21a18e430d08bbdf8277f3d" 2024-04-03T16:17:05.8962696Z }, 2024-04-03T16:17:05.8963035Z { 2024-04-03T16:17:05.8963631Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8964434Z "size": 1489, 2024-04-03T16:17:05.8965230Z "digest": "sha256:0fbc7e84a570fa00c6b12df3ffc46a2b4ff2786d9e5564886359fe90b556556d" 2024-04-03T16:17:05.8966117Z }, 2024-04-03T16:17:05.8966462Z { 2024-04-03T16:17:05.8967088Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8967875Z "size": 442747152, 2024-04-03T16:17:05.8968660Z "digest": "sha256:1570829375e6e129fa627aa95c6619f8006d73f425c85d72f4595e55a0f74e0f" 2024-04-03T16:17:05.8969523Z }, 2024-04-03T16:17:05.8969846Z { 2024-04-03T16:17:05.8970464Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8971276Z "size": 161, 2024-04-03T16:17:05.8972027Z "digest": "sha256:6281fee72d3f38f69e3d04981823f93b775ef06fd399fc2308e9d5c951ec7bc1" 2024-04-03T16:17:05.8972911Z }, 2024-04-03T16:17:05.8973254Z { 2024-04-03T16:17:05.8973855Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8974655Z "size": 564, 2024-04-03T16:17:05.8975423Z "digest": "sha256:d01dc0055b877335000d5b68ac1c5e71147255853e3de4edfa032d27feff6452" 2024-04-03T16:17:05.8976288Z }, 2024-04-03T16:17:05.8976632Z { 2024-04-03T16:17:05.8977249Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8978048Z "size": 33199039, 2024-04-03T16:17:05.8978857Z "digest": "sha256:2abb072c55408b8bf96c5a0a24c729cb31029e0d42ef0192a2eba0cab079933c" 2024-04-03T16:17:05.8979756Z }, 2024-04-03T16:17:05.8980084Z { 2024-04-03T16:17:05.8980700Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8981500Z "size": 104, 2024-04-03T16:17:05.8982254Z "digest": "sha256:1323b806817860cf618c3a24ee430b0ccdd26464c11bcc46f717b6cbf241820c" 2024-04-03T16:17:05.8983140Z }, 2024-04-03T16:17:05.8983484Z { 2024-04-03T16:17:05.8984077Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8984880Z "size": 424, 2024-04-03T16:17:05.8985676Z "digest": "sha256:473ae4beebc0bc7146babdb210cabec932238e00b4a3f74f9f0bf5d7bf577593" 2024-04-03T16:17:05.8986572Z }, 2024-04-03T16:17:05.8986912Z { 2024-04-03T16:17:05.8987523Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8988449Z "size": 20262067, 2024-04-03T16:17:05.8989257Z "digest": "sha256:289a48eb62b5823f2c7d86b1875b0edd766b78c9e0edd7e1d5f54e94f004f0dc" 2024-04-03T16:17:05.8990149Z }, 2024-04-03T16:17:05.8990470Z { 2024-04-03T16:17:05.8991085Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8991883Z "size": 439, 2024-04-03T16:17:05.8992658Z "digest": "sha256:6ac1d762661f0e7800aadeafce3ba5f8d3337ad5c27ff47253ac76bbb7895841" 2024-04-03T16:17:05.8993567Z }, 2024-04-03T16:17:05.8993906Z { 2024-04-03T16:17:05.8994848Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8995656Z "size": 699, 2024-04-03T16:17:05.8996458Z "digest": "sha256:01d4a84baddbd86b6da32efdf2e97a1088a27be6c25e024af3e1bc20f1db320e" 2024-04-03T16:17:05.8997360Z }, 2024-04-03T16:17:05.8997702Z { 2024-04-03T16:17:05.8998319Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.8999099Z "size": 142, 2024-04-03T16:17:05.8999875Z "digest": "sha256:796d028fd584b098f569e5427fcf37e2248f308b02af6185104c4faeeef48b05" 2024-04-03T16:17:05.9000753Z }, 2024-04-03T16:17:05.9001074Z { 2024-04-03T16:17:05.9001687Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9002486Z "size": 136, 2024-04-03T16:17:05.9003383Z "digest": "sha256:a3890e7012d35f6e60a4f6bdd11bf6bba9b20824a9bc7e4795e3f620f8eecdd5" 2024-04-03T16:17:05.9004300Z }, 2024-04-03T16:17:05.9004639Z { 2024-04-03T16:17:05.9005232Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9006026Z "size": 32, 2024-04-03T16:17:05.9006803Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:17:05.9007688Z }, 2024-04-03T16:17:05.9008031Z { 2024-04-03T16:17:05.9008642Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9009421Z "size": 188, 2024-04-03T16:17:05.9010194Z "digest": "sha256:b3d241c09291e0fe13d64493f03dbee53b27579eaa163e4d569d67ed310775f0" 2024-04-03T16:17:05.9011095Z }, 2024-04-03T16:17:05.9011418Z { 2024-04-03T16:17:05.9012031Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9012828Z "size": 564, 2024-04-03T16:17:05.9013556Z "digest": "sha256:c584774f42519400f1d08734f82c70984be83c66222c2bcf7b6d6703895f5c79" 2024-04-03T16:17:05.9014428Z }, 2024-04-03T16:17:05.9014777Z { 2024-04-03T16:17:05.9015370Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9016169Z "size": 43165822, 2024-04-03T16:17:05.9016984Z "digest": "sha256:e4b3bc5cda0494786fe54c1838bc55bce19728edf9a927c05bd5884dcbd6e205" 2024-04-03T16:17:05.9017868Z }, 2024-04-03T16:17:05.9018209Z { 2024-04-03T16:17:05.9018817Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9019594Z "size": 106, 2024-04-03T16:17:05.9020360Z "digest": "sha256:8b7274bb07ecf3280dfc69b15646503e9f67d0cf05be86b04469183960f72e64" 2024-04-03T16:17:05.9021254Z }, 2024-04-03T16:17:05.9021577Z { 2024-04-03T16:17:05.9022191Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9023029Z "size": 1192, 2024-04-03T16:17:05.9023759Z "digest": "sha256:f2934ba32628883717478385639d4c5f0403a09363b8cf2bb73b328e3b8cc042" 2024-04-03T16:17:05.9024626Z }, 2024-04-03T16:17:05.9024968Z { 2024-04-03T16:17:05.9025571Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9026371Z "size": 699, 2024-04-03T16:17:05.9027166Z "digest": "sha256:01d4a84baddbd86b6da32efdf2e97a1088a27be6c25e024af3e1bc20f1db320e" 2024-04-03T16:17:05.9028062Z }, 2024-04-03T16:17:05.9028403Z { 2024-04-03T16:17:05.9029019Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9029796Z "size": 137, 2024-04-03T16:17:05.9030572Z "digest": "sha256:d1c6f5fd2ce358717d4f731eb8c70e2757ed206be14c1f9999718a81f29d2874" 2024-04-03T16:17:05.9031624Z }, 2024-04-03T16:17:05.9031946Z { 2024-04-03T16:17:05.9032563Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9033362Z "size": 119, 2024-04-03T16:17:05.9034119Z "digest": "sha256:9c3c29d675a6f3566456d0b187f18d1a5deac38949fea55e068dbb7bc69d8bb4" 2024-04-03T16:17:05.9035344Z }, 2024-04-03T16:17:05.9035691Z { 2024-04-03T16:17:05.9036298Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9037102Z "size": 1704547419, 2024-04-03T16:17:05.9037891Z "digest": "sha256:e400461a29326092b6477a61196ef84989ebf7c7aee931714fd37259cc862d8b" 2024-04-03T16:17:05.9038783Z }, 2024-04-03T16:17:05.9039127Z { 2024-04-03T16:17:05.9039748Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9040524Z "size": 172, 2024-04-03T16:17:05.9041304Z "digest": "sha256:f3b4b7bb5d910fb4f87f578cc5c2180774bdff4c30831c40bc5425b29658682c" 2024-04-03T16:17:05.9042203Z }, 2024-04-03T16:17:05.9042535Z { 2024-04-03T16:17:05.9043155Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9043960Z "size": 1840, 2024-04-03T16:17:05.9044736Z "digest": "sha256:d39766ef1e6b8b4b6e2ae3d7d9f23f104b3c0309eefbd8bb4e5d3e362937299a" 2024-04-03T16:17:05.9045638Z }, 2024-04-03T16:17:05.9045983Z { 2024-04-03T16:17:05.9046724Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9047552Z "size": 7529780, 2024-04-03T16:17:05.9048353Z "digest": "sha256:8ce169be4b1d2efb4c8dbe9188699e646f24d62988505b7bc263e4605d20ff32" 2024-04-03T16:17:05.9049230Z }, 2024-04-03T16:17:05.9049574Z { 2024-04-03T16:17:05.9050189Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9050972Z "size": 164, 2024-04-03T16:17:05.9051746Z "digest": "sha256:fe2cd77be091e130f7e397bb03a127255f90a88edb85e9b91957619adac1d7b7" 2024-04-03T16:17:05.9052642Z }, 2024-04-03T16:17:05.9052967Z { 2024-04-03T16:17:05.9053592Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9054388Z "size": 7938, 2024-04-03T16:17:05.9055175Z "digest": "sha256:2a5bd85dd325ef3ddf1e8d9bd46e473bae5adf46aa7df529b7d898b7c4a42add" 2024-04-03T16:17:05.9056093Z }, 2024-04-03T16:17:05.9056438Z { 2024-04-03T16:17:05.9057038Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9057848Z "size": 8070, 2024-04-03T16:17:05.9058639Z "digest": "sha256:d1190269f41e7ab64b2eadaeba8697af4731d9667e9a95ac49d95c717878351f" 2024-04-03T16:17:05.9059516Z }, 2024-04-03T16:17:05.9059860Z { 2024-04-03T16:17:05.9060479Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9061259Z "size": 302, 2024-04-03T16:17:05.9062045Z "digest": "sha256:e8752c122b392a996acd66fc47ae9c18fe3f4960cbeee8bb1eeea5ab3dfbd9c2" 2024-04-03T16:17:05.9062948Z }, 2024-04-03T16:17:05.9063272Z { 2024-04-03T16:17:05.9064077Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9064882Z "size": 7620337, 2024-04-03T16:17:05.9065663Z "digest": "sha256:1eaea607f89d48ad9275ec68ec91f10e1cb3dda60bb0189f2793cef6780ad02f" 2024-04-03T16:17:05.9066563Z }, 2024-04-03T16:17:05.9066902Z { 2024-04-03T16:17:05.9067494Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9068291Z "size": 108, 2024-04-03T16:17:05.9069058Z "digest": "sha256:9ef6078991a0e9655c1656e4b3f24cdf5377a1c1ad41e4418b4a5b912856183f" 2024-04-03T16:17:05.9069916Z }, 2024-04-03T16:17:05.9070257Z { 2024-04-03T16:17:05.9070869Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9071652Z "size": 54145458, 2024-04-03T16:17:05.9072465Z "digest": "sha256:c0dab7a64f9be557be292ba18570e24ade19f386e38b36b5dfe3a5a844ef38eb" 2024-04-03T16:17:05.9073375Z }, 2024-04-03T16:17:05.9073696Z { 2024-04-03T16:17:05.9074627Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9075584Z "size": 498, 2024-04-03T16:17:05.9076330Z "digest": "sha256:20e7673e255ed89c9eca5e8f08025a96bf06263edb006590d7ff6137cf90730f" 2024-04-03T16:17:05.9077215Z }, 2024-04-03T16:17:05.9077554Z { 2024-04-03T16:17:05.9078151Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9078947Z "size": 1498859701, 2024-04-03T16:17:05.9079755Z "digest": "sha256:865db895296b5ec9e81d86e205660c33454f259b5dd7ed88f9402cbd87699077" 2024-04-03T16:17:05.9080637Z }, 2024-04-03T16:17:05.9080958Z { 2024-04-03T16:17:05.9081570Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9082350Z "size": 106, 2024-04-03T16:17:05.9083129Z "digest": "sha256:effd5b6e0122c9c96f918d64d4e1b49efb8b08c08324eea3054750d8d2b37652" 2024-04-03T16:17:05.9084024Z }, 2024-04-03T16:17:05.9084345Z { 2024-04-03T16:17:05.9084954Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9085748Z "size": 444, 2024-04-03T16:17:05.9086511Z "digest": "sha256:6150e99963233ce451f0f0e5a16b96087b4924cdbb89bb7701503081c6ca5027" 2024-04-03T16:17:05.9087367Z }, 2024-04-03T16:17:05.9087708Z { 2024-04-03T16:17:05.9088303Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9089099Z "size": 36532201, 2024-04-03T16:17:05.9090033Z "digest": "sha256:7ac5ac4b15768fbc183814207bf88e8b69cf1655e4651c06267591dad5d842d5" 2024-04-03T16:17:05.9090936Z }, 2024-04-03T16:17:05.9091259Z { 2024-04-03T16:17:05.9091872Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9092668Z "size": 111, 2024-04-03T16:17:05.9093417Z "digest": "sha256:f31a22b80d01bb61f572065e872c8276ef92389cb05b1f1b6853fda52ce462a2" 2024-04-03T16:17:05.9094299Z }, 2024-04-03T16:17:05.9094639Z { 2024-04-03T16:17:05.9095234Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9096031Z "size": 32, 2024-04-03T16:17:05.9096805Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:17:05.9097698Z }, 2024-04-03T16:17:05.9098039Z { 2024-04-03T16:17:05.9098650Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9099428Z "size": 32, 2024-04-03T16:17:05.9100205Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:17:05.9101097Z }, 2024-04-03T16:17:05.9101426Z { 2024-04-03T16:17:05.9102044Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:17:05.9102841Z "size": 32, 2024-04-03T16:17:05.9103599Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:17:05.9104496Z } 2024-04-03T16:17:05.9104836Z ] 2024-04-03T16:17:05.9105154Z } 2024-04-03T16:17:05.9263439Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-04-03T16:17:05.9264183Z tag=${ECR_DOCKER_IMAGE##*/} 2024-04-03T16:17:05.9264968Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-04-03T16:17:05.9275448Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:17:05.9276177Z env: 2024-04-03T16:17:05.9276567Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:17:05.9278040Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:17:05.9279587Z ##[endgroup] 2024-04-03T16:17:05.9311256Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9-16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:17:05.9397394Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-04-03T16:17:05.9398240Z with: 2024-04-03T16:17:05.9399595Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:17:05.9401332Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:17:05.9402237Z env: 2024-04-03T16:17:05.9402621Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:17:05.9403113Z ##[endgroup] 2024-04-03T16:17:05.9429454Z ##[group]Run set -x 2024-04-03T16:17:05.9429932Z set -x 2024-04-03T16:17:05.9430359Z set +e 2024-04-03T16:17:05.9430764Z  2024-04-03T16:17:05.9431131Z login() { 2024-04-03T16:17:05.9432085Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-04-03T16:17:05.9433128Z } 2024-04-03T16:17:05.9433494Z  2024-04-03T16:17:05.9433938Z retry () { 2024-04-03T16:17:05.9434734Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-04-03T16:17:05.9435376Z } 2024-04-03T16:17:05.9435759Z  2024-04-03T16:17:05.9436179Z retry login "${DOCKER_REGISTRY}" 2024-04-03T16:17:05.9436768Z  2024-04-03T16:17:05.9437150Z set -e 2024-04-03T16:17:05.9437809Z # ignore output since only exit code is used for conditional 2024-04-03T16:17:05.9438884Z # only pull docker image if it's not available locally 2024-04-03T16:17:05.9439991Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-04-03T16:17:05.9440990Z  retry docker pull "${DOCKER_IMAGE}" 2024-04-03T16:17:05.9441598Z fi 2024-04-03T16:17:05.9450566Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:17:05.9451290Z env: 2024-04-03T16:17:05.9451675Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:17:05.9453123Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:17:05.9454832Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:17:05.9455577Z ##[endgroup] 2024-04-03T16:17:05.9483005Z + set +e 2024-04-03T16:17:05.9483729Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:17:05.9484650Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:17:05.9487007Z + aws ecr get-login-password --region us-east-1 2024-04-03T16:17:05.9488506Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:17:06.4882165Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-04-03T16:17:06.4883612Z Configure a credential helper to remove this warning. See 2024-04-03T16:17:06.4885314Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-04-03T16:17:06.4886088Z 2024-04-03T16:17:06.4886302Z Login Succeeded 2024-04-03T16:17:06.4898058Z + set -e 2024-04-03T16:17:06.4899748Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:17:06.5140037Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:17:06.5142784Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:17:06.7457292Z 16884143090f148aa56809ceba26eeb699b5dc49: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9 2024-04-03T16:17:06.7458468Z 7a2c55901189: Pulling fs layer 2024-04-03T16:17:06.7459015Z 09d415c238d7: Pulling fs layer 2024-04-03T16:17:06.7459557Z 9fe6e2e61518: Pulling fs layer 2024-04-03T16:17:06.7460071Z 41f16248e682: Pulling fs layer 2024-04-03T16:17:06.7460839Z 95d7b7817039: Pulling fs layer 2024-04-03T16:17:06.7461417Z 8f6c90485347: Pulling fs layer 2024-04-03T16:17:06.7461925Z ab17245097e4: Pulling fs layer 2024-04-03T16:17:06.7462463Z dfecd7e9912b: Pulling fs layer 2024-04-03T16:17:06.7463003Z 464a8f745445: Pulling fs layer 2024-04-03T16:17:06.7463513Z d67c111fa588: Pulling fs layer 2024-04-03T16:17:06.7464048Z 0723abde19c0: Pulling fs layer 2024-04-03T16:17:06.7464583Z 020f47d0fa1b: Pulling fs layer 2024-04-03T16:17:06.7465268Z 8b29859251be: Pulling fs layer 2024-04-03T16:17:06.7465805Z 0e1508d7c9ab: Pulling fs layer 2024-04-03T16:17:06.7466341Z 89f208ed177f: Pulling fs layer 2024-04-03T16:17:06.7466860Z ddecf600b7ac: Pulling fs layer 2024-04-03T16:17:06.7467437Z 7965ac4a7eb7: Pulling fs layer 2024-04-03T16:17:06.7468367Z 6b9458cb1b82: Pulling fs layer 2024-04-03T16:17:06.7469247Z a40a390f581a: Pulling fs layer 2024-04-03T16:17:06.7470346Z 01d4a84baddb: Pulling fs layer 2024-04-03T16:17:06.7471539Z db8dcb001b22: Pulling fs layer 2024-04-03T16:17:06.7472493Z 3eaa6199e983: Pulling fs layer 2024-04-03T16:17:06.7473013Z dd253632212f: Pulling fs layer 2024-04-03T16:17:06.7473537Z 5e6597056347: Pulling fs layer 2024-04-03T16:17:06.7474049Z 95d7b7817039: Waiting 2024-04-03T16:17:06.7475067Z 8f6c90485347: Waiting 2024-04-03T16:17:06.7475543Z 858ba61782a0: Pulling fs layer 2024-04-03T16:17:06.7476332Z 9cb381d49846: Pulling fs layer 2024-04-03T16:17:06.7476851Z 610c26b5e9a0: Pulling fs layer 2024-04-03T16:17:06.7477399Z 03a7067427cf: Pulling fs layer 2024-04-03T16:17:06.7477905Z 41f16248e682: Waiting 2024-04-03T16:17:06.7478359Z b4f17b88d23e: Pulling fs layer 2024-04-03T16:17:06.7479152Z bc6bfcbe5530: Pulling fs layer 2024-04-03T16:17:06.7480004Z 5e6597056347: Waiting 2024-04-03T16:17:06.7480456Z df877d3106b5: Pulling fs layer 2024-04-03T16:17:06.7481258Z 0e1508d7c9ab: Waiting 2024-04-03T16:17:06.7481745Z 0fbc7e84a570: Pulling fs layer 2024-04-03T16:17:06.7482242Z ab17245097e4: Waiting 2024-04-03T16:17:06.7482869Z dd253632212f: Waiting 2024-04-03T16:17:06.7483347Z 1570829375e6: Pulling fs layer 2024-04-03T16:17:06.7483860Z 6281fee72d3f: Pulling fs layer 2024-04-03T16:17:06.7484397Z d01dc0055b87: Pulling fs layer 2024-04-03T16:17:06.7484929Z 2abb072c5540: Pulling fs layer 2024-04-03T16:17:06.7485418Z d67c111fa588: Waiting 2024-04-03T16:17:06.7485863Z 89f208ed177f: Waiting 2024-04-03T16:17:06.7486310Z 1323b8068178: Pulling fs layer 2024-04-03T16:17:06.7486839Z 473ae4beebc0: Pulling fs layer 2024-04-03T16:17:06.7487383Z 289a48eb62b5: Pulling fs layer 2024-04-03T16:17:06.7487896Z 6ac1d762661f: Pulling fs layer 2024-04-03T16:17:06.7488404Z ddecf600b7ac: Waiting 2024-04-03T16:17:06.7488852Z 8b29859251be: Waiting 2024-04-03T16:17:06.7489273Z 9cb381d49846: Waiting 2024-04-03T16:17:06.7489741Z 796d028fd584: Pulling fs layer 2024-04-03T16:17:06.7490268Z a3890e7012d3: Pulling fs layer 2024-04-03T16:17:06.7490782Z 4f4fb700ef54: Pulling fs layer 2024-04-03T16:17:06.7491290Z b4f17b88d23e: Waiting 2024-04-03T16:17:06.7491740Z 610c26b5e9a0: Waiting 2024-04-03T16:17:06.7492175Z bc6bfcbe5530: Waiting 2024-04-03T16:17:06.7492621Z df877d3106b5: Waiting 2024-04-03T16:17:06.7493085Z b3d241c09291: Pulling fs layer 2024-04-03T16:17:06.7493579Z 0fbc7e84a570: Waiting 2024-04-03T16:17:06.7494022Z d01dc0055b87: Waiting 2024-04-03T16:17:06.7494485Z c584774f4251: Pulling fs layer 2024-04-03T16:17:06.7494999Z e4b3bc5cda04: Pulling fs layer 2024-04-03T16:17:06.7495535Z 8b7274bb07ec: Pulling fs layer 2024-04-03T16:17:06.7496048Z 6281fee72d3f: Waiting 2024-04-03T16:17:06.7496505Z f2934ba32628: Pulling fs layer 2024-04-03T16:17:06.7497011Z 3eaa6199e983: Waiting 2024-04-03T16:17:06.7497482Z d1c6f5fd2ce3: Pulling fs layer 2024-04-03T16:17:06.7497995Z 9c3c29d675a6: Pulling fs layer 2024-04-03T16:17:06.7498505Z 0723abde19c0: Waiting 2024-04-03T16:17:06.7498950Z 289a48eb62b5: Waiting 2024-04-03T16:17:06.7499379Z 020f47d0fa1b: Waiting 2024-04-03T16:17:06.7499818Z a3890e7012d3: Waiting 2024-04-03T16:17:06.7500263Z 6b9458cb1b82: Waiting 2024-04-03T16:17:06.7500740Z e400461a2932: Pulling fs layer 2024-04-03T16:17:06.7501490Z f3b4b7bb5d91: Pulling fs layer 2024-04-03T16:17:06.7502068Z d39766ef1e6b: Pulling fs layer 2024-04-03T16:17:06.7502559Z c584774f4251: Waiting 2024-04-03T16:17:06.7503022Z 8ce169be4b1d: Pulling fs layer 2024-04-03T16:17:06.7503529Z 1323b8068178: Waiting 2024-04-03T16:17:06.7503953Z e4b3bc5cda04: Waiting 2024-04-03T16:17:06.7504424Z fe2cd77be091: Pulling fs layer 2024-04-03T16:17:06.7504955Z 2a5bd85dd325: Pulling fs layer 2024-04-03T16:17:06.7505584Z 858ba61782a0: Waiting 2024-04-03T16:17:06.7506022Z b3d241c09291: Waiting 2024-04-03T16:17:06.7506487Z d1190269f41e: Pulling fs layer 2024-04-03T16:17:06.7506969Z 1570829375e6: Waiting 2024-04-03T16:17:06.7507405Z 796d028fd584: Waiting 2024-04-03T16:17:06.7507866Z e8752c122b39: Pulling fs layer 2024-04-03T16:17:06.7508383Z 1eaea607f89d: Pulling fs layer 2024-04-03T16:17:06.7508913Z 9ef6078991a0: Pulling fs layer 2024-04-03T16:17:06.7509435Z d1c6f5fd2ce3: Waiting 2024-04-03T16:17:06.7509887Z c0dab7a64f9b: Pulling fs layer 2024-04-03T16:17:06.7510424Z 20e7673e255e: Pulling fs layer 2024-04-03T16:17:06.7510959Z 865db895296b: Pulling fs layer 2024-04-03T16:17:06.7511468Z effd5b6e0122: Pulling fs layer 2024-04-03T16:17:06.7511996Z 6150e9996323: Pulling fs layer 2024-04-03T16:17:06.7512519Z 7ac5ac4b1576: Pulling fs layer 2024-04-03T16:17:06.7513011Z 01d4a84baddb: Waiting 2024-04-03T16:17:06.7513459Z 9c3c29d675a6: Waiting 2024-04-03T16:17:06.7513923Z f31a22b80d01: Pulling fs layer 2024-04-03T16:17:06.7514864Z f3b4b7bb5d91: Waiting 2024-04-03T16:17:06.7515320Z d39766ef1e6b: Waiting 2024-04-03T16:17:06.7515766Z 9ef6078991a0: Waiting 2024-04-03T16:17:06.7516187Z 865db895296b: Waiting 2024-04-03T16:17:06.7516631Z effd5b6e0122: Waiting 2024-04-03T16:17:06.7517081Z 7965ac4a7eb7: Waiting 2024-04-03T16:17:06.7517506Z 1eaea607f89d: Waiting 2024-04-03T16:17:06.7517950Z f31a22b80d01: Waiting 2024-04-03T16:17:06.7518394Z 8ce169be4b1d: Waiting 2024-04-03T16:17:06.7518821Z 7ac5ac4b1576: Waiting 2024-04-03T16:17:06.7519266Z 473ae4beebc0: Waiting 2024-04-03T16:17:06.7519718Z 20e7673e255e: Waiting 2024-04-03T16:17:06.7520141Z f2934ba32628: Waiting 2024-04-03T16:17:06.7520580Z 2a5bd85dd325: Waiting 2024-04-03T16:17:06.7521017Z e400461a2932: Waiting 2024-04-03T16:17:06.7521433Z d1190269f41e: Waiting 2024-04-03T16:17:06.7521861Z 6150e9996323: Waiting 2024-04-03T16:17:06.7522300Z c0dab7a64f9b: Waiting 2024-04-03T16:17:06.7522725Z 6ac1d762661f: Waiting 2024-04-03T16:17:06.9133611Z 09d415c238d7: Verifying Checksum 2024-04-03T16:17:06.9134407Z 09d415c238d7: Download complete 2024-04-03T16:17:07.0442841Z 41f16248e682: Verifying Checksum 2024-04-03T16:17:07.0443578Z 41f16248e682: Download complete 2024-04-03T16:17:07.1103334Z 7a2c55901189: Download complete 2024-04-03T16:17:07.1342547Z 95d7b7817039: Verifying Checksum 2024-04-03T16:17:07.1343792Z 95d7b7817039: Download complete 2024-04-03T16:17:07.2213898Z ab17245097e4: Download complete 2024-04-03T16:17:07.3079626Z dfecd7e9912b: Verifying Checksum 2024-04-03T16:17:07.3080700Z dfecd7e9912b: Download complete 2024-04-03T16:17:07.3896737Z 464a8f745445: Verifying Checksum 2024-04-03T16:17:07.3897611Z 464a8f745445: Download complete 2024-04-03T16:17:07.4155936Z 9fe6e2e61518: Verifying Checksum 2024-04-03T16:17:07.4156630Z 9fe6e2e61518: Download complete 2024-04-03T16:17:07.4976900Z 0723abde19c0: Verifying Checksum 2024-04-03T16:17:07.4977600Z 0723abde19c0: Download complete 2024-04-03T16:17:07.5772043Z 020f47d0fa1b: Download complete 2024-04-03T16:17:08.0105109Z 7a2c55901189: Pull complete 2024-04-03T16:17:08.3110073Z 09d415c238d7: Pull complete 2024-04-03T16:17:09.5042102Z 9fe6e2e61518: Pull complete 2024-04-03T16:17:09.6402378Z 41f16248e682: Pull complete 2024-04-03T16:17:09.7675905Z 95d7b7817039: Pull complete 2024-04-03T16:17:14.2488854Z 8b29859251be: Verifying Checksum 2024-04-03T16:17:14.2489529Z 8b29859251be: Download complete 2024-04-03T16:17:14.3367557Z 0e1508d7c9ab: Verifying Checksum 2024-04-03T16:17:14.3368185Z 0e1508d7c9ab: Download complete 2024-04-03T16:17:14.4320798Z 89f208ed177f: Verifying Checksum 2024-04-03T16:17:14.4321811Z 89f208ed177f: Download complete 2024-04-03T16:17:14.5261987Z ddecf600b7ac: Verifying Checksum 2024-04-03T16:17:14.5263355Z ddecf600b7ac: Download complete 2024-04-03T16:17:15.5522229Z 7965ac4a7eb7: Verifying Checksum 2024-04-03T16:17:15.5542226Z 7965ac4a7eb7: Download complete 2024-04-03T16:17:15.6410308Z 6b9458cb1b82: Verifying Checksum 2024-04-03T16:17:15.6411000Z 6b9458cb1b82: Download complete 2024-04-03T16:17:15.7137689Z a40a390f581a: Verifying Checksum 2024-04-03T16:17:15.7138706Z a40a390f581a: Download complete 2024-04-03T16:17:15.8229575Z 01d4a84baddb: Verifying Checksum 2024-04-03T16:17:15.8230188Z 01d4a84baddb: Download complete 2024-04-03T16:17:21.0124109Z 8f6c90485347: Verifying Checksum 2024-04-03T16:17:21.0124869Z 8f6c90485347: Download complete 2024-04-03T16:17:21.0912565Z 3eaa6199e983: Verifying Checksum 2024-04-03T16:17:21.0913310Z 3eaa6199e983: Download complete 2024-04-03T16:17:21.1882477Z dd253632212f: Download complete 2024-04-03T16:17:21.2814029Z 5e6597056347: Verifying Checksum 2024-04-03T16:17:21.2814777Z 5e6597056347: Download complete 2024-04-03T16:17:21.3563568Z 858ba61782a0: Verifying Checksum 2024-04-03T16:17:21.3564679Z 858ba61782a0: Download complete 2024-04-03T16:17:21.4382785Z 9cb381d49846: Verifying Checksum 2024-04-03T16:17:21.4383502Z 9cb381d49846: Download complete 2024-04-03T16:17:22.7213040Z 610c26b5e9a0: Verifying Checksum 2024-04-03T16:17:22.7213712Z 610c26b5e9a0: Download complete 2024-04-03T16:17:22.8085632Z 03a7067427cf: Verifying Checksum 2024-04-03T16:17:22.8086303Z 03a7067427cf: Download complete 2024-04-03T16:17:22.8973100Z b4f17b88d23e: Verifying Checksum 2024-04-03T16:17:22.9841526Z b4f17b88d23e: Download complete 2024-04-03T16:17:22.9842143Z bc6bfcbe5530: Verifying Checksum 2024-04-03T16:17:22.9842730Z bc6bfcbe5530: Download complete 2024-04-03T16:17:23.0909173Z df877d3106b5: Verifying Checksum 2024-04-03T16:17:23.0909830Z df877d3106b5: Download complete 2024-04-03T16:17:23.1871920Z 0fbc7e84a570: Verifying Checksum 2024-04-03T16:17:23.1872596Z 0fbc7e84a570: Download complete 2024-04-03T16:17:30.3286196Z 1570829375e6: Verifying Checksum 2024-04-03T16:17:30.3286867Z 1570829375e6: Download complete 2024-04-03T16:17:30.4233215Z 6281fee72d3f: Download complete 2024-04-03T16:17:30.5114486Z d01dc0055b87: Verifying Checksum 2024-04-03T16:17:30.5115359Z d01dc0055b87: Download complete 2024-04-03T16:17:31.3152120Z 2abb072c5540: Verifying Checksum 2024-04-03T16:17:31.3152762Z 2abb072c5540: Download complete 2024-04-03T16:17:31.4160209Z 1323b8068178: Verifying Checksum 2024-04-03T16:17:31.4160827Z 1323b8068178: Download complete 2024-04-03T16:17:31.5086707Z 473ae4beebc0: Download complete 2024-04-03T16:17:31.9990272Z 289a48eb62b5: Verifying Checksum 2024-04-03T16:17:31.9990904Z 289a48eb62b5: Download complete 2024-04-03T16:17:32.0061874Z d67c111fa588: Verifying Checksum 2024-04-03T16:17:32.0062449Z d67c111fa588: Download complete 2024-04-03T16:17:32.1024850Z 6ac1d762661f: Verifying Checksum 2024-04-03T16:17:32.1025943Z 6ac1d762661f: Download complete 2024-04-03T16:17:32.1060606Z 796d028fd584: Verifying Checksum 2024-04-03T16:17:32.1061376Z 796d028fd584: Download complete 2024-04-03T16:17:32.1168800Z 4f4fb700ef54: Verifying Checksum 2024-04-03T16:17:32.1169452Z 4f4fb700ef54: Download complete 2024-04-03T16:17:32.2135421Z a3890e7012d3: Verifying Checksum 2024-04-03T16:17:32.2136634Z a3890e7012d3: Download complete 2024-04-03T16:17:32.2378864Z b3d241c09291: Download complete 2024-04-03T16:17:32.2992184Z c584774f4251: Verifying Checksum 2024-04-03T16:17:32.2992828Z c584774f4251: Download complete 2024-04-03T16:17:32.3835585Z 8b7274bb07ec: Verifying Checksum 2024-04-03T16:17:32.3836766Z 8b7274bb07ec: Download complete 2024-04-03T16:17:32.4729712Z f2934ba32628: Verifying Checksum 2024-04-03T16:17:32.4730694Z f2934ba32628: Download complete 2024-04-03T16:17:32.5478705Z d1c6f5fd2ce3: Verifying Checksum 2024-04-03T16:17:32.5479884Z d1c6f5fd2ce3: Download complete 2024-04-03T16:17:32.6373576Z 9c3c29d675a6: Verifying Checksum 2024-04-03T16:17:32.6374793Z 9c3c29d675a6: Download complete 2024-04-03T16:17:33.4826492Z e4b3bc5cda04: Verifying Checksum 2024-04-03T16:17:33.4827195Z e4b3bc5cda04: Download complete 2024-04-03T16:17:33.6575807Z d39766ef1e6b: Verifying Checksum 2024-04-03T16:17:33.6576472Z d39766ef1e6b: Download complete 2024-04-03T16:17:34.0378864Z 8ce169be4b1d: Verifying Checksum 2024-04-03T16:17:34.0379522Z 8ce169be4b1d: Download complete 2024-04-03T16:17:34.1218313Z fe2cd77be091: Verifying Checksum 2024-04-03T16:17:34.1219529Z fe2cd77be091: Download complete 2024-04-03T16:17:34.2752330Z 2a5bd85dd325: Download complete 2024-04-03T16:17:34.4298278Z d1190269f41e: Download complete 2024-04-03T16:17:34.5166757Z e8752c122b39: Verifying Checksum 2024-04-03T16:17:34.5167383Z e8752c122b39: Download complete 2024-04-03T16:17:34.7968500Z 1eaea607f89d: Verifying Checksum 2024-04-03T16:17:34.7969448Z 1eaea607f89d: Download complete 2024-04-03T16:17:34.8910068Z 9ef6078991a0: Verifying Checksum 2024-04-03T16:17:34.8910694Z 9ef6078991a0: Download complete 2024-04-03T16:17:36.6891653Z c0dab7a64f9b: Verifying Checksum 2024-04-03T16:17:36.6892355Z c0dab7a64f9b: Download complete 2024-04-03T16:17:36.7822224Z 20e7673e255e: Verifying Checksum 2024-04-03T16:17:36.7822851Z 20e7673e255e: Download complete 2024-04-03T16:17:43.1009568Z 8f6c90485347: Pull complete 2024-04-03T16:17:45.0914465Z ab17245097e4: Pull complete 2024-04-03T16:17:46.9702480Z dfecd7e9912b: Pull complete 2024-04-03T16:17:48.8798675Z 464a8f745445: Pull complete 2024-04-03T16:18:13.0197790Z 865db895296b: Verifying Checksum 2024-04-03T16:18:13.0198464Z 865db895296b: Download complete 2024-04-03T16:18:13.1982577Z effd5b6e0122: Verifying Checksum 2024-04-03T16:18:13.1983260Z effd5b6e0122: Download complete 2024-04-03T16:18:13.3060005Z 6150e9996323: Verifying Checksum 2024-04-03T16:18:13.3060670Z 6150e9996323: Download complete 2024-04-03T16:18:14.0882845Z 7ac5ac4b1576: Verifying Checksum 2024-04-03T16:18:14.0884049Z 7ac5ac4b1576: Download complete 2024-04-03T16:18:14.1771739Z f31a22b80d01: Download complete 2024-04-03T16:18:16.3389737Z e400461a2932: Verifying Checksum 2024-04-03T16:18:16.3390895Z e400461a2932: Download complete 2024-04-03T16:18:20.2769834Z db8dcb001b22: Verifying Checksum 2024-04-03T16:18:20.2770523Z db8dcb001b22: Download complete 2024-04-03T16:18:28.2377234Z d67c111fa588: Pull complete 2024-04-03T16:18:30.1308661Z 0723abde19c0: Pull complete 2024-04-03T16:18:32.0092513Z 020f47d0fa1b: Pull complete 2024-04-03T16:18:46.6040794Z 8b29859251be: Pull complete 2024-04-03T16:18:48.4480690Z 0e1508d7c9ab: Pull complete 2024-04-03T16:18:49.3872351Z 89f208ed177f: Pull complete 2024-04-03T16:18:49.4913024Z ddecf600b7ac: Pull complete 2024-04-03T16:18:51.8442822Z 7965ac4a7eb7: Pull complete 2024-04-03T16:18:51.9559941Z 6b9458cb1b82: Pull complete 2024-04-03T16:18:52.0711102Z a40a390f581a: Pull complete 2024-04-03T16:18:52.1838023Z 01d4a84baddb: Pull complete 2024-04-03T16:19:45.0891130Z db8dcb001b22: Pull complete 2024-04-03T16:19:46.9677964Z 3eaa6199e983: Pull complete 2024-04-03T16:19:48.9692816Z dd253632212f: Pull complete 2024-04-03T16:19:50.8137773Z 5e6597056347: Pull complete 2024-04-03T16:19:52.6573002Z 858ba61782a0: Pull complete 2024-04-03T16:19:54.4988092Z 9cb381d49846: Pull complete 2024-04-03T16:20:01.6785917Z 610c26b5e9a0: Pull complete 2024-04-03T16:20:04.6220050Z 03a7067427cf: Pull complete 2024-04-03T16:20:07.4162398Z b4f17b88d23e: Pull complete 2024-04-03T16:20:07.5578104Z bc6bfcbe5530: Pull complete 2024-04-03T16:20:07.6589605Z df877d3106b5: Pull complete 2024-04-03T16:20:07.7509598Z 0fbc7e84a570: Pull complete 2024-04-03T16:20:17.4638750Z 1570829375e6: Pull complete 2024-04-03T16:20:19.5350315Z 6281fee72d3f: Pull complete 2024-04-03T16:20:22.2969388Z d01dc0055b87: Pull complete 2024-04-03T16:20:25.7218337Z 2abb072c5540: Pull complete 2024-04-03T16:20:27.5748967Z 1323b8068178: Pull complete 2024-04-03T16:20:29.4523496Z 473ae4beebc0: Pull complete 2024-04-03T16:20:31.7742863Z 289a48eb62b5: Pull complete 2024-04-03T16:20:33.6222452Z 6ac1d762661f: Pull complete 2024-04-03T16:20:37.3592360Z 796d028fd584: Pull complete 2024-04-03T16:20:39.2367466Z a3890e7012d3: Pull complete 2024-04-03T16:20:41.5475589Z 4f4fb700ef54: Pull complete 2024-04-03T16:20:43.4958985Z b3d241c09291: Pull complete 2024-04-03T16:20:45.4088805Z c584774f4251: Pull complete 2024-04-03T16:20:48.9819010Z e4b3bc5cda04: Pull complete 2024-04-03T16:20:49.1040835Z 8b7274bb07ec: Pull complete 2024-04-03T16:20:49.2041020Z f2934ba32628: Pull complete 2024-04-03T16:20:49.4120035Z d1c6f5fd2ce3: Pull complete 2024-04-03T16:20:49.5199354Z 9c3c29d675a6: Pull complete 2024-04-03T16:21:27.9018937Z e400461a2932: Pull complete 2024-04-03T16:21:29.7485644Z f3b4b7bb5d91: Pull complete 2024-04-03T16:21:31.6553506Z d39766ef1e6b: Pull complete 2024-04-03T16:21:33.6842384Z 8ce169be4b1d: Pull complete 2024-04-03T16:21:35.5598655Z fe2cd77be091: Pull complete 2024-04-03T16:21:37.4382175Z 2a5bd85dd325: Pull complete 2024-04-03T16:21:39.3116159Z d1190269f41e: Pull complete 2024-04-03T16:21:41.2449873Z e8752c122b39: Pull complete 2024-04-03T16:21:44.2001050Z 1eaea607f89d: Pull complete 2024-04-03T16:21:46.0199464Z 9ef6078991a0: Pull complete 2024-04-03T16:21:50.1881442Z c0dab7a64f9b: Pull complete 2024-04-03T16:21:52.1456895Z 20e7673e255e: Pull complete 2024-04-03T16:22:14.3601430Z 865db895296b: Pull complete 2024-04-03T16:22:14.4613037Z effd5b6e0122: Pull complete 2024-04-03T16:22:14.5645260Z 6150e9996323: Pull complete 2024-04-03T16:22:15.3197266Z 7ac5ac4b1576: Pull complete 2024-04-03T16:22:15.4193816Z f31a22b80d01: Pull complete 2024-04-03T16:22:15.7455692Z Digest: sha256:68e0ca2e38253fb3bd10c84d380560df0602da1b16bccdfd7d4dd7ac22529759 2024-04-03T16:22:15.7490215Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:22:15.7519657Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:22:15.7735304Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-04-03T16:22:15.7736115Z with: 2024-04-03T16:22:15.7736499Z driver-version: 535.54.03 2024-04-03T16:22:15.7736984Z env: 2024-04-03T16:22:15.7737363Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:15.7737833Z ##[endgroup] 2024-04-03T16:22:15.7800676Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-04-03T16:22:15.7801480Z with: 2024-04-03T16:22:15.7801844Z timeout_minutes: 10 2024-04-03T16:22:15.7802309Z max_attempts: 3 2024-04-03T16:22:15.7848769Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-04-03T16:22:15.7896468Z retry_wait_seconds: 10 2024-04-03T16:22:15.7896998Z polling_interval_seconds: 1 2024-04-03T16:22:15.7897546Z warning_on_retry: true 2024-04-03T16:22:15.7898030Z continue_on_error: false 2024-04-03T16:22:15.7898509Z env: 2024-04-03T16:22:15.7898887Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:15.7899375Z DRIVER_VERSION: 535.54.03 2024-04-03T16:22:15.7899869Z ##[endgroup] 2024-04-03T16:22:15.8772088Z == Installing nvidia driver NVIDIA-Linux-x86_64-535.54.03.run == 2024-04-03T16:22:15.8774474Z + pre_install_nvidia_driver_amzn2 2024-04-03T16:22:15.8776789Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-04-03T16:22:16.4684953Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-04-03T16:22:16.5348034Z No Match for argument: nvidia-driver-latest-dkms 2024-04-03T16:22:16.5774604Z No Packages marked for removal 2024-04-03T16:22:16.5946310Z + install_nvidia_driver_common 2024-04-03T16:22:16.5950758Z + echo 'Before installing NVIDIA driver' 2024-04-03T16:22:16.5952069Z Before installing NVIDIA driver 2024-04-03T16:22:16.5953778Z + lspci 2024-04-03T16:22:16.6174458Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-04-03T16:22:16.6176651Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-04-03T16:22:16.6178822Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-04-03T16:22:16.6180877Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-04-03T16:22:16.6182636Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-04-03T16:22:16.6184503Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-04-03T16:22:16.6186768Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-04-03T16:22:16.6189096Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-04-03T16:22:16.6191373Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-04-03T16:22:16.6192957Z + lsmod 2024-04-03T16:22:16.6198747Z Module Size Used by 2024-04-03T16:22:16.6199922Z backlight 16384 0 2024-04-03T16:22:16.6200974Z xt_conntrack 16384 1 2024-04-03T16:22:16.6202001Z ipt_MASQUERADE 16384 1 2024-04-03T16:22:16.6203197Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-04-03T16:22:16.6204501Z nf_conntrack_netlink 49152 0 2024-04-03T16:22:16.6205710Z nfnetlink 16384 2 nf_conntrack_netlink 2024-04-03T16:22:16.6207258Z xfrm_user 45056 1 2024-04-03T16:22:16.6208368Z xfrm_algo 16384 1 xfrm_user 2024-04-03T16:22:16.6209585Z iptable_nat 16384 1 2024-04-03T16:22:16.6210575Z nf_conntrack_ipv4 16384 3 2024-04-03T16:22:16.6211678Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-04-03T16:22:16.6212965Z nf_nat_ipv4 16384 1 iptable_nat 2024-04-03T16:22:16.6214044Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-04-03T16:22:16.6216609Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-04-03T16:22:16.6218502Z xt_addrtype 16384 2 2024-04-03T16:22:16.6219038Z iptable_filter 16384 1 2024-04-03T16:22:16.6219593Z br_netfilter 24576 0 2024-04-03T16:22:16.6220184Z bridge 172032 1 br_netfilter 2024-04-03T16:22:16.6220800Z stp 16384 1 bridge 2024-04-03T16:22:16.6221408Z llc 16384 2 bridge,stp 2024-04-03T16:22:16.6222013Z overlay 86016 0 2024-04-03T16:22:16.6222522Z sunrpc 393216 1 2024-04-03T16:22:16.6223055Z dm_mirror 28672 0 2024-04-03T16:22:16.6223632Z dm_region_hash 20480 1 dm_mirror 2024-04-03T16:22:16.6224316Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-04-03T16:22:16.6225045Z dm_mod 143360 2 dm_log,dm_mirror 2024-04-03T16:22:16.6225687Z dax 69632 1 dm_mod 2024-04-03T16:22:16.6226231Z sb_edac 24576 0 2024-04-03T16:22:16.6226785Z crc32_pclmul 16384 0 2024-04-03T16:22:16.6227371Z ghash_clmulni_intel 16384 0 2024-04-03T16:22:16.6227918Z pcbc 16384 0 2024-04-03T16:22:16.6228424Z aesni_intel 188416 0 2024-04-03T16:22:16.6228999Z aes_x86_64 20480 1 aesni_intel 2024-04-03T16:22:16.6229644Z crypto_simd 16384 1 aesni_intel 2024-04-03T16:22:16.6230275Z glue_helper 16384 1 aesni_intel 2024-04-03T16:22:16.6231072Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-04-03T16:22:16.6231851Z mousedev 24576 0 2024-04-03T16:22:16.6232355Z evdev 20480 3 2024-04-03T16:22:16.6232887Z pcc_cpufreq 16384 0 2024-04-03T16:22:16.6233419Z psmouse 32768 0 2024-04-03T16:22:16.6233922Z button 16384 0 2024-04-03T16:22:16.6235145Z ena 139264 0 2024-04-03T16:22:16.6235677Z ptp 20480 1 ena 2024-04-03T16:22:16.6236377Z pps_core 20480 1 ptp 2024-04-03T16:22:16.6237175Z ata_piix 36864 0 2024-04-03T16:22:16.6237842Z libata 266240 1 ata_piix 2024-04-03T16:22:16.6238455Z xen_blkfront 49152 2 2024-04-03T16:22:16.6238980Z crc32c_intel 24576 0 2024-04-03T16:22:16.6239522Z scsi_mod 245760 1 libata 2024-04-03T16:22:16.6240076Z autofs4 49152 2 2024-04-03T16:22:16.6240594Z + modinfo nvidia 2024-04-03T16:22:16.6241565Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-04-03T16:22:16.6242558Z firmware: nvidia/535.54.03/gsp_tu10x.bin 2024-04-03T16:22:16.6243252Z firmware: nvidia/535.54.03/gsp_ga10x.bin 2024-04-03T16:22:16.6243963Z alias: char-major-195-* 2024-04-03T16:22:16.6244490Z version: 535.54.03 2024-04-03T16:22:16.6244976Z supported: external 2024-04-03T16:22:16.6245456Z license: NVIDIA 2024-04-03T16:22:16.6245939Z srcversion: EA9C7EF32617E104C8240C4 2024-04-03T16:22:16.6246619Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-04-03T16:22:16.6247334Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-04-03T16:22:16.6248017Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-04-03T16:22:16.6248721Z depends: i2c-core,drm 2024-04-03T16:22:16.6249235Z retpoline: Y 2024-04-03T16:22:16.6249653Z name: nvidia 2024-04-03T16:22:16.6250593Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-04-03T16:22:16.6251562Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-04-03T16:22:16.6252483Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-04-03T16:22:16.6253345Z parm: NVreg_ResmanDebugLevel:int 2024-04-03T16:22:16.6253978Z parm: NVreg_RmLogonRC:int 2024-04-03T16:22:16.6254601Z parm: NVreg_ModifyDeviceFiles:int 2024-04-03T16:22:16.6255239Z parm: NVreg_DeviceFileUID:int 2024-04-03T16:22:16.6255861Z parm: NVreg_DeviceFileGID:int 2024-04-03T16:22:16.6256529Z parm: NVreg_DeviceFileMode:int 2024-04-03T16:22:16.6257275Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-04-03T16:22:16.6258095Z parm: NVreg_UsePageAttributeTable:int 2024-04-03T16:22:16.6258786Z parm: NVreg_EnablePCIeGen3:int 2024-04-03T16:22:16.6259381Z parm: NVreg_EnableMSI:int 2024-04-03T16:22:16.6259973Z parm: NVreg_TCEBypassMode:int 2024-04-03T16:22:16.6260638Z parm: NVreg_EnableStreamMemOPs:int 2024-04-03T16:22:16.6261386Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-04-03T16:22:16.6262226Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-04-03T16:22:16.6263039Z parm: NVreg_EnableS0ixPowerManagement:int 2024-04-03T16:22:16.6263906Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-04-03T16:22:16.6264774Z parm: NVreg_DynamicPowerManagement:int 2024-04-03T16:22:16.6265662Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-04-03T16:22:16.6266532Z parm: NVreg_EnableGpuFirmware:int 2024-04-03T16:22:16.6267219Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-04-03T16:22:16.6267985Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-04-03T16:22:16.6268768Z parm: NVreg_EnableUserNUMAManagement:int 2024-04-03T16:22:16.6269461Z parm: NVreg_MemoryPoolSize:int 2024-04-03T16:22:16.6270131Z parm: NVreg_KMallocHeapMaxSize:int 2024-04-03T16:22:16.6270827Z parm: NVreg_VMallocHeapMaxSize:int 2024-04-03T16:22:16.6271485Z parm: NVreg_IgnoreMMIOCheck:int 2024-04-03T16:22:16.6272117Z parm: NVreg_NvLinkDisable:int 2024-04-03T16:22:16.6272837Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-04-03T16:22:16.6273579Z parm: NVreg_RegisterPCIDriver:int 2024-04-03T16:22:16.6274870Z parm: NVreg_EnableResizableBar:int 2024-04-03T16:22:16.6275596Z parm: NVreg_EnableDbgBreakpoint:int 2024-04-03T16:22:16.6276438Z parm: NVreg_RegistryDwords:charp 2024-04-03T16:22:16.6277158Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-04-03T16:22:16.6277846Z parm: NVreg_RmMsg:charp 2024-04-03T16:22:16.6278414Z parm: NVreg_GpuBlacklist:charp 2024-04-03T16:22:16.6279084Z parm: NVreg_TemporaryFilePath:charp 2024-04-03T16:22:16.6279758Z parm: NVreg_ExcludedGpus:charp 2024-04-03T16:22:16.6280396Z parm: NVreg_DmaRemapPeerMmio:int 2024-04-03T16:22:16.6281086Z parm: NVreg_RmNvlinkBandwidth:charp 2024-04-03T16:22:16.6281751Z parm: rm_firmware_active:charp 2024-04-03T16:22:16.6282323Z + HAS_NVIDIA_DRIVER=0 2024-04-03T16:22:16.6282865Z ++ command -v nvidia-smi 2024-04-03T16:22:16.6283427Z + '[' -x /usr/bin/nvidia-smi ']' 2024-04-03T16:22:16.6283929Z + set +e 2024-04-03T16:22:16.6284648Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-04-03T16:22:20.2181887Z + INSTALLED_DRIVER_VERSION=535.54.03 2024-04-03T16:22:20.2182588Z + NVIDIA_SMI_STATUS=0 2024-04-03T16:22:20.2183235Z + '[' 0 -ne 0 ']' 2024-04-03T16:22:20.2183730Z + '[' 535.54.03 '!=' 535.54.03 ']' 2024-04-03T16:22:20.2184276Z + HAS_NVIDIA_DRIVER=1 2024-04-03T16:22:20.2185330Z + echo 'NVIDIA driver (535.54.03) has already been installed. Skipping NVIDIA driver installation' 2024-04-03T16:22:20.2186342Z + set -e 2024-04-03T16:22:20.2186772Z + '[' 1 -eq 0 ']' 2024-04-03T16:22:20.2187895Z NVIDIA driver (535.54.03) has already been installed. Skipping NVIDIA driver installation 2024-04-03T16:22:20.2188944Z + post_install_nvidia_driver_common 2024-04-03T16:22:20.2189502Z + sudo modprobe nvidia 2024-04-03T16:22:20.2324075Z + echo 'After installing NVIDIA driver' 2024-04-03T16:22:20.2324686Z + lspci 2024-04-03T16:22:20.2325880Z After installing NVIDIA driver 2024-04-03T16:22:20.2539164Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-04-03T16:22:20.2540336Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-04-03T16:22:20.2541476Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-04-03T16:22:20.2542566Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-04-03T16:22:20.2543498Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-04-03T16:22:20.2544490Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-04-03T16:22:20.2545675Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-04-03T16:22:20.2546906Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-04-03T16:22:20.2548107Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-04-03T16:22:20.2548933Z + lsmod 2024-04-03T16:22:20.2564556Z Module Size Used by 2024-04-03T16:22:20.2565149Z nvidia_uvm 1466368 0 2024-04-03T16:22:20.2565703Z nvidia 56422400 1 nvidia_uvm 2024-04-03T16:22:20.2566314Z drm 421888 1 nvidia 2024-04-03T16:22:20.2566933Z i2c_core 77824 2 nvidia,drm 2024-04-03T16:22:20.2567519Z backlight 16384 0 2024-04-03T16:22:20.2568058Z xt_conntrack 16384 1 2024-04-03T16:22:20.2568596Z ipt_MASQUERADE 16384 1 2024-04-03T16:22:20.2569198Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-04-03T16:22:20.2569899Z nf_conntrack_netlink 49152 0 2024-04-03T16:22:20.2570546Z nfnetlink 16384 2 nf_conntrack_netlink 2024-04-03T16:22:20.2571210Z xfrm_user 45056 1 2024-04-03T16:22:20.2571749Z xfrm_algo 16384 1 xfrm_user 2024-04-03T16:22:20.2572338Z iptable_nat 16384 1 2024-04-03T16:22:20.2572878Z nf_conntrack_ipv4 16384 3 2024-04-03T16:22:20.2573475Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-04-03T16:22:20.2574157Z nf_nat_ipv4 16384 1 iptable_nat 2024-04-03T16:22:20.2574906Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-04-03T16:22:20.2576267Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-04-03T16:22:20.2577774Z xt_addrtype 16384 2 2024-04-03T16:22:20.2578318Z iptable_filter 16384 1 2024-04-03T16:22:20.2578848Z br_netfilter 24576 0 2024-04-03T16:22:20.2579432Z bridge 172032 1 br_netfilter 2024-04-03T16:22:20.2580066Z stp 16384 1 bridge 2024-04-03T16:22:20.2580656Z llc 16384 2 bridge,stp 2024-04-03T16:22:20.2581258Z overlay 86016 0 2024-04-03T16:22:20.2581785Z sunrpc 393216 1 2024-04-03T16:22:20.2582293Z dm_mirror 28672 0 2024-04-03T16:22:20.2582852Z dm_region_hash 20480 1 dm_mirror 2024-04-03T16:22:20.2583550Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-04-03T16:22:20.2584262Z dm_mod 143360 2 dm_log,dm_mirror 2024-04-03T16:22:20.2584898Z dax 69632 1 dm_mod 2024-04-03T16:22:20.2585458Z sb_edac 24576 0 2024-04-03T16:22:20.2585979Z crc32_pclmul 16384 0 2024-04-03T16:22:20.2586523Z ghash_clmulni_intel 16384 0 2024-04-03T16:22:20.2587062Z pcbc 16384 0 2024-04-03T16:22:20.2587567Z aesni_intel 188416 0 2024-04-03T16:22:20.2588127Z aes_x86_64 20480 1 aesni_intel 2024-04-03T16:22:20.2588767Z crypto_simd 16384 1 aesni_intel 2024-04-03T16:22:20.2589535Z glue_helper 16384 1 aesni_intel 2024-04-03T16:22:20.2590352Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-04-03T16:22:20.2591124Z mousedev 24576 0 2024-04-03T16:22:20.2591630Z evdev 20480 3 2024-04-03T16:22:20.2592158Z pcc_cpufreq 16384 0 2024-04-03T16:22:20.2592891Z psmouse 32768 0 2024-04-03T16:22:20.2593400Z button 16384 0 2024-04-03T16:22:20.2593913Z ena 139264 0 2024-04-03T16:22:20.2594888Z ptp 20480 1 ena 2024-04-03T16:22:20.2595448Z pps_core 20480 1 ptp 2024-04-03T16:22:20.2596003Z ata_piix 36864 0 2024-04-03T16:22:20.2596555Z libata 266240 1 ata_piix 2024-04-03T16:22:20.2597122Z xen_blkfront 49152 2 2024-04-03T16:22:20.2597651Z crc32c_intel 24576 0 2024-04-03T16:22:20.2598191Z scsi_mod 245760 1 libata 2024-04-03T16:22:20.2598740Z autofs4 49152 2 2024-04-03T16:22:20.2599258Z + modinfo nvidia 2024-04-03T16:22:20.2600241Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-04-03T16:22:20.2601241Z firmware: nvidia/535.54.03/gsp_tu10x.bin 2024-04-03T16:22:20.2601914Z firmware: nvidia/535.54.03/gsp_ga10x.bin 2024-04-03T16:22:20.2602625Z alias: char-major-195-* 2024-04-03T16:22:20.2603169Z version: 535.54.03 2024-04-03T16:22:20.2603641Z supported: external 2024-04-03T16:22:20.2604119Z license: NVIDIA 2024-04-03T16:22:20.2604616Z srcversion: EA9C7EF32617E104C8240C4 2024-04-03T16:22:20.2605280Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-04-03T16:22:20.2605997Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-04-03T16:22:20.2606697Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-04-03T16:22:20.2607378Z depends: i2c-core,drm 2024-04-03T16:22:20.2607885Z retpoline: Y 2024-04-03T16:22:20.2608315Z name: nvidia 2024-04-03T16:22:20.2609096Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-04-03T16:22:20.2610046Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-04-03T16:22:20.2610976Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-04-03T16:22:20.2611817Z parm: NVreg_ResmanDebugLevel:int 2024-04-03T16:22:20.2612450Z parm: NVreg_RmLogonRC:int 2024-04-03T16:22:20.2613066Z parm: NVreg_ModifyDeviceFiles:int 2024-04-03T16:22:20.2613699Z parm: NVreg_DeviceFileUID:int 2024-04-03T16:22:20.2614496Z parm: NVreg_DeviceFileGID:int 2024-04-03T16:22:20.2615124Z parm: NVreg_DeviceFileMode:int 2024-04-03T16:22:20.2615858Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-04-03T16:22:20.2616666Z parm: NVreg_UsePageAttributeTable:int 2024-04-03T16:22:20.2617349Z parm: NVreg_EnablePCIeGen3:int 2024-04-03T16:22:20.2617941Z parm: NVreg_EnableMSI:int 2024-04-03T16:22:20.2618542Z parm: NVreg_TCEBypassMode:int 2024-04-03T16:22:20.2619198Z parm: NVreg_EnableStreamMemOPs:int 2024-04-03T16:22:20.2619958Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-04-03T16:22:20.2620777Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-04-03T16:22:20.2621582Z parm: NVreg_EnableS0ixPowerManagement:int 2024-04-03T16:22:20.2622457Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-04-03T16:22:20.2623308Z parm: NVreg_DynamicPowerManagement:int 2024-04-03T16:22:20.2624192Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-04-03T16:22:20.2625056Z parm: NVreg_EnableGpuFirmware:int 2024-04-03T16:22:20.2625743Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-04-03T16:22:20.2626509Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-04-03T16:22:20.2627297Z parm: NVreg_EnableUserNUMAManagement:int 2024-04-03T16:22:20.2627992Z parm: NVreg_MemoryPoolSize:int 2024-04-03T16:22:20.2628781Z parm: NVreg_KMallocHeapMaxSize:int 2024-04-03T16:22:20.2629500Z parm: NVreg_VMallocHeapMaxSize:int 2024-04-03T16:22:20.2630157Z parm: NVreg_IgnoreMMIOCheck:int 2024-04-03T16:22:20.2630793Z parm: NVreg_NvLinkDisable:int 2024-04-03T16:22:20.2631512Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-04-03T16:22:20.2632251Z parm: NVreg_RegisterPCIDriver:int 2024-04-03T16:22:20.2632935Z parm: NVreg_EnableResizableBar:int 2024-04-03T16:22:20.2633631Z parm: NVreg_EnableDbgBreakpoint:int 2024-04-03T16:22:20.2634588Z parm: NVreg_RegistryDwords:charp 2024-04-03T16:22:20.2635314Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-04-03T16:22:20.2636001Z parm: NVreg_RmMsg:charp 2024-04-03T16:22:20.2636564Z parm: NVreg_GpuBlacklist:charp 2024-04-03T16:22:20.2637226Z parm: NVreg_TemporaryFilePath:charp 2024-04-03T16:22:20.2637932Z parm: NVreg_ExcludedGpus:charp 2024-04-03T16:22:20.2638580Z parm: NVreg_DmaRemapPeerMmio:int 2024-04-03T16:22:20.2639266Z parm: NVreg_RmNvlinkBandwidth:charp 2024-04-03T16:22:20.2639924Z parm: rm_firmware_active:charp 2024-04-03T16:22:20.2640478Z + set +e 2024-04-03T16:22:20.2640907Z + nvidia-smi 2024-04-03T16:22:21.7354560Z Wed Apr 3 16:22:21 2024 2024-04-03T16:22:21.7356052Z +---------------------------------------------------------------------------------------+ 2024-04-03T16:22:21.7357247Z | NVIDIA-SMI 535.54.03 Driver Version: 535.54.03 CUDA Version: 12.2 | 2024-04-03T16:22:21.7358410Z |-----------------------------------------+----------------------+----------------------+ 2024-04-03T16:22:21.7359566Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-04-03T16:22:21.7360853Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-04-03T16:22:21.7361859Z | | | MIG M. | 2024-04-03T16:22:21.7362656Z |=========================================+======================+======================| 2024-04-03T16:22:21.8402263Z | 0 Tesla M60 Off | 00000000:00:1D.0 Off | 0 | 2024-04-03T16:22:21.8403480Z | N/A 28C P0 39W / 150W | 0MiB / 7680MiB | 57% Default | 2024-04-03T16:22:21.8404487Z | | | N/A | 2024-04-03T16:22:21.8405466Z +-----------------------------------------+----------------------+----------------------+ 2024-04-03T16:22:21.8406761Z | 1 Tesla M60 Off | 00000000:00:1E.0 Off | 0 | 2024-04-03T16:22:21.8407750Z | N/A 33C P0 38W / 150W | 0MiB / 7680MiB | 6% Default | 2024-04-03T16:22:21.8408671Z | | | N/A | 2024-04-03T16:22:21.8409647Z +-----------------------------------------+----------------------+----------------------+ 2024-04-03T16:22:21.8410482Z 2024-04-03T16:22:21.8411417Z +---------------------------------------------------------------------------------------+ 2024-04-03T16:22:21.8412324Z | Processes: | 2024-04-03T16:22:21.8413344Z | GPU GI CI PID Type Process name GPU Memory | 2024-04-03T16:22:21.8414330Z | ID ID Usage | 2024-04-03T16:22:21.8415148Z |=======================================================================================| 2024-04-03T16:22:21.8416091Z | No running processes found | 2024-04-03T16:22:21.8417333Z +---------------------------------------------------------------------------------------+ 2024-04-03T16:22:22.6809773Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-04-03T16:22:24.1218698Z Tesla M60 2024-04-03T16:22:25.0014722Z + NVIDIA_SMI_STATUS=0 2024-04-03T16:22:25.0016073Z + '[' 0 -eq 0 ']' 2024-04-03T16:22:25.0016778Z + echo 'INFO: Ignoring allowed status 0' 2024-04-03T16:22:25.0017397Z + set -e 2024-04-03T16:22:25.0017797Z INFO: Ignoring allowed status 0 2024-04-03T16:22:25.0022225Z == Installing nvidia container toolkit for amzn2 == 2024-04-03T16:22:25.0025869Z + sudo yum install -y yum-utils 2024-04-03T16:22:25.5942535Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-04-03T16:22:27.4881017Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-04-03T16:22:27.4882014Z Nothing to do 2024-04-03T16:22:27.6547637Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-04-03T16:22:28.2457993Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-04-03T16:22:28.2794116Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-04-03T16:22:28.2796449Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-04-03T16:22:28.2797855Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-04-03T16:22:28.2950968Z + sudo yum install -y nvidia-docker2 2024-04-03T16:22:28.8665780Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-04-03T16:22:30.7418344Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-04-03T16:22:30.7419336Z Nothing to do 2024-04-03T16:22:30.9067831Z + sudo systemctl restart docker 2024-04-03T16:22:47.9070130Z Command completed after 1 attempt(s). 2024-04-03T16:22:47.9161673Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-04-03T16:22:47.9182053Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-04-03T16:22:47.9183084Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-04-03T16:22:47.9184034Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:22:47.9193920Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:22:47.9195393Z env: 2024-04-03T16:22:47.9195789Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:47.9196439Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:47.9197114Z ##[endgroup] 2024-04-03T16:22:48.6187275Z Defaulting to user installation because normal site-packages is not writeable 2024-04-03T16:22:49.1264842Z Collecting psutil==5.9.1 2024-04-03T16:22:49.1480851Z Downloading psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-04-03T16:22:49.2074225Z Collecting nvidia-ml-py==11.525.84 2024-04-03T16:22:49.2136049Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-04-03T16:22:49.2682437Z Installing collected packages: psutil, nvidia-ml-py 2024-04-03T16:22:49.4813617Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-04-03T16:22:49.7212454Z Prepare all required actions 2024-04-03T16:22:49.7213208Z Getting action download info 2024-04-03T16:22:49.9066817Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-04-03T16:22:50.1400692Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-04-03T16:22:50.2726669Z ##[group]Run ./.github/actions/download-build-artifacts 2024-04-03T16:22:50.2727384Z with: 2024-04-03T16:22:50.2727814Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-04-03T16:22:50.2728438Z s3-bucket: gha-artifacts 2024-04-03T16:22:50.2728915Z env: 2024-04-03T16:22:50.2729277Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:50.2729922Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:50.2730616Z ##[endgroup] 2024-04-03T16:22:50.3284627Z ##[group]Run seemethere/download-artifact-s3@v4 2024-04-03T16:22:50.3285270Z with: 2024-04-03T16:22:50.3285761Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-04-03T16:22:50.3286372Z s3-bucket: gha-artifacts 2024-04-03T16:22:50.3286871Z region: us-east-1 2024-04-03T16:22:50.3287292Z env: 2024-04-03T16:22:50.3287653Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:50.3288298Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:50.3288993Z ##[endgroup] 2024-04-03T16:22:51.0124925Z (node:90248) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-04-03T16:22:51.0126602Z 2024-04-03T16:22:51.0126993Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-04-03T16:22:51.0128051Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-04-03T16:22:51.0129280Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-04-03T16:22:51.1339938Z Found 1 objects with prefix pytorch/pytorch/8536950919/linux-focal-cuda11.8-py3.10-gcc9/ 2024-04-03T16:22:51.1342018Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-04-03T16:22:58.7299292Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-04-03T16:22:58.7311495Z Artifact download has finished successfully 2024-04-03T16:22:58.7541779Z ##[group]Run unzip -o artifacts.zip 2024-04-03T16:22:58.7542407Z unzip -o artifacts.zip 2024-04-03T16:22:58.7552060Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:22:58.7552789Z env: 2024-04-03T16:22:58.7553179Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:58.7553835Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:58.7554849Z ##[endgroup] 2024-04-03T16:22:58.7616296Z Archive: artifacts.zip 2024-04-03T16:22:58.7618577Z creating: dist/ 2024-04-03T16:23:01.5622521Z inflating: dist/torch-2.4.0a0+gitd253bd7-cp310-cp310-linux_x86_64.whl 2024-04-03T16:23:01.5623516Z creating: build/custom_test_artifacts/ 2024-04-03T16:23:01.5624368Z creating: build/custom_test_artifacts/custom-op-build/ 2024-04-03T16:23:01.5625416Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-04-03T16:23:01.5626646Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-04-03T16:23:01.5633894Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-04-03T16:23:01.5635573Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-04-03T16:23:01.5636979Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-04-03T16:23:01.5638834Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-04-03T16:23:01.5640278Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-04-03T16:23:01.5641910Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-04-03T16:23:01.5643742Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-04-03T16:23:01.5645261Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-04-03T16:23:01.5646738Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-04-03T16:23:01.5648449Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-04-03T16:23:01.5650177Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-04-03T16:23:01.5651896Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-04-03T16:23:01.5653574Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-04-03T16:23:01.5655316Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-04-03T16:23:01.5657067Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-04-03T16:23:01.5658611Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-04-03T16:23:01.5660089Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-04-03T16:23:01.5724166Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-04-03T16:23:01.5726298Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-04-03T16:23:01.5728476Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-04-03T16:23:01.5730676Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-04-03T16:23:01.5732901Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-04-03T16:23:01.5734983Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-04-03T16:23:01.5737077Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-04-03T16:23:01.5739209Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-04-03T16:23:01.5741315Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-04-03T16:23:01.5789362Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-04-03T16:23:01.5847730Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-04-03T16:23:01.5849807Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-04-03T16:23:01.5851751Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-04-03T16:23:01.5853615Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-04-03T16:23:01.5855438Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-04-03T16:23:01.5857534Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-04-03T16:23:01.5859319Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-04-03T16:23:01.5861146Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-04-03T16:23:01.5944596Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-04-03T16:23:01.6032816Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-04-03T16:23:01.6034840Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-04-03T16:23:01.6036343Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-04-03T16:23:01.6037630Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-04-03T16:23:01.6039021Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-04-03T16:23:01.6040404Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-04-03T16:23:01.6041931Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-04-03T16:23:01.6043640Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-04-03T16:23:01.6045295Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-04-03T16:23:01.6046844Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-04-03T16:23:01.6048451Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-04-03T16:23:01.6050044Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-04-03T16:23:01.6051673Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-04-03T16:23:01.6053280Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-04-03T16:23:01.6054843Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-04-03T16:23:01.6072765Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-04-03T16:23:01.6267108Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-04-03T16:23:01.6268604Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-04-03T16:23:01.6270202Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-04-03T16:23:01.6272026Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-04-03T16:23:01.6273791Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-04-03T16:23:01.6275710Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-04-03T16:23:01.6277382Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-04-03T16:23:01.6279092Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-04-03T16:23:01.6280798Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-04-03T16:23:01.6282484Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-04-03T16:23:01.6284138Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-04-03T16:23:01.6305185Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-04-03T16:23:01.6425494Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-04-03T16:23:01.6427282Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-03T16:23:01.6428876Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-04-03T16:23:01.6430516Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-04-03T16:23:01.6431904Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-04-03T16:23:01.6433221Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-04-03T16:23:01.6434849Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-04-03T16:23:01.6436128Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-04-03T16:23:01.6437286Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-04-03T16:23:01.6438507Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-04-03T16:23:01.6600057Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-04-03T16:23:01.6690979Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-04-03T16:23:01.6692115Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-04-03T16:23:01.6693168Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-04-03T16:23:01.6694363Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-04-03T16:23:01.6702376Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-04-03T16:23:01.6703765Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-04-03T16:23:01.6705151Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-04-03T16:23:01.6706593Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-04-03T16:23:01.6708012Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-04-03T16:23:01.6709639Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-04-03T16:23:01.6711252Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-04-03T16:23:01.6712728Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-04-03T16:23:01.6714497Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-04-03T16:23:01.6716231Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-04-03T16:23:01.6717937Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-04-03T16:23:01.6719629Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-04-03T16:23:01.6721313Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-04-03T16:23:01.6723039Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-04-03T16:23:01.6724736Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-04-03T16:23:01.6726266Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-04-03T16:23:01.6727747Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-04-03T16:23:01.6791690Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-04-03T16:23:01.6793808Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-04-03T16:23:01.6796511Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-04-03T16:23:01.6798712Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-04-03T16:23:01.6800897Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-04-03T16:23:01.6803113Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-04-03T16:23:01.6805215Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-04-03T16:23:01.6807293Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-04-03T16:23:01.6809342Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-04-03T16:23:01.6856909Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-04-03T16:23:01.6915231Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-04-03T16:23:01.6917326Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-04-03T16:23:01.6919222Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-04-03T16:23:01.6921066Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-04-03T16:23:01.6922876Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-04-03T16:23:01.6924700Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-04-03T16:23:01.6926465Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-04-03T16:23:01.6928265Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-04-03T16:23:01.7012097Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-04-03T16:23:01.7100875Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-04-03T16:23:01.7102661Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-04-03T16:23:01.7104125Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-04-03T16:23:01.7105391Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-04-03T16:23:01.7106703Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-04-03T16:23:01.7108115Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-04-03T16:23:01.7109697Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-04-03T16:23:01.7111443Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-04-03T16:23:01.7113144Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-04-03T16:23:01.7115030Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-04-03T16:23:01.7116697Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-04-03T16:23:01.7118328Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-04-03T16:23:01.7119985Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-04-03T16:23:01.7121866Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-04-03T16:23:01.7123494Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-04-03T16:23:01.7140761Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-04-03T16:23:01.7233731Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-04-03T16:23:01.7235774Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-03T16:23:01.7237370Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-04-03T16:23:01.7238826Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-04-03T16:23:01.7240185Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-04-03T16:23:01.7241490Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-04-03T16:23:01.7242791Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-04-03T16:23:01.7244031Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-04-03T16:23:01.7245159Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-04-03T16:23:01.7246349Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-04-03T16:23:01.7316271Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-04-03T16:23:01.7317384Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-04-03T16:23:01.7318501Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-04-03T16:23:01.7319779Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-04-03T16:23:01.7327999Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-04-03T16:23:01.7329473Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-04-03T16:23:01.7330956Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-04-03T16:23:01.7332496Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-04-03T16:23:01.7334033Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-04-03T16:23:01.7335775Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-04-03T16:23:01.7337499Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-04-03T16:23:01.7339132Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-04-03T16:23:01.7340702Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-04-03T16:23:01.7342523Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-04-03T16:23:01.7344314Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-04-03T16:23:01.7346103Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-04-03T16:23:01.7347904Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-04-03T16:23:01.7349731Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-04-03T16:23:01.7351532Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-04-03T16:23:01.7353160Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-04-03T16:23:01.7355283Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-04-03T16:23:01.7417057Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-04-03T16:23:01.7419270Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-04-03T16:23:01.7421747Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-04-03T16:23:01.7424101Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-04-03T16:23:01.7426395Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-04-03T16:23:01.7428547Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-04-03T16:23:01.7430744Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-04-03T16:23:01.7432940Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-04-03T16:23:01.7435418Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-04-03T16:23:01.7482563Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-04-03T16:23:01.7540665Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-04-03T16:23:01.7542840Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-04-03T16:23:01.7544857Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-04-03T16:23:01.7546790Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-04-03T16:23:01.7548699Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-04-03T16:23:01.7550634Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-04-03T16:23:01.7552491Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-04-03T16:23:01.7554693Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-04-03T16:23:01.7637527Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-04-03T16:23:01.7725962Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-04-03T16:23:01.7727827Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-04-03T16:23:01.7729382Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-04-03T16:23:01.7730744Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-04-03T16:23:01.7732191Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-04-03T16:23:01.7733670Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-04-03T16:23:01.7735366Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-04-03T16:23:01.7737241Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-04-03T16:23:01.7739337Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-04-03T16:23:01.7741014Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-04-03T16:23:01.7742775Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-04-03T16:23:01.7744704Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-04-03T16:23:01.7746504Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-04-03T16:23:01.7748248Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-04-03T16:23:01.7749997Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-04-03T16:23:01.7751859Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-04-03T16:23:01.7924700Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-04-03T16:23:01.7926437Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-04-03T16:23:01.7928214Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-04-03T16:23:01.7930181Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-04-03T16:23:01.7932079Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-04-03T16:23:01.7933874Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-04-03T16:23:01.7935732Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-04-03T16:23:01.7937596Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-04-03T16:23:01.7939488Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-04-03T16:23:01.7941329Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-04-03T16:23:01.7943165Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-04-03T16:23:01.7962386Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-04-03T16:23:01.8042411Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-04-03T16:23:01.8044349Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-03T16:23:01.8046066Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-04-03T16:23:01.8047582Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-04-03T16:23:01.8049016Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-04-03T16:23:01.8050434Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-04-03T16:23:01.8051862Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-04-03T16:23:01.8053202Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-04-03T16:23:01.8054434Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-04-03T16:23:01.8055703Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-04-03T16:23:01.8202138Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-04-03T16:23:01.8263956Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-04-03T16:23:01.8264890Z creating: build/lib/ 2024-04-03T16:23:01.8278280Z inflating: build/lib/libpthreadpool.a 2024-04-03T16:23:01.8279274Z inflating: build/lib/libclog.a 2024-04-03T16:23:01.8282759Z inflating: build/lib/libnnpack_reference_layers.a 2024-04-03T16:23:01.8293802Z inflating: build/lib/libcpuinfo_internals.a 2024-04-03T16:23:01.8305507Z inflating: build/lib/libcpuinfo.a 2024-04-03T16:23:01.8434719Z inflating: build/lib/libprotobuf-lite.a 2024-04-03T16:23:01.9097159Z inflating: build/lib/libprotobuf.a 2024-04-03T16:23:01.9191800Z inflating: build/lib/libgtest.a 2024-04-03T16:23:01.9300384Z inflating: build/lib/libbenchmark.a 2024-04-03T16:23:01.9311329Z inflating: build/lib/libittnotify.a 2024-04-03T16:23:01.9403942Z inflating: build/lib/libasmjit.a 2024-04-03T16:23:01.9444372Z inflating: build/lib/libtensorpipe_uv.a 2024-04-03T16:23:01.9629503Z inflating: build/lib/libgloo.a 2024-04-03T16:23:01.9660911Z inflating: build/lib/libfmt.a 2024-04-03T16:23:01.9663378Z inflating: build/lib/libcaffe2_nvrtc.so 2024-04-03T16:23:01.9794132Z inflating: build/lib/libc10.so 2024-04-03T16:23:01.9795290Z inflating: build/lib/libfoxi_loader.a 2024-04-03T16:23:01.9797572Z inflating: build/lib/libtorch_global_deps.so 2024-04-03T16:23:01.9816877Z inflating: build/lib/libqnnpack.a 2024-04-03T16:23:01.9845225Z inflating: build/lib/libpytorch_qnnpack.a 2024-04-03T16:23:02.0577334Z inflating: build/lib/libprotoc.a 2024-04-03T16:23:02.0603425Z inflating: build/lib/libgmock.a 2024-04-03T16:23:02.0604425Z inflating: build/lib/libgtest_main.a 2024-04-03T16:23:02.0605453Z inflating: build/lib/libbenchmark_main.a 2024-04-03T16:23:02.0631903Z inflating: build/lib/libnnpack.a 2024-04-03T16:23:03.4417367Z inflating: build/lib/libdnnl.a 2024-04-03T16:23:03.5249560Z inflating: build/lib/libtensorpipe.a 2024-04-03T16:23:03.5327769Z inflating: build/lib/libc10_cuda.so 2024-04-03T16:23:03.7146774Z inflating: build/lib/libfbgemm.a 2024-04-03T16:23:03.7147450Z inflating: build/lib/libgmock_main.a 2024-04-03T16:23:03.7519336Z inflating: build/lib/libtensorpipe_cuda.a 2024-04-03T16:23:03.8262281Z inflating: build/lib/libkineto.a 2024-04-03T16:23:03.8826675Z inflating: build/lib/libgloo_cuda.a 2024-04-03T16:23:03.8883044Z inflating: build/lib/libcaffe2_protos.a 2024-04-03T16:23:03.8943389Z inflating: build/lib/libonnx_proto.a 2024-04-03T16:23:03.9211562Z inflating: build/lib/libXNNPACK.a 2024-04-03T16:23:04.0194435Z inflating: build/lib/libonnx.a 2024-04-03T16:23:07.4267759Z inflating: build/lib/libtorch_cpu.so 2024-04-03T16:23:07.4273323Z inflating: build/lib/libshm.so 2024-04-03T16:23:07.4280336Z inflating: build/lib/libunbox_lib.a 2024-04-03T16:23:10.2584949Z inflating: build/lib/libtorch_cuda.so 2024-04-03T16:23:10.2586347Z inflating: build/lib/libtorch.so 2024-04-03T16:23:10.2590122Z inflating: build/lib/libc10d_cuda_test.so 2024-04-03T16:23:11.5887598Z inflating: build/lib/libtorch_cuda_linalg.so 2024-04-03T16:23:11.5917537Z inflating: build/lib/libjitbackend_test.so 2024-04-03T16:23:11.6000522Z inflating: build/lib/libtorchbind_test.so 2024-04-03T16:23:11.6037342Z inflating: build/lib/libbackend_with_compiler.so 2024-04-03T16:23:11.8768824Z inflating: build/lib/libtorch_python.so 2024-04-03T16:23:11.8818681Z inflating: build/lib/libnnapi_backend.so 2024-04-03T16:23:11.8819340Z creating: build/bin/ 2024-04-03T16:23:11.8892166Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-04-03T16:23:11.8967129Z inflating: build/bin/c10_DeviceGuard_test 2024-04-03T16:23:11.9040494Z inflating: build/bin/c10_Device_test 2024-04-03T16:23:11.9126687Z inflating: build/bin/c10_DispatchKeySet_test 2024-04-03T16:23:11.9198202Z inflating: build/bin/c10_StreamGuard_test 2024-04-03T16:23:11.9276160Z inflating: build/bin/c10_Scalar_test 2024-04-03T16:23:11.9350235Z inflating: build/bin/c10_SymInt_test 2024-04-03T16:23:11.9429423Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-04-03T16:23:11.9510963Z inflating: build/bin/c10_InlineStreamGuard_test 2024-04-03T16:23:11.9592866Z inflating: build/bin/c10_SizesAndStrides_test 2024-04-03T16:23:11.9696421Z inflating: build/bin/c10_cow_test 2024-04-03T16:23:11.9774302Z inflating: build/bin/c10_Bitset_test 2024-04-03T16:23:11.9845589Z inflating: build/bin/c10_ConstexprCrc_test 2024-04-03T16:23:11.9918236Z inflating: build/bin/c10_DeadlockDetection_test 2024-04-03T16:23:11.9992172Z inflating: build/bin/c10_Half_test 2024-04-03T16:23:12.0074080Z inflating: build/bin/c10_LeftRight_test 2024-04-03T16:23:12.0154357Z inflating: build/bin/c10_Metaprogramming_test 2024-04-03T16:23:12.0227367Z inflating: build/bin/c10_Synchronized_test 2024-04-03T16:23:12.0308709Z inflating: build/bin/c10_ThreadLocal_test 2024-04-03T16:23:12.0383643Z inflating: build/bin/c10_TypeIndex_test 2024-04-03T16:23:12.0457965Z inflating: build/bin/c10_TypeList_test 2024-04-03T16:23:12.0529532Z inflating: build/bin/c10_TypeTraits_test 2024-04-03T16:23:12.0605414Z inflating: build/bin/c10_accumulate_test 2024-04-03T16:23:12.0686609Z inflating: build/bin/c10_bfloat16_test 2024-04-03T16:23:12.0760099Z inflating: build/bin/c10_bit_cast_test 2024-04-03T16:23:12.0842373Z inflating: build/bin/c10_complex_math_test 2024-04-03T16:23:12.0923643Z inflating: build/bin/c10_complex_test 2024-04-03T16:23:12.1001161Z inflating: build/bin/c10_exception_test 2024-04-03T16:23:12.1074672Z inflating: build/bin/c10_flags_test 2024-04-03T16:23:12.1147635Z inflating: build/bin/c10_generic_math_test 2024-04-03T16:23:12.1221801Z inflating: build/bin/c10_irange_test 2024-04-03T16:23:12.1461990Z inflating: build/bin/c10_intrusive_ptr_test 2024-04-03T16:23:12.1543120Z inflating: build/bin/c10_logging_test 2024-04-03T16:23:12.1652421Z inflating: build/bin/c10_optional_test 2024-04-03T16:23:12.1743357Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-04-03T16:23:12.1821877Z inflating: build/bin/c10_registry_test 2024-04-03T16:23:12.2042435Z inflating: build/bin/c10_small_vector_test 2024-04-03T16:23:12.2117755Z inflating: build/bin/c10_ssize_test 2024-04-03T16:23:12.2193255Z inflating: build/bin/c10_string_util_test 2024-04-03T16:23:12.2266783Z inflating: build/bin/c10_tempfile_test 2024-04-03T16:23:12.2352528Z inflating: build/bin/c10_string_view_test 2024-04-03T16:23:12.2423242Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-04-03T16:23:12.2505010Z inflating: build/bin/c10_typeid_test 2024-04-03T16:23:12.3156000Z inflating: build/bin/protoc-3.13.0.0 2024-04-03T16:23:12.3806628Z inflating: build/bin/protoc 2024-04-03T16:23:12.3883967Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-04-03T16:23:12.3961331Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-04-03T16:23:12.4038291Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-04-03T16:23:12.4114065Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-04-03T16:23:12.4191169Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-04-03T16:23:12.4268431Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-04-03T16:23:12.4345072Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-04-03T16:23:12.4416780Z inflating: build/bin/c10_cuda_CUDATest 2024-04-03T16:23:12.4897030Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-04-03T16:23:12.5417558Z inflating: build/bin/vec_test_all_types_AVX2 2024-04-03T16:23:12.5918918Z inflating: build/bin/vec_test_all_types_AVX512 2024-04-03T16:23:12.5996397Z inflating: build/bin/FileStoreTest 2024-04-03T16:23:12.6077558Z inflating: build/bin/TCPStoreTest 2024-04-03T16:23:12.6155074Z inflating: build/bin/HashStoreTest 2024-04-03T16:23:12.6175182Z inflating: build/bin/ProcessGroupMPITest 2024-04-03T16:23:12.6181206Z inflating: build/bin/torch_shm_manager 2024-04-03T16:23:12.6259632Z inflating: build/bin/test_edge_op_registration 2024-04-03T16:23:12.6263739Z inflating: build/bin/example_allreduce 2024-04-03T16:23:12.6339590Z inflating: build/bin/Dimname_test 2024-04-03T16:23:12.6447437Z inflating: build/bin/Dict_test 2024-04-03T16:23:12.6541538Z inflating: build/bin/MaybeOwned_test 2024-04-03T16:23:12.6627888Z inflating: build/bin/apply_utils_test 2024-04-03T16:23:12.6711630Z inflating: build/bin/NamedTensor_test 2024-04-03T16:23:12.6797723Z inflating: build/bin/atest 2024-04-03T16:23:12.6889630Z inflating: build/bin/basic 2024-04-03T16:23:12.6969689Z inflating: build/bin/broadcast_test 2024-04-03T16:23:12.7043794Z inflating: build/bin/cpu_allocator_test 2024-04-03T16:23:12.7128846Z inflating: build/bin/cpu_generator_test 2024-04-03T16:23:12.7206405Z inflating: build/bin/cpu_profiling_allocator_test 2024-04-03T16:23:12.7341991Z inflating: build/bin/cpu_rng_test 2024-04-03T16:23:12.7416581Z inflating: build/bin/dispatch_key_set_test 2024-04-03T16:23:12.7489718Z inflating: build/bin/dlconvertor_test 2024-04-03T16:23:12.7574045Z inflating: build/bin/extension_backend_test 2024-04-03T16:23:12.7654559Z inflating: build/bin/half_test 2024-04-03T16:23:12.7794047Z inflating: build/bin/ivalue_test 2024-04-03T16:23:12.7866670Z inflating: build/bin/lazy_tensor_test 2024-04-03T16:23:12.7946164Z inflating: build/bin/math_kernel_test 2024-04-03T16:23:12.8025224Z inflating: build/bin/memory_format_test 2024-04-03T16:23:12.8102555Z inflating: build/bin/memory_overlapping_test 2024-04-03T16:23:12.8180116Z inflating: build/bin/mobile_memory_cleanup 2024-04-03T16:23:12.8253801Z inflating: build/bin/operator_name_test 2024-04-03T16:23:12.8335536Z inflating: build/bin/native_test 2024-04-03T16:23:12.8410899Z inflating: build/bin/operators_test 2024-04-03T16:23:12.8486450Z inflating: build/bin/packedtensoraccessor_test 2024-04-03T16:23:12.8584348Z inflating: build/bin/pow_test 2024-04-03T16:23:12.8668118Z inflating: build/bin/quantized_test 2024-04-03T16:23:12.8741863Z inflating: build/bin/reduce_ops_test 2024-04-03T16:23:12.8816508Z inflating: build/bin/reportMemoryUsage_test 2024-04-03T16:23:12.8898942Z inflating: build/bin/scalar_tensor_test 2024-04-03T16:23:12.8984466Z inflating: build/bin/scalar_test 2024-04-03T16:23:12.9059999Z inflating: build/bin/StorageUtils_test 2024-04-03T16:23:12.9136665Z inflating: build/bin/stride_properties_test 2024-04-03T16:23:12.9251839Z inflating: build/bin/tensor_iterator_test 2024-04-03T16:23:12.9333123Z inflating: build/bin/type_ptr_test 2024-04-03T16:23:12.9337325Z inflating: build/bin/thread_init_test 2024-04-03T16:23:12.9416361Z inflating: build/bin/test_parallel 2024-04-03T16:23:12.9492463Z inflating: build/bin/undefined_tensor_test 2024-04-03T16:23:12.9579873Z inflating: build/bin/type_test 2024-04-03T16:23:12.9582154Z inflating: build/bin/verify_api_visibility 2024-04-03T16:23:12.9683726Z inflating: build/bin/legacy_vmap_test 2024-04-03T16:23:12.9758985Z inflating: build/bin/weakref_test 2024-04-03T16:23:12.9834005Z inflating: build/bin/wrapdim_test 2024-04-03T16:23:12.9921841Z inflating: build/bin/IListRef_test 2024-04-03T16:23:12.9997525Z inflating: build/bin/xla_tensor_test 2024-04-03T16:23:13.0152105Z inflating: build/bin/List_test 2024-04-03T16:23:13.0329976Z inflating: build/bin/kernel_function_legacy_test 2024-04-03T16:23:13.0427189Z inflating: build/bin/KernelFunction_test 2024-04-03T16:23:13.0568017Z inflating: build/bin/kernel_function_test 2024-04-03T16:23:13.0755668Z inflating: build/bin/kernel_lambda_legacy_test 2024-04-03T16:23:13.0906846Z inflating: build/bin/kernel_lambda_test 2024-04-03T16:23:13.0996184Z inflating: build/bin/kernel_stackbased_test 2024-04-03T16:23:13.1136596Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-04-03T16:23:13.1211144Z inflating: build/bin/CppSignature_test 2024-04-03T16:23:13.1282823Z inflating: build/bin/op_allowlist_test 2024-04-03T16:23:13.1722428Z inflating: build/bin/op_registration_test 2024-04-03T16:23:13.1803998Z inflating: build/bin/backend_fallback_test 2024-04-03T16:23:13.1896699Z inflating: build/bin/inline_container_test 2024-04-03T16:23:13.1974074Z inflating: build/bin/cuda_apply_test 2024-04-03T16:23:13.2047876Z inflating: build/bin/cuda_allocator_test 2024-04-03T16:23:13.2134375Z inflating: build/bin/cuda_atomic_ops_test 2024-04-03T16:23:13.2214037Z inflating: build/bin/cuda_caching_host_allocator_test 2024-04-03T16:23:13.2316778Z inflating: build/bin/cuda_complex_math_test 2024-04-03T16:23:13.2388978Z inflating: build/bin/cuda_device_test 2024-04-03T16:23:13.2475267Z inflating: build/bin/cuda_complex_test 2024-04-03T16:23:13.2559940Z inflating: build/bin/cuda_cub_test 2024-04-03T16:23:13.2633961Z inflating: build/bin/cuda_dlconvertor_test 2024-04-03T16:23:13.2708649Z inflating: build/bin/cuda_integer_divider_test 2024-04-03T16:23:13.2804049Z inflating: build/bin/cuda_distributions_test 2024-04-03T16:23:13.2876736Z inflating: build/bin/cuda_half_test 2024-04-03T16:23:13.2959215Z inflating: build/bin/cuda_generator_test 2024-04-03T16:23:13.3030890Z inflating: build/bin/cuda_optional_test 2024-04-03T16:23:13.3103809Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-04-03T16:23:13.3180494Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-04-03T16:23:13.3268564Z inflating: build/bin/cuda_stream_test 2024-04-03T16:23:13.3343820Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-04-03T16:23:13.3415934Z inflating: build/bin/cuda_cudnn_test 2024-04-03T16:23:13.3492684Z inflating: build/bin/cuda_vectorized_test 2024-04-03T16:23:13.3513839Z inflating: build/bin/tutorial_tensorexpr 2024-04-03T16:23:13.3609804Z inflating: build/bin/ProcessGroupGlooTest 2024-04-03T16:23:13.3693166Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-04-03T16:23:13.3784998Z inflating: build/bin/ProcessGroupNCCLTest 2024-04-03T16:23:13.3862106Z inflating: build/bin/ProcessGroupUCCTest 2024-04-03T16:23:13.3952667Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-04-03T16:23:13.4033442Z inflating: build/bin/test_dist_autograd 2024-04-03T16:23:13.4037365Z inflating: build/bin/parallel_benchmark 2024-04-03T16:23:13.4138722Z inflating: build/bin/test_cpp_rpc 2024-04-03T16:23:13.4237437Z inflating: build/bin/test_mobile_nnc 2024-04-03T16:23:13.4250096Z inflating: build/bin/aot_model_compiler_test 2024-04-03T16:23:13.5471049Z inflating: build/bin/test_tensorexpr 2024-04-03T16:23:13.5975747Z inflating: build/bin/test_lazy 2024-04-03T16:23:13.6801259Z inflating: build/bin/test_jit 2024-04-03T16:23:13.8523493Z inflating: build/bin/test_api 2024-04-03T16:23:13.8524123Z creating: .additional_ci_files/ 2024-04-03T16:23:13.8576168Z inflating: .additional_ci_files/test-times.json 2024-04-03T16:23:13.8786380Z inflating: .additional_ci_files/test-class-times.json 2024-04-03T16:23:13.8855568Z ##[group]Run rm artifacts.zip 2024-04-03T16:23:13.8856124Z rm artifacts.zip 2024-04-03T16:23:13.8866390Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:23:13.8867111Z env: 2024-04-03T16:23:13.8867491Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:13.8868115Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:13.8868802Z ##[endgroup] 2024-04-03T16:23:14.0185289Z ##[group]Run df -H 2024-04-03T16:23:14.0185725Z df -H 2024-04-03T16:23:14.0195753Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:23:14.0196683Z env: 2024-04-03T16:23:14.0197075Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:14.0197707Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:14.0198412Z ##[endgroup] 2024-04-03T16:23:14.0237693Z Filesystem Size Used Avail Use% Mounted on 2024-04-03T16:23:14.0238596Z devtmpfs 129G 0 129G 0% /dev 2024-04-03T16:23:14.0239268Z tmpfs 129G 0 129G 0% /dev/shm 2024-04-03T16:23:14.0240884Z tmpfs 129G 590k 129G 1% /run 2024-04-03T16:23:14.0242189Z tmpfs 129G 0 129G 0% /sys/fs/cgroup 2024-04-03T16:23:14.0242912Z /dev/xvda1 162G 37G 125G 23% / 2024-04-03T16:23:14.0243552Z tmpfs 26G 0 26G 0% /run/user/0 2024-04-03T16:23:14.0301049Z Prepare all required actions 2024-04-03T16:23:14.0301723Z Getting action download info 2024-04-03T16:23:14.2651483Z ##[group]Run ./.github/actions/download-td-artifacts 2024-04-03T16:23:14.2652138Z with: 2024-04-03T16:23:14.2652496Z env: 2024-04-03T16:23:14.2652902Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:14.2653534Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:14.2654232Z ##[endgroup] 2024-04-03T16:23:14.2723712Z ##[group]Run seemethere/download-artifact-s3@v4 2024-04-03T16:23:14.2724352Z with: 2024-04-03T16:23:14.2724711Z name: td_results 2024-04-03T16:23:14.2725157Z s3-bucket: gha-artifacts 2024-04-03T16:23:14.2725651Z region: us-east-1 2024-04-03T16:23:14.2726050Z env: 2024-04-03T16:23:14.2726426Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:14.2727064Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:14.2727736Z ##[endgroup] 2024-04-03T16:23:14.9545029Z (node:90270) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-04-03T16:23:14.9547115Z 2024-04-03T16:23:14.9547889Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-04-03T16:23:14.9549972Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-04-03T16:23:14.9552419Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-04-03T16:23:15.0598131Z Found 1 objects with prefix pytorch/pytorch/8536950919/td_results/ 2024-04-03T16:23:15.0599559Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-04-03T16:23:15.1265345Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-04-03T16:23:15.1277279Z Artifact download has finished successfully 2024-04-03T16:23:15.1477713Z ##[group]Run mkdir -p .additional_ci_files 2024-04-03T16:23:15.1478401Z mkdir -p .additional_ci_files 2024-04-03T16:23:15.1479228Z mv td_results.json .additional_ci_files/td_results.json 2024-04-03T16:23:15.1489061Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:23:15.1489761Z env: 2024-04-03T16:23:15.1490148Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:15.1490794Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:15.1491514Z ##[endgroup] 2024-04-03T16:23:15.1588388Z ##[group]Run .github/scripts/parse_ref.py 2024-04-03T16:23:15.1589115Z .github/scripts/parse_ref.py 2024-04-03T16:23:15.1598809Z shell: /usr/bin/bash -e {0} 2024-04-03T16:23:15.1599325Z env: 2024-04-03T16:23:15.1599712Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:15.1600347Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:15.1601042Z ##[endgroup] 2024-04-03T16:23:15.1945586Z Prepare all required actions 2024-04-03T16:23:15.2004182Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-04-03T16:23:15.2004840Z with: 2024-04-03T16:23:15.2005934Z github-token: *** 2024-04-03T16:23:15.2006383Z env: 2024-04-03T16:23:15.2006768Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:15.2007418Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:15.2008095Z ##[endgroup] 2024-04-03T16:23:15.2033316Z ##[group]Run set -eux 2024-04-03T16:23:15.2033793Z set -eux 2024-04-03T16:23:15.2034916Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-04-03T16:23:15.2044408Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:23:15.2045109Z env: 2024-04-03T16:23:15.2045497Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:15.2046142Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:15.2047070Z GITHUB_TOKEN: *** 2024-04-03T16:23:15.2047497Z ##[endgroup] 2024-04-03T16:23:15.2075455Z + python3 .github/scripts/get_workflow_job_id.py 8536950919 i-00c317a4521719f93 2024-04-03T16:23:25.3541535Z setting job-id=23402706124 2024-04-03T16:23:25.3542820Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:23:25.3826618Z Prepare all required actions 2024-04-03T16:23:25.3827288Z Getting action download info 2024-04-03T16:23:25.4957763Z ##[group]Run ./.github/actions/filter-test-configs 2024-04-03T16:23:25.4958404Z with: 2024-04-03T16:23:25.4959056Z github-token: *** 2024-04-03T16:23:25.4961195Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-04-03T16:23:25.4963899Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:23:25.4964935Z env: 2024-04-03T16:23:25.4965303Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:25.4965947Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:25.4966636Z ##[endgroup] 2024-04-03T16:23:25.5029643Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-04-03T16:23:25.5030441Z with: 2024-04-03T16:23:25.5030821Z shell: bash 2024-04-03T16:23:25.5031247Z timeout_minutes: 10 2024-04-03T16:23:25.5031694Z max_attempts: 5 2024-04-03T16:23:25.5032138Z retry_wait_seconds: 30 2024-04-03T16:23:25.5033273Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-04-03T16:23:25.5034841Z polling_interval_seconds: 1 2024-04-03T16:23:25.5035365Z warning_on_retry: true 2024-04-03T16:23:25.5035859Z continue_on_error: false 2024-04-03T16:23:25.5036323Z env: 2024-04-03T16:23:25.5036708Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:25.5037352Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:25.5038329Z GITHUB_TOKEN: *** 2024-04-03T16:23:25.5038763Z ##[endgroup] 2024-04-03T16:23:25.5859050Z + python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-04-03T16:23:25.9095521Z Defaulting to user installation because normal site-packages is not writeable 2024-04-03T16:23:26.0715890Z Collecting requests==2.26.0 2024-04-03T16:23:26.0900351Z Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB) 2024-04-03T16:23:26.2901749Z Collecting pyyaml==6.0.1 2024-04-03T16:23:26.2948736Z Downloading PyYAML-6.0.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (670 kB) 2024-04-03T16:23:26.3734469Z Collecting idna<4,>=2.5; python_version >= "3" 2024-04-03T16:23:26.3779118Z Downloading idna-3.6-py3-none-any.whl (61 kB) 2024-04-03T16:23:26.4592881Z Collecting certifi>=2017.4.17 2024-04-03T16:23:26.4640797Z Downloading certifi-2024.2.2-py3-none-any.whl (163 kB) 2024-04-03T16:23:26.8635284Z Collecting charset-normalizer~=2.0.0; python_version >= "3" 2024-04-03T16:23:26.8684505Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-04-03T16:23:27.0200932Z Collecting urllib3<1.27,>=1.21.1 2024-04-03T16:23:27.0246861Z Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 2024-04-03T16:23:27.1534427Z Installing collected packages: idna, certifi, charset-normalizer, urllib3, requests, pyyaml 2024-04-03T16:23:27.2917231Z WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2024-04-03T16:23:27.2919014Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-04-03T16:23:27.5084505Z Successfully installed certifi-2024.2.2 charset-normalizer-2.0.12 idna-3.6 pyyaml-6.0.1 requests-2.26.0 urllib3-1.26.18 2024-04-03T16:23:27.5850915Z Command completed after 1 attempt(s). 2024-04-03T16:23:27.5912592Z ##[group]Run set -x 2024-04-03T16:23:27.5913068Z set -x 2024-04-03T16:23:27.5913465Z  2024-04-03T16:23:27.5914690Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-04-03T16:23:27.5915741Z # in runner workspace 2024-04-03T16:23:27.5916489Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-04-03T16:23:27.5926615Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:23:27.5927362Z env: 2024-04-03T16:23:27.5927756Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:27.5928382Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:27.5929074Z ##[endgroup] 2024-04-03T16:23:27.5960577Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-04-03T16:23:27.6265377Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-04-03T16:23:27.6266147Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-04-03T16:23:27.6266803Z echo "Job name: ${JOB_NAME}" 2024-04-03T16:23:27.6267367Z  2024-04-03T16:23:27.6268125Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-04-03T16:23:27.6269102Z # in runner workspace 2024-04-03T16:23:27.6269928Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-04-03T16:23:27.6270864Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-04-03T16:23:27.6271498Z  --job-name "${JOB_NAME}" \ 2024-04-03T16:23:27.6273807Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" \ 2024-04-03T16:23:27.6276437Z  --pr-number "${PR_NUMBER}" \ 2024-04-03T16:23:27.6277046Z  --tag "${TAG}" \ 2024-04-03T16:23:27.6277582Z  --event-name "${EVENT_NAME}" \ 2024-04-03T16:23:27.6278221Z  --schedule "${SCHEDULE}" \ 2024-04-03T16:23:27.6278824Z  --branch "${HEAD_BRANCH}" 2024-04-03T16:23:27.6287967Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:23:27.6288660Z env: 2024-04-03T16:23:27.6289056Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:27.6289712Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:27.6290663Z GITHUB_TOKEN: *** 2024-04-03T16:23:27.6291603Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:23:27.6292652Z PR_NUMBER: 123154 2024-04-03T16:23:27.6293072Z TAG: 2024-04-03T16:23:27.6293465Z EVENT_NAME: pull_request 2024-04-03T16:23:27.6293952Z SCHEDULE: 2024-04-03T16:23:27.6294335Z HEAD_BRANCH: 2024-04-03T16:23:27.6294757Z ##[endgroup] 2024-04-03T16:23:27.6322457Z Workflow: pull 2024-04-03T16:23:27.6323722Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:23:27.8684816Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-04-03T16:23:27.9901745Z fatal: unknown commit main 2024-04-03T16:23:27.9931634Z /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/filter_test_configs.py:458: UserWarning: failed to get commit messages: Command '['git', 'cherry', '-v', 'main']' returned non-zero exit status 128. 2024-04-03T16:23:27.9933956Z warnings.warn(f"failed to get commit messages: {e}") 2024-04-03T16:23:28.0086026Z ##[group]Run echo "Filtered matrix:" 2024-04-03T16:23:28.0086663Z echo "Filtered matrix:" 2024-04-03T16:23:28.0089098Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" 2024-04-03T16:23:28.0091297Z  2024-04-03T16:23:28.0091665Z echo 2024-04-03T16:23:28.0092204Z echo "Is the current job unstable? False" 2024-04-03T16:23:28.0092867Z  2024-04-03T16:23:28.0093228Z echo 2024-04-03T16:23:28.0093741Z echo "Is keep-going label set? False" 2024-04-03T16:23:28.0094375Z  2024-04-03T16:23:28.0094900Z echo 2024-04-03T16:23:28.0095370Z echo "Renabled issues? " 2024-04-03T16:23:28.0104476Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:23:28.0105198Z env: 2024-04-03T16:23:28.0105575Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:28.0106242Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:28.0106943Z ##[endgroup] 2024-04-03T16:23:28.0134942Z Filtered matrix: 2024-04-03T16:23:28.0136965Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}]} 2024-04-03T16:23:28.0139213Z 2024-04-03T16:23:28.0139457Z Is the current job unstable? False 2024-04-03T16:23:28.0139869Z 2024-04-03T16:23:28.0140193Z Is keep-going label set? False 2024-04-03T16:23:28.0140567Z 2024-04-03T16:23:28.0140739Z Renabled issues? 2024-04-03T16:23:28.0202222Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-04-03T16:23:28.0203280Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-04-03T16:23:28.0212438Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:23:28.0213158Z env: 2024-04-03T16:23:28.0213545Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:28.0214196Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:28.0214900Z JOB_TIMEOUT: 360 2024-04-03T16:23:28.0215316Z ##[endgroup] 2024-04-03T16:23:28.0335878Z ##[group]Run set -x 2024-04-03T16:23:28.0336453Z set -x 2024-04-03T16:23:28.0336880Z  2024-04-03T16:23:28.0337385Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-04-03T16:23:28.0338221Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-04-03T16:23:28.0339034Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-04-03T16:23:28.0339769Z  TEST_COMMAND=.ci/onnx/test.sh 2024-04-03T16:23:28.0340356Z else 2024-04-03T16:23:28.0340829Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-04-03T16:23:28.0341437Z fi 2024-04-03T16:23:28.0341827Z  2024-04-03T16:23:28.0342482Z # detached container should get cleaned up by teardown_ec2_linux 2024-04-03T16:23:28.0343604Z # TODO: Stop building test binaries as part of the build phase 2024-04-03T16:23:28.0344592Z # Used for GPU_FLAG since that doesn't play nice 2024-04-03T16:23:28.0345426Z # shellcheck disable=SC2086,SC2090 2024-04-03T16:23:28.0346083Z container_name=$(docker run \ 2024-04-03T16:23:28.0346685Z  ${GPU_FLAG:-} \ 2024-04-03T16:23:28.0347219Z  -e BUILD_ENVIRONMENT \ 2024-04-03T16:23:28.0347774Z  -e PR_NUMBER \ 2024-04-03T16:23:28.0348289Z  -e GITHUB_ACTIONS \ 2024-04-03T16:23:28.0348854Z  -e GITHUB_REPOSITORY \ 2024-04-03T16:23:28.0349417Z  -e GITHUB_WORKFLOW \ 2024-04-03T16:23:28.0349969Z  -e GITHUB_JOB \ 2024-04-03T16:23:28.0350485Z  -e GITHUB_RUN_ID \ 2024-04-03T16:23:28.0351015Z  -e GITHUB_RUN_NUMBER \ 2024-04-03T16:23:28.0351596Z  -e GITHUB_RUN_ATTEMPT \ 2024-04-03T16:23:28.0352167Z  -e JOB_ID \ 2024-04-03T16:23:28.0352632Z  -e JOB_NAME \ 2024-04-03T16:23:28.0353344Z  -e BASE_SHA \ 2024-04-03T16:23:28.0353836Z  -e BRANCH \ 2024-04-03T16:23:28.0354611Z  -e SHA1 \ 2024-04-03T16:23:28.0355117Z  -e AWS_DEFAULT_REGION \ 2024-04-03T16:23:28.0355698Z  -e IN_WHEEL_TEST \ 2024-04-03T16:23:28.0356220Z  -e SHARD_NUMBER \ 2024-04-03T16:23:28.0356750Z  -e TEST_CONFIG \ 2024-04-03T16:23:28.0357282Z  -e NUM_TEST_SHARDS \ 2024-04-03T16:23:28.0357824Z  -e REENABLED_ISSUES \ 2024-04-03T16:23:28.0358421Z  -e CONTINUE_THROUGH_ERROR \ 2024-04-03T16:23:28.0359032Z  -e VERBOSE_TEST_LOGS \ 2024-04-03T16:23:28.0359587Z  -e NO_TEST_TIMEOUT \ 2024-04-03T16:23:28.0360127Z  -e NO_TD \ 2024-04-03T16:23:28.0360613Z  -e PR_LABELS \ 2024-04-03T16:23:28.0361177Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-04-03T16:23:28.0361835Z  -e SCCACHE_BUCKET \ 2024-04-03T16:23:28.0362415Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-04-03T16:23:28.0362985Z  -e XLA_CUDA \ 2024-04-03T16:23:28.0363561Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-04-03T16:23:28.0364307Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-04-03T16:23:28.0365061Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-04-03T16:23:28.0365784Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-04-03T16:23:28.0366459Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-04-03T16:23:28.0367060Z  -e DASHBOARD_TAG \ 2024-04-03T16:23:28.0367727Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-04-03T16:23:28.0368504Z  --ulimit stack=10485760:83886080 \ 2024-04-03T16:23:28.0369205Z  --security-opt seccomp=unconfined \ 2024-04-03T16:23:28.0369852Z  --cap-add=SYS_PTRACE \ 2024-04-03T16:23:28.0370418Z  --ipc=host \ 2024-04-03T16:23:28.0370931Z  --shm-size="${SHM_SIZE}" \ 2024-04-03T16:23:28.0371483Z  --tty \ 2024-04-03T16:23:28.0371944Z  --detach \ 2024-04-03T16:23:28.0372457Z  --name="${container_name}" \ 2024-04-03T16:23:28.0373040Z  --user jenkins \ 2024-04-03T16:23:28.0373911Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-04-03T16:23:28.0374739Z  -w /var/lib/jenkins/workspace \ 2024-04-03T16:23:28.0375347Z  "${DOCKER_IMAGE}" 2024-04-03T16:23:28.0375852Z ) 2024-04-03T16:23:28.0376433Z # Propagate download.pytorch.org IP to container 2024-04-03T16:23:28.0377764Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-04-03T16:23:28.0379183Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-04-03T16:23:28.0380515Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-04-03T16:23:28.0389847Z shell: /usr/bin/bash -e {0} 2024-04-03T16:23:28.0390365Z env: 2024-04-03T16:23:28.0390740Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:28.0391392Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:28.0392242Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-04-03T16:23:28.0392920Z PR_NUMBER: 123154 2024-04-03T16:23:28.0393409Z GITHUB_REPOSITORY: pytorch/pytorch 2024-04-03T16:23:28.0394002Z GITHUB_WORKFLOW: pull 2024-04-03T16:23:28.0394705Z GITHUB_JOB: test 2024-04-03T16:23:28.0395155Z GITHUB_RUN_ID: 8536950919 2024-04-03T16:23:28.0395666Z GITHUB_RUN_NUMBER: 192590 2024-04-03T16:23:28.0396156Z GITHUB_RUN_ATTEMPT: 2 2024-04-03T16:23:28.0396625Z JOB_ID: 23402706124 2024-04-03T16:23:28.0397561Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:23:28.0398603Z BRANCH: pull/123154 2024-04-03T16:23:28.0399156Z SHA1: d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:23:28.0399904Z BASE_SHA: 1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:23:28.0400726Z TEST_CONFIG: distributed 2024-04-03T16:23:28.0401228Z SHARD_NUMBER: 2 2024-04-03T16:23:28.0401665Z NUM_TEST_SHARDS: 3 2024-04-03T16:23:28.0402108Z REENABLED_ISSUES: 2024-04-03T16:23:28.0402598Z CONTINUE_THROUGH_ERROR: False 2024-04-03T16:23:28.0403153Z VERBOSE_TEST_LOGS: False 2024-04-03T16:23:28.0403645Z NO_TEST_TIMEOUT: False 2024-04-03T16:23:28.0404122Z NO_TD: False 2024-04-03T16:23:28.0404670Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-04-03T16:23:28.0405356Z SCCACHE_S3_KEY_PREFIX: pull 2024-04-03T16:23:28.0405880Z SHM_SIZE: 2g 2024-04-03T16:23:28.0407250Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:23:28.0408752Z XLA_CUDA: 2024-04-03T16:23:28.0409428Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-04-03T16:23:28.0410317Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-04-03T16:23:28.0410940Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-04-03T16:23:28.0411517Z DASHBOARD_TAG: 2024-04-03T16:23:28.0411969Z HUGGING_FACE_HUB_TOKEN: 2024-04-03T16:23:28.0412465Z ##[endgroup] 2024-04-03T16:23:28.0440853Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-04-03T16:23:28.0441728Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-04-03T16:23:28.0442443Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-04-03T16:23:28.0452196Z +++ nproc --ignore=2 2024-04-03T16:23:28.0489728Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_8536950919 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:23:39.1741454Z + container_name=7033b537529c546187ef7c776250a79cad9cc38d4987347f20cdbf86b60e2c90 2024-04-03T16:23:39.1743164Z + grep download.pytorch.org /etc/hosts 2024-04-03T16:23:39.1745444Z + docker exec -i 7033b537529c546187ef7c776250a79cad9cc38d4987347f20cdbf86b60e2c90 sudo bash -c '/bin/cat >> /etc/hosts' 2024-04-03T16:23:39.2635949Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-04-03T16:23:39.2692868Z + echo DOCKER_CONTAINER_ID=7033b537529c546187ef7c776250a79cad9cc38d4987347f20cdbf86b60e2c90 2024-04-03T16:23:39.2697680Z ++ echo dist/torch-2.4.0a0+gitd253bd7-cp310-cp310-linux_x86_64.whl 2024-04-03T16:23:39.2700613Z + docker exec -t 7033b537529c546187ef7c776250a79cad9cc38d4987347f20cdbf86b60e2c90 sh -c 'pip install dist/torch-2.4.0a0+gitd253bd7-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-04-03T16:23:39.8831314Z Processing ./dist/torch-2.4.0a0+gitd253bd7-cp310-cp310-linux_x86_64.whl 2024-04-03T16:23:41.1711620Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (3.13.1) 2024-04-03T16:23:41.1722097Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (4.10.0) 2024-04-03T16:23:41.1727001Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (1.12) 2024-04-03T16:23:41.1732328Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (2.8.8) 2024-04-03T16:23:41.1737369Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (3.1.3) 2024-04-03T16:23:41.1742553Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (2024.2.0) 2024-04-03T16:23:41.1758599Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (3.3.0) 2024-04-03T16:23:41.1852509Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.4.0a0+gitd253bd7) (1.21.2) 2024-04-03T16:23:41.2538280Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.4.0a0+gitd253bd7) (2.1.5) 2024-04-03T16:23:41.2793548Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.4.0a0+gitd253bd7) (1.3.0) 2024-04-03T16:23:42.4257824Z Installing collected packages: torch 2024-04-03T16:23:56.2835973Z Successfully installed torch-2.4.0a0+gitd253bd7 2024-04-03T16:23:56.3587516Z + echo 'Environment variables:' 2024-04-03T16:23:56.3588550Z Environment variables: 2024-04-03T16:23:56.3589037Z + env 2024-04-03T16:23:56.3596647Z INSTALLED_DB=yes 2024-04-03T16:23:56.3597298Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-04-03T16:23:56.3597858Z NVIDIA_VISIBLE_DEVICES=all 2024-04-03T16:23:56.3598418Z NV_NVML_DEV_VERSION=11.8.86-1 2024-04-03T16:23:56.3599302Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:23:56.3600147Z CONTINUE_THROUGH_ERROR=False 2024-04-03T16:23:56.3601215Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-04-03T16:23:56.3602020Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-04-03T16:23:56.3602835Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-04-03T16:23:56.3603531Z HOSTNAME=7033b537529c 2024-04-03T16:23:56.3605170Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_cc00ac8f-955d-4f0f-bada-1fb2a36d2255 2024-04-03T16:23:56.3606412Z GITHUB_ACTION=__self 2024-04-03T16:23:56.3606916Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-04-03T16:23:56.3610073Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-04-03T16:23:56.3613507Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-04-03T16:23:56.3614252Z NV_NVTX_VERSION=11.8.86-1 2024-04-03T16:23:56.3614745Z GITHUB_RUN_NUMBER=192590 2024-04-03T16:23:56.3615257Z TEST_CONFIG=distributed 2024-04-03T16:23:56.3615772Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-04-03T16:23:56.3616432Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-04-03T16:23:56.3617104Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-04-03T16:23:56.3617748Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-04-03T16:23:56.3618338Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-04-03T16:23:56.3618893Z GITHUB_TRIGGERING_ACTOR=soulitzer 2024-04-03T16:23:56.3619558Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-04-03T16:23:56.3620300Z GITHUB_REF_TYPE=branch 2024-04-03T16:23:56.3620775Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-04-03T16:23:56.3621320Z NCCL_VERSION=2.16.2-1 2024-04-03T16:23:56.3621878Z BASE_SHA=1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:23:56.3622493Z XLA_CUDA= 2024-04-03T16:23:56.3622896Z HUGGING_FACE_HUB_TOKEN= 2024-04-03T16:23:56.3625484Z *** 2024-04-03T16:23:56.3625891Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-04-03T16:23:56.3626462Z GITHUB_REPOSITORY_ID=65600975 2024-04-03T16:23:56.3627239Z GITHUB_ACTIONS=true 2024-04-03T16:23:56.3627697Z NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:56.3628430Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-04-03T16:23:56.3629210Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-04-03T16:23:56.3629878Z SHA1=d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:23:56.3630617Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-04-03T16:23:56.3631333Z GITHUB_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:23:56.3632364Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/123154/merge 2024-04-03T16:23:56.3633290Z UCC_HOME=/usr 2024-04-03T16:23:56.3633771Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-04-03T16:23:56.3634838Z VERBOSE_TEST_LOGS=False 2024-04-03T16:23:56.3635337Z NVIDIA_PRODUCT_NAME=CUDA 2024-04-03T16:23:56.3636155Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-04-03T16:23:56.3636829Z GITHUB_REF=refs/pull/123154/merge 2024-04-03T16:23:56.3637432Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-04-03T16:23:56.3638026Z SHARD_NUMBER=2 2024-04-03T16:23:56.3638451Z GITHUB_REF_PROTECTED=false 2024-04-03T16:23:56.3638954Z HOME=/var/lib/jenkins 2024-04-03T16:23:56.3639474Z GITHUB_API_URL=https://api.github.com 2024-04-03T16:23:56.3640100Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-04-03T16:23:56.3640782Z UCX_COMMIT=00bcc6bb18fc282eb160623b4c0d300147f579af 2024-04-03T16:23:56.3641470Z SCCACHE_S3_KEY_PREFIX=pull 2024-04-03T16:23:56.3641955Z CUDA_VERSION=11.8.0 2024-04-03T16:23:56.3642571Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-04-03T16:23:56.3643212Z NUM_TEST_SHARDS=3 2024-04-03T16:23:56.3643627Z UCX_HOME=/usr 2024-04-03T16:23:56.3644373Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-04-03T16:23:56.3645979Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_cc00ac8f-955d-4f0f-bada-1fb2a36d2255 2024-04-03T16:23:56.3647772Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:23:56.3649594Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_cc00ac8f-955d-4f0f-bada-1fb2a36d2255 2024-04-03T16:23:56.3651420Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-04-03T16:23:56.3652417Z GITHUB_EVENT_NAME=pull_request 2024-04-03T16:23:56.3652931Z DASHBOARD_TAG= 2024-04-03T16:23:56.3653363Z GITHUB_RUN_ID=8536950919 2024-04-03T16:23:56.3654026Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-04-03T16:23:56.3654787Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-04-03T16:23:56.3656267Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_cc00ac8f-955d-4f0f-bada-1fb2a36d2255 2024-04-03T16:23:56.3657556Z GITHUB_ACTOR=caic99 2024-04-03T16:23:56.3658051Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-04-03T16:23:56.3658601Z PR_NUMBER=123154 2024-04-03T16:23:56.3659040Z GITHUB_RUN_ATTEMPT=2 2024-04-03T16:23:56.3659500Z ANACONDA_PYTHON_VERSION=3.10 2024-04-03T16:23:56.3660137Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-04-03T16:23:56.3660813Z TERM=xterm 2024-04-03T16:23:56.3661267Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-04-03T16:23:56.3661858Z INSTALLED_VISION=yes 2024-04-03T16:23:56.3662324Z BRANCH=pull/123154 2024-04-03T16:23:56.3662771Z OPENSSL_ROOT_DIR=/opt/openssl 2024-04-03T16:23:56.3663342Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-04-03T16:23:56.3663947Z CUDA_PATH=/usr/local/cuda 2024-04-03T16:23:56.3665029Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-04-03T16:23:56.3666131Z GITHUB_SERVER_URL=https://github.com 2024-04-03T16:23:56.3666835Z UCC_COMMIT=7cb07a76ccedad7e56ceb136b865eb9319c258ea 2024-04-03T16:23:56.3667499Z REENABLED_ISSUES= 2024-04-03T16:23:56.3667920Z SHLVL=1 2024-04-03T16:23:56.3668286Z MAX_JOBS=30 2024-04-03T16:23:56.3668713Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-04-03T16:23:56.3669218Z NVARCH=x86_64 2024-04-03T16:23:56.3669641Z GITHUB_ACTOR_ID=78061359 2024-04-03T16:23:56.3670289Z GITHUB_WORKFLOW_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:23:56.3671208Z GITHUB_REF_NAME=123154/merge 2024-04-03T16:23:56.3671831Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-04-03T16:23:56.3672789Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-04-03T16:23:56.3673627Z GITHUB_JOB=test 2024-04-03T16:23:56.3674580Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-04-03T16:23:56.3675386Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-04-03T16:23:56.3676129Z NO_TEST_TIMEOUT=False 2024-04-03T16:23:56.3676680Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-04-03T16:23:56.3677286Z GITHUB_REPOSITORY=pytorch/pytorch 2024-04-03T16:23:56.3677884Z NV_NVPROF_VERSION=11.8.87-1 2024-04-03T16:23:56.3678404Z GITHUB_RETENTION_DAYS=90 2024-04-03T16:23:56.3678886Z OPENSSL_DIR=/opt/openssl 2024-04-03T16:23:56.3679390Z GITHUB_ACTION_REPOSITORY= 2024-04-03T16:23:56.3680924Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:23:56.3682506Z GITHUB_BASE_REF=main 2024-04-03T16:23:56.3682995Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-04-03T16:23:56.3683518Z CI=true 2024-04-03T16:23:56.3683958Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-04-03T16:23:56.3684556Z GITHUB_REPOSITORY_OWNER=pytorch 2024-04-03T16:23:56.3685091Z JOB_ID=23402706124 2024-04-03T16:23:56.3685518Z INSTALLED_PROTOBUF=yes 2024-04-03T16:23:56.3686026Z GITHUB_HEAD_REF=patch-1 2024-04-03T16:23:56.3686506Z GITHUB_ACTION_REF= 2024-04-03T16:23:56.3687110Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-04-03T16:23:56.3687790Z GITHUB_WORKFLOW=pull 2024-04-03T16:23:56.3688275Z DEBIAN_FRONTEND=noninteractive 2024-04-03T16:23:56.3689605Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_cc00ac8f-955d-4f0f-bada-1fb2a36d2255 2024-04-03T16:23:56.3690823Z NO_TD=False 2024-04-03T16:23:56.3691253Z SKIP_SCCACHE_INITIALIZATION=1 2024-04-03T16:23:56.3691754Z _=/usr/bin/env 2024-04-03T16:23:56.3692451Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-04-03T16:23:56.3845275Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-04-03T16:23:56.3847565Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-04-03T16:23:56.3849036Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-04-03T16:23:56.3850332Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-04-03T16:23:56.3851447Z + BUILD_DIR=build 2024-04-03T16:23:56.3852144Z + BUILD_RENAMED_DIR=build_renamed 2024-04-03T16:23:56.3852717Z + BUILD_BIN_DIR=build/bin 2024-04-03T16:23:56.3853203Z + SHARD_NUMBER=2 2024-04-03T16:23:56.3853617Z + NUM_TEST_SHARDS=3 2024-04-03T16:23:56.3854070Z + export VALGRIND=ON 2024-04-03T16:23:56.3854520Z + VALGRIND=ON 2024-04-03T16:23:56.3855146Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-04-03T16:23:56.3855825Z + [[ 0 == \1 ]] 2024-04-03T16:23:56.3856256Z + [[ False == \1 ]] 2024-04-03T16:23:56.3856865Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-04-03T16:23:56.3857592Z ++ realpath build/custom_test_artifacts 2024-04-03T16:23:56.3863222Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-04-03T16:23:56.3864208Z + [[ -n '' ]] 2024-04-03T16:23:56.3867107Z ++ dirname .ci/pytorch/test.sh 2024-04-03T16:23:56.3875908Z + source .ci/pytorch/common.sh 2024-04-03T16:23:56.3880050Z +++ dirname .ci/pytorch/common.sh 2024-04-03T16:23:56.3888527Z ++ source .ci/pytorch/common_utils.sh 2024-04-03T16:23:56.3891172Z +++ declare -f -t trap_add 2024-04-03T16:23:56.3898972Z ++ set -ex 2024-04-03T16:23:56.3899575Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-04-03T16:23:56.3900270Z ++ BUILD_TEST_LIBTORCH=0 2024-04-03T16:23:56.3900837Z + echo 'Environment variables' 2024-04-03T16:23:56.3901361Z Environment variables 2024-04-03T16:23:56.3901809Z + env 2024-04-03T16:23:56.3910092Z INSTALLED_DB=yes 2024-04-03T16:23:56.3911155Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-04-03T16:23:56.3911808Z NVIDIA_VISIBLE_DEVICES=all 2024-04-03T16:23:56.3912880Z NV_NVML_DEV_VERSION=11.8.86-1 2024-04-03T16:23:56.3914611Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:23:56.3915488Z CONTINUE_THROUGH_ERROR=False 2024-04-03T16:23:56.3916506Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-04-03T16:23:56.3917245Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-04-03T16:23:56.3918033Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-04-03T16:23:56.3918720Z HOSTNAME=7033b537529c 2024-04-03T16:23:56.3920552Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_cc00ac8f-955d-4f0f-bada-1fb2a36d2255 2024-04-03T16:23:56.3922374Z GITHUB_ACTION=__self 2024-04-03T16:23:56.3923086Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-04-03T16:23:56.3927343Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-04-03T16:23:56.3930673Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-04-03T16:23:56.3931415Z NV_NVTX_VERSION=11.8.86-1 2024-04-03T16:23:56.3931922Z GITHUB_RUN_NUMBER=192590 2024-04-03T16:23:56.3932398Z TEST_CONFIG=distributed 2024-04-03T16:23:56.3932913Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-04-03T16:23:56.3933590Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-04-03T16:23:56.3934238Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-04-03T16:23:56.3934864Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-04-03T16:23:56.3935471Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-04-03T16:23:56.3935999Z GITHUB_TRIGGERING_ACTOR=soulitzer 2024-04-03T16:23:56.3936712Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-04-03T16:23:56.3937390Z GITHUB_REF_TYPE=branch 2024-04-03T16:23:56.3937935Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-04-03T16:23:56.3938463Z NCCL_VERSION=2.16.2-1 2024-04-03T16:23:56.3939018Z BASE_SHA=1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:23:56.3939843Z XLA_CUDA= 2024-04-03T16:23:56.3940251Z HUGGING_FACE_HUB_TOKEN= 2024-04-03T16:23:56.3940826Z *** 2024-04-03T16:23:56.3941220Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-04-03T16:23:56.3941766Z GITHUB_REPOSITORY_ID=65600975 2024-04-03T16:23:56.3942285Z GITHUB_ACTIONS=true 2024-04-03T16:23:56.3942759Z NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:56.3943449Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-04-03T16:23:56.3944222Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-04-03T16:23:56.3944902Z SHA1=d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:23:56.3945621Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-04-03T16:23:56.3946330Z GITHUB_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:23:56.3947376Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/123154/merge 2024-04-03T16:23:56.3948293Z UCC_HOME=/usr 2024-04-03T16:23:56.3948776Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-04-03T16:23:56.3949358Z VERBOSE_TEST_LOGS=False 2024-04-03T16:23:56.3949839Z NVIDIA_PRODUCT_NAME=CUDA 2024-04-03T16:23:56.3950500Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-04-03T16:23:56.3951178Z GITHUB_REF=refs/pull/123154/merge 2024-04-03T16:23:56.3951754Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-04-03T16:23:56.3952288Z SHARD_NUMBER=2 2024-04-03T16:23:56.3952725Z GITHUB_REF_PROTECTED=false 2024-04-03T16:23:56.3953205Z HOME=/var/lib/jenkins 2024-04-03T16:23:56.3953719Z GITHUB_API_URL=https://api.github.com 2024-04-03T16:23:56.3954598Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-04-03T16:23:56.3955267Z UCX_COMMIT=00bcc6bb18fc282eb160623b4c0d300147f579af 2024-04-03T16:23:56.3955956Z SCCACHE_S3_KEY_PREFIX=pull 2024-04-03T16:23:56.3956458Z CUDA_VERSION=11.8.0 2024-04-03T16:23:56.3957048Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-04-03T16:23:56.3957860Z NUM_TEST_SHARDS=3 2024-04-03T16:23:56.3958287Z UCX_HOME=/usr 2024-04-03T16:23:56.3959016Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-04-03T16:23:56.3960621Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_cc00ac8f-955d-4f0f-bada-1fb2a36d2255 2024-04-03T16:23:56.3962430Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:23:56.3964229Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_cc00ac8f-955d-4f0f-bada-1fb2a36d2255 2024-04-03T16:23:56.3965871Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-04-03T16:23:56.3966846Z GITHUB_EVENT_NAME=pull_request 2024-04-03T16:23:56.3967372Z DASHBOARD_TAG= 2024-04-03T16:23:56.3967781Z GITHUB_RUN_ID=8536950919 2024-04-03T16:23:56.3968436Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-04-03T16:23:56.3969210Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-04-03T16:23:56.3970686Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_cc00ac8f-955d-4f0f-bada-1fb2a36d2255 2024-04-03T16:23:56.3971973Z GITHUB_ACTOR=caic99 2024-04-03T16:23:56.3972489Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-04-03T16:23:56.3973023Z PR_NUMBER=123154 2024-04-03T16:23:56.3973459Z GITHUB_RUN_ATTEMPT=2 2024-04-03T16:23:56.3973906Z VALGRIND=ON 2024-04-03T16:23:56.3974305Z ANACONDA_PYTHON_VERSION=3.10 2024-04-03T16:23:56.3974936Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-04-03T16:23:56.3975599Z TERM=xterm 2024-04-03T16:23:56.3976060Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-04-03T16:23:56.3976652Z INSTALLED_VISION=yes 2024-04-03T16:23:56.3977113Z BRANCH=pull/123154 2024-04-03T16:23:56.3977556Z OPENSSL_ROOT_DIR=/opt/openssl 2024-04-03T16:23:56.3978128Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-04-03T16:23:56.3978736Z CUDA_PATH=/usr/local/cuda 2024-04-03T16:23:56.3979816Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-04-03T16:23:56.3980933Z GITHUB_SERVER_URL=https://github.com 2024-04-03T16:23:56.3981638Z UCC_COMMIT=7cb07a76ccedad7e56ceb136b865eb9319c258ea 2024-04-03T16:23:56.3982299Z REENABLED_ISSUES= 2024-04-03T16:23:56.3982857Z SHLVL=1 2024-04-03T16:23:56.3983252Z MAX_JOBS=30 2024-04-03T16:23:56.3983681Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-04-03T16:23:56.3984187Z NVARCH=x86_64 2024-04-03T16:23:56.3984608Z GITHUB_ACTOR_ID=78061359 2024-04-03T16:23:56.3985261Z GITHUB_WORKFLOW_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:23:56.3986008Z GITHUB_REF_NAME=123154/merge 2024-04-03T16:23:56.3986627Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-04-03T16:23:56.3987573Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-04-03T16:23:56.3988408Z GITHUB_JOB=test 2024-04-03T16:23:56.3988983Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-04-03T16:23:56.3989769Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-04-03T16:23:56.3990510Z NO_TEST_TIMEOUT=False 2024-04-03T16:23:56.3991062Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-04-03T16:23:56.3991674Z GITHUB_REPOSITORY=pytorch/pytorch 2024-04-03T16:23:56.3992276Z NV_NVPROF_VERSION=11.8.87-1 2024-04-03T16:23:56.3992798Z GITHUB_RETENTION_DAYS=90 2024-04-03T16:23:56.3993280Z OPENSSL_DIR=/opt/openssl 2024-04-03T16:23:56.3993780Z GITHUB_ACTION_REPOSITORY= 2024-04-03T16:23:56.3995609Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:23:56.3997184Z GITHUB_BASE_REF=main 2024-04-03T16:23:56.3997673Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-04-03T16:23:56.3998196Z CI=true 2024-04-03T16:23:56.3998629Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-04-03T16:23:56.3999229Z GITHUB_REPOSITORY_OWNER=pytorch 2024-04-03T16:23:56.3999768Z JOB_ID=23402706124 2024-04-03T16:23:56.4000194Z INSTALLED_PROTOBUF=yes 2024-04-03T16:23:56.4000702Z GITHUB_HEAD_REF=patch-1 2024-04-03T16:23:56.4001181Z GITHUB_ACTION_REF= 2024-04-03T16:23:56.4001961Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-04-03T16:23:56.4002641Z GITHUB_WORKFLOW=pull 2024-04-03T16:23:56.4003125Z DEBIAN_FRONTEND=noninteractive 2024-04-03T16:23:56.4004452Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_cc00ac8f-955d-4f0f-bada-1fb2a36d2255 2024-04-03T16:23:56.4005667Z NO_TD=False 2024-04-03T16:23:56.4006092Z SKIP_SCCACHE_INITIALIZATION=1 2024-04-03T16:23:56.4006593Z _=/usr/bin/env 2024-04-03T16:23:56.4007070Z + echo 'Testing pytorch' 2024-04-03T16:23:56.4007562Z Testing pytorch 2024-04-03T16:23:56.4008006Z + export LANG=C.UTF-8 2024-04-03T16:23:56.4008490Z + LANG=C.UTF-8 2024-04-03T16:23:56.4008913Z + PR_NUMBER=123154 2024-04-03T16:23:56.4009374Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-04-03T16:23:56.4010020Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-04-03T16:23:56.4010829Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-04-03T16:23:56.4011506Z + [[ distributed == \s\l\o\w ]] 2024-04-03T16:23:56.4012318Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-04-03T16:23:56.4013236Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-04-03T16:23:56.4013975Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-04-03T16:23:56.4014653Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-04-03T16:23:56.4015271Z + [[ distributed == *crossref* ]] 2024-04-03T16:23:56.4015981Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-04-03T16:23:56.4016816Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-04-03T16:23:56.4017672Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-04-03T16:23:56.4018430Z + pip_install --user ninja==1.10.2 2024-04-03T16:23:56.4019190Z + pip install --progress-bar off --user ninja==1.10.2 2024-04-03T16:23:56.9772673Z Collecting ninja==1.10.2 2024-04-03T16:23:56.9984922Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-04-03T16:23:57.0204229Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-04-03T16:23:58.1801906Z Installing collected packages: ninja 2024-04-03T16:23:58.1915315Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-04-03T16:23:58.1917256Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-04-03T16:23:58.1992334Z Successfully installed ninja-1.10.2 2024-04-03T16:23:58.2765470Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:23:58.2768666Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:23:58.2770474Z + install_tlparse 2024-04-03T16:23:58.2771124Z + pip_install --user tlparse==0.3.7 2024-04-03T16:23:58.2771959Z + pip install --progress-bar off --user tlparse==0.3.7 2024-04-03T16:23:58.8391375Z Collecting tlparse==0.3.7 2024-04-03T16:23:58.8591864Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-04-03T16:23:58.8714608Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-04-03T16:24:00.0491951Z Installing collected packages: tlparse 2024-04-03T16:24:00.1043858Z Successfully installed tlparse-0.3.7 2024-04-03T16:24:00.1788734Z ++ python -m site --user-base 2024-04-03T16:24:00.2048702Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:24:00.2051706Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-04-03T16:24:00.2052965Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-04-03T16:24:00.2054235Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-04-03T16:24:00.2055595Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-04-03T16:24:00.2058106Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-04-03T16:24:00.2059080Z + cd test 2024-04-03T16:24:00.2059835Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-04-03T16:24:02.2486820Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-04-03T16:24:02.2487810Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-04-03T16:24:02.2490540Z + DYNAMO_BENCHMARK_FLAGS=() 2024-04-03T16:24:02.2491755Z + [[ distributed == *dynamo_eager* ]] 2024-04-03T16:24:02.2492589Z + [[ distributed == *aot_eager* ]] 2024-04-03T16:24:02.2493187Z + [[ distributed == *aot_inductor* ]] 2024-04-03T16:24:02.2493783Z + [[ distributed == *inductor* ]] 2024-04-03T16:24:02.2494336Z + [[ distributed == *dynamic* ]] 2024-04-03T16:24:02.2494918Z + [[ distributed == *cpu_inductor* ]] 2024-04-03T16:24:02.2495748Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-04-03T16:24:02.2509684Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *tbb* ]] 2024-04-03T16:24:02.2531023Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-04-03T16:24:02.2532273Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-04-03T16:24:02.2533588Z + cd test 2024-04-03T16:24:02.2534937Z + python -c 'import torch; print(torch.__config__.show())' 2024-04-03T16:24:04.1182316Z PyTorch built with: 2024-04-03T16:24:04.1183599Z - GCC 9.4 2024-04-03T16:24:04.1184388Z - C++ Version: 201703 2024-04-03T16:24:04.1185653Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-04-03T16:24:04.1187225Z - Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361) 2024-04-03T16:24:04.1188171Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-04-03T16:24:04.1188904Z - LAPACK is enabled (usually provided by MKL) 2024-04-03T16:24:04.1189735Z - NNPACK is enabled 2024-04-03T16:24:04.1209116Z - CPU capability usage: AVX2 2024-04-03T16:24:04.1209839Z - CUDA Runtime 11.8 2024-04-03T16:24:04.1210929Z - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52 2024-04-03T16:24:04.1211768Z - CuDNN 8.7 2024-04-03T16:24:04.1212217Z - Magma 2.6.1 2024-04-03T16:24:04.1223008Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-04-03T16:24:04.1233112Z 2024-04-03T16:24:04.3845979Z + cd test 2024-04-03T16:24:04.3847209Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-04-03T16:24:06.1689316Z ATen/Parallel: 2024-04-03T16:24:06.1690432Z at::get_num_threads() : 16 2024-04-03T16:24:06.1691220Z at::get_num_interop_threads() : 32 2024-04-03T16:24:06.1691826Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-04-03T16:24:06.1692815Z omp_get_max_threads() : 16 2024-04-03T16:24:06.1694199Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-04-03T16:24:06.1695393Z mkl_get_max_threads() : 16 2024-04-03T16:24:06.1696300Z Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361) 2024-04-03T16:24:06.1697193Z std::thread::hardware_concurrency() : 32 2024-04-03T16:24:06.1697810Z Environment variables: 2024-04-03T16:24:06.1698301Z OMP_NUM_THREADS : [not set] 2024-04-03T16:24:06.1698811Z MKL_NUM_THREADS : [not set] 2024-04-03T16:24:06.1699340Z ATen parallel backend: OpenMP 2024-04-03T16:24:06.1699706Z 2024-04-03T16:24:06.4058979Z + [[ distributed == *backward* ]] 2024-04-03T16:24:06.4060124Z + [[ distributed == *xla* ]] 2024-04-03T16:24:06.4060677Z + [[ distributed == *executorch* ]] 2024-04-03T16:24:06.4061307Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-04-03T16:24:06.4062265Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-04-03T16:24:06.4063020Z + [[ distributed == distributed ]] 2024-04-03T16:24:06.4063581Z + test_distributed 2024-04-03T16:24:06.4064150Z + echo 'Testing distributed python tests' 2024-04-03T16:24:06.4064792Z Testing distributed python tests 2024-04-03T16:24:06.4065678Z + python test/run_test.py --distributed-tests --shard 2 3 --verbose 2024-04-03T16:24:06.5358135Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-04-03T16:24:06.5359807Z import pkg_resources 2024-04-03T16:24:10.0584978Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-04-03T16:24:10.1418655Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-04-03T16:24:10.1998702Z Ignoring disabled issues: [''] 2024-04-03T16:24:10.2099374Z Found test times from artifacts 2024-04-03T16:24:10.2486383Z Found test times from artifacts 2024-04-03T16:24:10.2498392Z Running all tests 2024-04-03T16:24:10.2660031Z Name: tests to run (est. time: 69.16min) 2024-04-03T16:24:10.2660938Z Serial tests (76): 2024-04-03T16:24:10.2661529Z distributed/checkpoint/test_nested_dict 1/1 2024-04-03T16:24:10.2662213Z distributed/_tensor/test_dtensor 1/1 2024-04-03T16:24:10.2662888Z distributed/_spmd/test_transformation 1/1 2024-04-03T16:24:10.2663582Z distributed/test_distributed_spawn 3/7 2024-04-03T16:24:10.2664293Z distributed/test_distributed_spawn 6/7 2024-04-03T16:24:10.2665008Z distributed/checkpoint/test_fsdp_optim_state 1/1 2024-04-03T16:24:10.2665724Z distributed/_tensor/test_api 1/1 2024-04-03T16:24:10.2666361Z distributed/_tensor/test_random_ops 1/1 2024-04-03T16:24:10.2667093Z distributed/checkpoint/test_dtensor_resharding 1/1 2024-04-03T16:24:10.2667935Z distributed/_tensor/experimental/test_tp_transform 1/1 2024-04-03T16:24:10.2668835Z distributed/_composable/fsdp/test_fully_shard_autograd 1/1 2024-04-03T16:24:10.2669642Z distributed/test_compute_comm_reordering 1/1 2024-04-03T16:24:10.2670447Z distributed/tensor/parallel/test_ddp_2d_parallel 1/1 2024-04-03T16:24:10.2671208Z distributed/checkpoint/test_fsspec 1/1 2024-04-03T16:24:10.2671916Z distributed/_tensor/test_experimental_ops 1/1 2024-04-03T16:24:10.2672601Z distributed/fsdp/test_fsdp_memory 1/1 2024-04-03T16:24:10.2673285Z distributed/_tensor/test_convolution_ops 1/1 2024-04-03T16:24:10.2674007Z distributed/_tensor/test_redistribute 1/1 2024-04-03T16:24:10.2675002Z distributed/test_collective_utils 1/1 2024-04-03T16:24:10.2675747Z distributed/tensor/parallel/test_tp_random_state 1/1 2024-04-03T16:24:10.2676487Z distributed/_tensor/test_math_ops 1/1 2024-04-03T16:24:10.2677203Z distributed/checkpoint/e2e/test_e2e_save_and_load 1/1 2024-04-03T16:24:10.2678090Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 2024-04-03T16:24:10.2679289Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 2024-04-03T16:24:10.2680181Z distributed/checkpoint/test_state_dict_utils 1/1 2024-04-03T16:24:10.2681009Z distributed/tensor/parallel/test_fsdp_2d_parallel 1/1 2024-04-03T16:24:10.2681746Z distributed/test_fake_pg 1/1 2024-04-03T16:24:10.2682415Z distributed/tensor/parallel/test_parallelize_api 1/1 2024-04-03T16:24:10.2683311Z distributed/_composable/fully_shard/test_fully_shard_util 1/1 2024-04-03T16:24:10.2684097Z distributed/test_c10d_pypg 1/1 2024-04-03T16:24:10.2684757Z distributed/fsdp/test_distributed_checkpoint 1/1 2024-04-03T16:24:10.2685473Z distributed/_tensor/test_utils 1/1 2024-04-03T16:24:10.2686206Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 2024-04-03T16:24:10.2687177Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 1/1 2024-04-03T16:24:10.2688079Z distributed/checkpoint/test_utils 1/1 2024-04-03T16:24:10.2688738Z distributed/_tensor/test_view_ops 1/1 2024-04-03T16:24:10.2689512Z distributed/_composable/fully_shard/test_fully_shard_init 1/1 2024-04-03T16:24:10.2690305Z distributed/_shard/test_sharder 1/1 2024-04-03T16:24:10.2690961Z distributed/fsdp/test_fsdp_overlap 1/1 2024-04-03T16:24:10.2691605Z distributed/test_c10d_logger 1/1 2024-04-03T16:24:10.2692315Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 2024-04-03T16:24:10.2693156Z distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 2024-04-03T16:24:10.2693932Z distributed/fsdp/test_fsdp_tp_integration 1/1 2024-04-03T16:24:10.2694620Z distributed/test_multi_threaded_pg 1/1 2024-04-03T16:24:10.2695304Z distributed/nn/jit/test_instantiator 1/1 2024-04-03T16:24:10.2696082Z distributed/checkpoint/test_file_system_checkpoint 1/1 2024-04-03T16:24:10.2696843Z distributed/checkpoint/test_planner 1/1 2024-04-03T16:24:10.2697512Z distributed/fsdp/test_fsdp_apply 1/1 2024-04-03T16:24:10.2698206Z distributed/pipeline/sync/test_pipeline 1/1 2024-04-03T16:24:10.2698998Z distributed/_composable/test_replicate_with_compiler 1/1 2024-04-03T16:24:10.2699804Z distributed/fsdp/test_fsdp_state_dict 1/2 2024-04-03T16:24:10.2700600Z distributed/fsdp/test_fsdp_core 2/2 2024-04-03T16:24:10.2701347Z distributed/optim/test_apply_optimizer_in_backward 1/1 2024-04-03T16:24:10.2702090Z distributed/fsdp/test_utils 1/1 2024-04-03T16:24:10.2702733Z distributed/pipeline/sync/test_phony 1/1 2024-04-03T16:24:10.2703430Z distributed/test_c10d_object_collectives 1/1 2024-04-03T16:24:10.2704132Z distributed/pipeline/sync/test_bugs 1/1 2024-04-03T16:24:10.2704872Z distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 2024-04-03T16:24:10.2705632Z distributed/pipeline/sync/skip/test_gpipe 1/1 2024-04-03T16:24:10.2706334Z distributed/algorithms/test_join 1/1 2024-04-03T16:24:10.2707092Z distributed/_shard/sharded_tensor/ops/test_embedding 1/1 2024-04-03T16:24:10.2707980Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 2024-04-03T16:24:10.2708842Z distributed/rpc/cuda/test_tensorpipe_agent 2/2 2024-04-03T16:24:10.2709540Z distributed/test_c10d_common 1/1 2024-04-03T16:24:10.2710176Z distributed/pipeline/sync/test_stream 1/1 2024-04-03T16:24:10.2710851Z distributed/test_c10d_spawn_nccl 1/1 2024-04-03T16:24:10.2711461Z distributed/test_nccl 1/1 2024-04-03T16:24:10.2711982Z distributed/test_c10d_gloo 1/1 2024-04-03T16:24:10.2712587Z distributed/elastic/events/lib_test 1/1 2024-04-03T16:24:10.2713276Z distributed/elastic/metrics/api_test 1/1 2024-04-03T16:24:10.2714012Z distributed/elastic/multiprocessing/api_test 1/1 2024-04-03T16:24:10.2715011Z distributed/elastic/timer/local_timer_example 1/1 2024-04-03T16:24:10.2715809Z distributed/elastic/timer/local_timer_test 1/1 2024-04-03T16:24:10.2716573Z distributed/elastic/utils/distributed_test 1/1 2024-04-03T16:24:10.2717300Z distributed/elastic/utils/logging_test 1/1 2024-04-03T16:24:10.2718157Z distributed/elastic/utils/util_test 1/1 2024-04-03T16:24:10.2718784Z Parallel tests (0): 2024-04-03T16:24:10.2719263Z Name: excluded (est. time: 0.0min) 2024-04-03T16:24:10.2719824Z Serial tests (0): 2024-04-03T16:24:10.2720281Z Parallel tests (0): 2024-04-03T16:24:10.2721139Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-04-03T16:24:10.2735935Z Running distributed/checkpoint/test_nested_dict 1/1 ... [2024-04-03 16:24:10.273163] 2024-04-03T16:24:10.2743714Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_nested_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:24:10.273837] 2024-04-03T16:24:14.3473216Z 2024-04-03T16:24:14.3475967Z distributed/checkpoint/test_nested_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_nested_dict_1.1_3521e461febaea11_.log 2024-04-03T16:24:14.3479160Z Running 2 items in this shard: test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_flattening_round_trip, test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_mapping 2024-04-03T16:24:14.3481441Z 2024-04-03T16:24:14.3482469Z Running distributed/_tensor/test_dtensor 1/1 ... [2024-04-03 16:24:14.347901] 2024-04-03T16:24:14.3489084Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:24:14.348380] 2024-04-03T16:28:22.4625766Z 2024-04-03T16:28:22.4627847Z distributed/_tensor/test_dtensor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_1.1_4e5b12d772640e2b_.log 2024-04-03T16:28:22.4673244Z Running 57 items in this shard: test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_async_output_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_async_output_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_constructor_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_constructor_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_new_empty_strided_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_new_empty_strided_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_properties_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_properties_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_save_load_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_save_load_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_hash_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_hash_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_read_only_after_set_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_read_only_after_set_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_stride_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_stride_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_negative_dim_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_negative_dim_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_then_to_local_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_then_to_local_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding_raise_error_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding_raise_error_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_grad_hint_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_grad_hint_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_sync_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_sync_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_meta_dtensor_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_meta_dtensor_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_modules_w_meta_dtensor_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_modules_w_meta_dtensor_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local_grad_hint_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local_grad_hint_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_default_value_sub_mesh_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_default_value_sub_mesh_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_device_mesh_nd_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_device_mesh_nd_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_2d_mesh_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_2d_mesh_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_api_device_mesh_context_manager_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_api_device_mesh_context_manager_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_device_mesh_device_conversion_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_device_mesh_device_conversion_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_spec_local_shard_offset_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_spec_local_shard_offset_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_from_local_sub_mesh_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_from_local_sub_mesh_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_implicit_replication_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_implicit_replication_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_redistribute_sub_mesh_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_redistribute_sub_mesh_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::TestDTensorPlacementTypes::test_split_tensor_1D_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::TestDTensorPlacementTypes::test_split_tensor_1D_use_native_funcol_True 2024-04-03T16:28:22.4717068Z 2024-04-03T16:28:22.4717727Z Running distributed/_spmd/test_transformation 1/1 ... [2024-04-03 16:28:22.462879] 2024-04-03T16:28:22.4720556Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_spmd/test_transformation.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:28:22.463325] 2024-04-03T16:28:26.4986260Z 2024-04-03T16:28:26.4988526Z distributed/_spmd/test_transformation 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._spmd.test_transformation_1.1_e661aec4dca40fac_.log 2024-04-03T16:28:26.4990135Z 2024-04-03T16:28:26.4991235Z Running distributed/test_distributed_spawn 3/7 ... [2024-04-03 16:28:26.498759] 2024-04-03T16:28:26.5002532Z /usr/bin/mpiexec 2024-04-03T16:28:26.5003214Z MPI not available -- MPI backend tests will be skipped 2024-04-03T16:28:26.5005018Z Running distributed tests for the test backend with env init_method 2024-04-03T16:28:26.5013697Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:28:26.500885] 2024-04-03T16:28:30.4380686Z 2024-04-03T16:28:30.4382651Z distributed/test_distributed_spawn 3/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.7_c8bbdcd6db047027_.log 2024-04-03T16:28:30.4385983Z Running 0 items in this shard: 2024-04-03T16:28:30.4386419Z 2024-04-03T16:28:30.4396105Z Running distributed tests for the test backend with file init_method 2024-04-03T16:28:30.4418897Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:28:30.441346] 2024-04-03T16:28:34.3375951Z 2024-04-03T16:28:34.3378985Z distributed/test_distributed_spawn 3/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.7_bd3266d02baf3957_.log 2024-04-03T16:28:34.3380914Z Running 0 items in this shard: 2024-04-03T16:28:34.3381301Z 2024-04-03T16:28:34.3388363Z Running distributed tests for the nccl backend with env init_method 2024-04-03T16:28:34.3397744Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:28:34.339173] 2024-04-03T16:32:42.7677065Z 2024-04-03T16:32:42.7682157Z distributed/test_distributed_spawn 3/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.7_a938b6a40ce37f58_.log 2024-04-03T16:32:42.7714788Z Running 35 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2024-04-03T16:32:42.7743317Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2024-04-03T16:32:42.7745678Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2024-04-03T16:32:42.7747809Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2024-04-03T16:32:42.7749928Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2024-04-03T16:32:42.7752053Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async 2024-04-03T16:32:42.7754006Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2024-04-03T16:32:42.7756860Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2024-04-03T16:32:42.7759069Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2024-04-03T16:32:42.7761498Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group 2024-04-03T16:32:42.7763687Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2024-04-03T16:32:42.7765723Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2024-04-03T16:32:42.7767694Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2024-04-03T16:32:42.7769683Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2024-04-03T16:32:42.7771939Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2024-04-03T16:32:42.7774327Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2024-04-03T16:32:42.7776385Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2024-04-03T16:32:42.7778350Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2024-04-03T16:32:42.7780284Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2024-04-03T16:32:42.7782308Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2024-04-03T16:32:42.7784439Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2024-04-03T16:32:42.7786482Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2024-04-03T16:32:42.7788513Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2024-04-03T16:32:42.7790373Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2024-04-03T16:32:42.7792397Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup 2024-04-03T16:32:42.7795167Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2024-04-03T16:32:42.7797203Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2024-04-03T16:32:42.7799133Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups 2024-04-03T16:32:42.7801138Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2024-04-03T16:32:42.7803270Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2024-04-03T16:32:42.7805290Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2024-04-03T16:32:42.7807250Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2024-04-03T16:32:42.7809123Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2024-04-03T16:32:42.7811020Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2024-04-03T16:32:42.7813236Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2024-04-03T16:32:42.7815193Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2024-04-03T16:32:42.7816237Z 2024-04-03T16:32:42.7816683Z Running distributed tests for the nccl backend with file init_method 2024-04-03T16:32:42.7819512Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:32:42.771091] 2024-04-03T16:36:52.0775532Z 2024-04-03T16:36:52.0779808Z distributed/test_distributed_spawn 3/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.7_905aac7628a68065_.log 2024-04-03T16:36:52.0811798Z Running 35 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2024-04-03T16:36:52.0840963Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2024-04-03T16:36:52.0843350Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2024-04-03T16:36:52.0845479Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2024-04-03T16:36:52.0847593Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2024-04-03T16:36:52.0849733Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async 2024-04-03T16:36:52.0851717Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2024-04-03T16:36:52.0853790Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2024-04-03T16:36:52.0855988Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2024-04-03T16:36:52.0858218Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group 2024-04-03T16:36:52.0860573Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2024-04-03T16:36:52.0862653Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2024-04-03T16:36:52.0864609Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2024-04-03T16:36:52.0866571Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2024-04-03T16:36:52.0868840Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2024-04-03T16:36:52.0871227Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2024-04-03T16:36:52.0873282Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2024-04-03T16:36:52.0875739Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2024-04-03T16:36:52.0877684Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2024-04-03T16:36:52.0879707Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2024-04-03T16:36:52.0881841Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2024-04-03T16:36:52.0883857Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2024-04-03T16:36:52.0885894Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2024-04-03T16:36:52.0887742Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2024-04-03T16:36:52.0889775Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup 2024-04-03T16:36:52.0891832Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2024-04-03T16:36:52.0893864Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2024-04-03T16:36:52.0895794Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups 2024-04-03T16:36:52.0897790Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2024-04-03T16:36:52.0899903Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2024-04-03T16:36:52.0901934Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2024-04-03T16:36:52.0903895Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2024-04-03T16:36:52.0905747Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2024-04-03T16:36:52.0907652Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2024-04-03T16:36:52.0909689Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2024-04-03T16:36:52.0911764Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2024-04-03T16:36:52.0912822Z 2024-04-03T16:36:52.0913243Z Running distributed tests for the gloo backend with env init_method 2024-04-03T16:36:52.0916557Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:36:52.080273] 2024-04-03T16:41:00.0405978Z 2024-04-03T16:41:00.0407960Z distributed/test_distributed_spawn 3/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.7_39f019b06941c4a0_.log 2024-04-03T16:41:00.0437776Z Running 35 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2024-04-03T16:41:00.0466279Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2024-04-03T16:41:00.0468666Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2024-04-03T16:41:00.0470771Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2024-04-03T16:41:00.0472895Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2024-04-03T16:41:00.0475716Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async 2024-04-03T16:41:00.0477700Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2024-04-03T16:41:00.0479766Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2024-04-03T16:41:00.0481958Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2024-04-03T16:41:00.0484188Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group 2024-04-03T16:41:00.0486563Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2024-04-03T16:41:00.0488599Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2024-04-03T16:41:00.0490559Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2024-04-03T16:41:00.0492546Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2024-04-03T16:41:00.0494814Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2024-04-03T16:41:00.0497168Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2024-04-03T16:41:00.0499218Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2024-04-03T16:41:00.0501187Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2024-04-03T16:41:00.0503124Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2024-04-03T16:41:00.0505122Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2024-04-03T16:41:00.0507250Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2024-04-03T16:41:00.0509283Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2024-04-03T16:41:00.0511153Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2024-04-03T16:41:00.0513165Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2024-04-03T16:41:00.0515956Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup 2024-04-03T16:41:00.0518041Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2024-04-03T16:41:00.0520050Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2024-04-03T16:41:00.0521975Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups 2024-04-03T16:41:00.0523953Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2024-04-03T16:41:00.0526089Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2024-04-03T16:41:00.0528101Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2024-04-03T16:41:00.0530057Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2024-04-03T16:41:00.0531923Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2024-04-03T16:41:00.0533834Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2024-04-03T16:41:00.0535891Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2024-04-03T16:41:00.0538046Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2024-04-03T16:41:00.0539105Z 2024-04-03T16:41:00.0539539Z Running distributed tests for the gloo backend with file init_method 2024-04-03T16:41:00.0542403Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:41:00.043400] 2024-04-03T16:45:07.2641370Z 2024-04-03T16:45:07.2643543Z distributed/test_distributed_spawn 3/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.7_2b5384d13697351a_.log 2024-04-03T16:45:07.2672234Z Running 35 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2024-04-03T16:45:07.2701273Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2024-04-03T16:45:07.2703653Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2024-04-03T16:45:07.2705763Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2024-04-03T16:45:07.2707850Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2024-04-03T16:45:07.2710000Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async 2024-04-03T16:45:07.2711974Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2024-04-03T16:45:07.2714050Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2024-04-03T16:45:07.2716934Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2024-04-03T16:45:07.2719181Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group 2024-04-03T16:45:07.2721365Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2024-04-03T16:45:07.2723595Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2024-04-03T16:45:07.2725563Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2024-04-03T16:45:07.2727546Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2024-04-03T16:45:07.2729819Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2024-04-03T16:45:07.2732187Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2024-04-03T16:45:07.2734268Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2024-04-03T16:45:07.2736256Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2024-04-03T16:45:07.2738203Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2024-04-03T16:45:07.2740228Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2024-04-03T16:45:07.2742339Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2024-04-03T16:45:07.2744378Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2024-04-03T16:45:07.2746245Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2024-04-03T16:45:07.2748239Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2024-04-03T16:45:07.2750281Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup 2024-04-03T16:45:07.2752363Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2024-04-03T16:45:07.2755085Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2024-04-03T16:45:07.2757021Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups 2024-04-03T16:45:07.2759004Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2024-04-03T16:45:07.2761118Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2024-04-03T16:45:07.2763169Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2024-04-03T16:45:07.2765102Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2024-04-03T16:45:07.2766972Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2024-04-03T16:45:07.2768886Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2024-04-03T16:45:07.2770918Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2024-04-03T16:45:07.2772847Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2024-04-03T16:45:07.2773911Z 2024-04-03T16:45:07.2774325Z Running distributed tests for the ucc backend with env init_method 2024-04-03T16:45:07.2777311Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:45:07.266968] 2024-04-03T16:49:22.1234107Z 2024-04-03T16:49:22.1237515Z distributed/test_distributed_spawn 3/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.7_3ef940ee4f2485e3_.log 2024-04-03T16:49:22.1267837Z Running 35 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2024-04-03T16:49:22.1297146Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2024-04-03T16:49:22.1299509Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2024-04-03T16:49:22.1301613Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2024-04-03T16:49:22.1303721Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2024-04-03T16:49:22.1305834Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async 2024-04-03T16:49:22.1307809Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2024-04-03T16:49:22.1309876Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2024-04-03T16:49:22.1312065Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2024-04-03T16:49:22.1314965Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group 2024-04-03T16:49:22.1317184Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2024-04-03T16:49:22.1319418Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2024-04-03T16:49:22.1321384Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2024-04-03T16:49:22.1323355Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2024-04-03T16:49:22.1325621Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2024-04-03T16:49:22.1327988Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2024-04-03T16:49:22.1330028Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2024-04-03T16:49:22.1331975Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2024-04-03T16:49:22.1333955Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2024-04-03T16:49:22.1335974Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2024-04-03T16:49:22.1338100Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2024-04-03T16:49:22.1340115Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2024-04-03T16:49:22.1341977Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2024-04-03T16:49:22.1343814Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2024-04-03T16:49:22.1345969Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup 2024-04-03T16:49:22.1348041Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2024-04-03T16:49:22.1350074Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2024-04-03T16:49:22.1351994Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups 2024-04-03T16:49:22.1353977Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2024-04-03T16:49:22.1356802Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2024-04-03T16:49:22.1358843Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2024-04-03T16:49:22.1360806Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2024-04-03T16:49:22.1362662Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2024-04-03T16:49:22.1364569Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2024-04-03T16:49:22.1366598Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2024-04-03T16:49:22.1368539Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2024-04-03T16:49:22.1369582Z 2024-04-03T16:49:22.1370001Z Running distributed tests for the ucc backend with file init_method 2024-04-03T16:49:22.1372993Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:49:22.126857] 2024-04-03T16:53:36.7989378Z 2024-04-03T16:53:36.7995935Z distributed/test_distributed_spawn 3/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.7_861b207ce5857cae_.log 2024-04-03T16:53:36.8046822Z Running 35 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2024-04-03T16:53:36.8085595Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2024-04-03T16:53:36.8087980Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2024-04-03T16:53:36.8090078Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2024-04-03T16:53:36.8092191Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2024-04-03T16:53:36.8094314Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async 2024-04-03T16:53:36.8096309Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2024-04-03T16:53:36.8098352Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2024-04-03T16:53:36.8100540Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2024-04-03T16:53:36.8102766Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group 2024-04-03T16:53:36.8104953Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2024-04-03T16:53:36.8106979Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2024-04-03T16:53:36.8109164Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2024-04-03T16:53:36.8111220Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2024-04-03T16:53:36.8113488Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2024-04-03T16:53:36.8116435Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2024-04-03T16:53:36.8118487Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2024-04-03T16:53:36.8120446Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2024-04-03T16:53:36.8122397Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2024-04-03T16:53:36.8124408Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2024-04-03T16:53:36.8126536Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2024-04-03T16:53:36.8128558Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2024-04-03T16:53:36.8130420Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2024-04-03T16:53:36.8132318Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2024-04-03T16:53:36.8134524Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup 2024-04-03T16:53:36.8136590Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2024-04-03T16:53:36.8138626Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2024-04-03T16:53:36.8140533Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups 2024-04-03T16:53:36.8142512Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2024-04-03T16:53:36.8144626Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2024-04-03T16:53:36.8146631Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2024-04-03T16:53:36.8148595Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2024-04-03T16:53:36.8150469Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2024-04-03T16:53:36.8152383Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2024-04-03T16:53:36.8154987Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2024-04-03T16:53:36.8156953Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2024-04-03T16:53:36.8158012Z 2024-04-03T16:53:36.8158647Z Running distributed/test_distributed_spawn 6/7 ... [2024-04-03 16:53:36.803668] 2024-04-03T16:53:36.8159532Z /usr/bin/mpiexec 2024-04-03T16:53:36.8160181Z MPI not available -- MPI backend tests will be skipped 2024-04-03T16:53:36.8161128Z Running distributed tests for the test backend with env init_method 2024-04-03T16:53:36.8164074Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:53:36.806134] 2024-04-03T16:53:40.7248434Z 2024-04-03T16:53:40.7250724Z distributed/test_distributed_spawn 6/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.7_3e03558aad63f984_.log 2024-04-03T16:53:40.7252450Z Running 0 items in this shard: 2024-04-03T16:53:40.7252833Z 2024-04-03T16:53:40.7261197Z Running distributed tests for the test backend with file init_method 2024-04-03T16:53:40.7270296Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:53:40.726551] 2024-04-03T16:53:44.6602480Z 2024-04-03T16:53:44.6604687Z distributed/test_distributed_spawn 6/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.7_b7df6e87edaa0a76_.log 2024-04-03T16:53:44.6606411Z Running 0 items in this shard: 2024-04-03T16:53:44.6606798Z 2024-04-03T16:53:44.6614293Z Running distributed tests for the nccl backend with env init_method 2024-04-03T16:53:44.6622393Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:53:44.661751] 2024-04-03T16:58:46.9441563Z 2024-04-03T16:58:46.9444114Z distributed/test_distributed_spawn 6/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.7_31a190479c17e3c1_.log 2024-04-03T16:58:46.9484304Z Running 43 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_new_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-04-03T16:58:46.9520269Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2024-04-03T16:58:46.9522708Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2024-04-03T16:58:46.9524761Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2024-04-03T16:58:46.9526755Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2024-04-03T16:58:46.9528764Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2024-04-03T16:58:46.9530810Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2024-04-03T16:58:46.9532881Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup 2024-04-03T16:58:46.9534889Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min 2024-04-03T16:58:46.9537089Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2024-04-03T16:58:46.9539150Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2024-04-03T16:58:46.9541292Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2024-04-03T16:58:46.9543446Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2024-04-03T16:58:46.9545611Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group 2024-04-03T16:58:46.9547808Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2024-04-03T16:58:46.9549871Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2024-04-03T16:58:46.9551840Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2024-04-03T16:58:46.9553887Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2024-04-03T16:58:46.9556174Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2024-04-03T16:58:46.9558034Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2024-04-03T16:58:46.9560005Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2024-04-03T16:58:46.9562263Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping 2024-04-03T16:58:46.9564549Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2024-04-03T16:58:46.9566850Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2024-04-03T16:58:46.9568960Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook 2024-04-03T16:58:46.9570959Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2024-04-03T16:58:46.9572921Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2024-04-03T16:58:46.9575134Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2024-04-03T16:58:46.9577530Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params 2024-04-03T16:58:46.9579919Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2024-04-03T16:58:46.9582182Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2024-04-03T16:58:46.9584143Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states 2024-04-03T16:58:46.9586073Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2024-04-03T16:58:46.9588075Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_new_group 2024-04-03T16:58:46.9590181Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2024-04-03T16:58:46.9592027Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2024-04-03T16:58:46.9593854Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group 2024-04-03T16:58:46.9596117Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2024-04-03T16:58:46.9598125Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2024-04-03T16:58:46.9600376Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2024-04-03T16:58:46.9602711Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2024-04-03T16:58:46.9604714Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2024-04-03T16:58:46.9606588Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2024-04-03T16:58:46.9608541Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-04-03T16:58:46.9609669Z 2024-04-03T16:58:46.9610118Z Running distributed tests for the nccl backend with file init_method 2024-04-03T16:58:46.9612963Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:58:46.947563] 2024-04-03T17:03:49.6527371Z 2024-04-03T17:03:49.6532628Z distributed/test_distributed_spawn 6/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.7_f14d43229be38a32_.log 2024-04-03T17:03:49.6570651Z Running 43 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_new_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-04-03T17:03:49.6606396Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2024-04-03T17:03:49.6608824Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2024-04-03T17:03:49.6610869Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2024-04-03T17:03:49.6612855Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2024-04-03T17:03:49.6614903Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2024-04-03T17:03:49.6616935Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2024-04-03T17:03:49.6618991Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup 2024-04-03T17:03:49.6621162Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min 2024-04-03T17:03:49.6623220Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2024-04-03T17:03:49.6625283Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2024-04-03T17:03:49.6627401Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2024-04-03T17:03:49.6629601Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2024-04-03T17:03:49.6631773Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group 2024-04-03T17:03:49.6634021Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2024-04-03T17:03:49.6636754Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2024-04-03T17:03:49.6638737Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2024-04-03T17:03:49.6640793Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2024-04-03T17:03:49.6642749Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2024-04-03T17:03:49.6644582Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2024-04-03T17:03:49.6646722Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2024-04-03T17:03:49.6648850Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping 2024-04-03T17:03:49.6651109Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2024-04-03T17:03:49.6653400Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2024-04-03T17:03:49.6655505Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook 2024-04-03T17:03:49.6657518Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2024-04-03T17:03:49.6659466Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2024-04-03T17:03:49.6661686Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2024-04-03T17:03:49.6664068Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params 2024-04-03T17:03:49.6666478Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2024-04-03T17:03:49.6668726Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2024-04-03T17:03:49.6670697Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states 2024-04-03T17:03:49.6672759Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2024-04-03T17:03:49.6675619Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_new_group 2024-04-03T17:03:49.6677607Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2024-04-03T17:03:49.6679433Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2024-04-03T17:03:49.6681251Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group 2024-04-03T17:03:49.6683153Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2024-04-03T17:03:49.6685152Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2024-04-03T17:03:49.6687419Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2024-04-03T17:03:49.6689748Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2024-04-03T17:03:49.6691763Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2024-04-03T17:03:49.6693638Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2024-04-03T17:03:49.6695586Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-04-03T17:03:49.6696713Z 2024-04-03T17:03:49.6697326Z Running distributed tests for the gloo backend with env init_method 2024-04-03T17:03:49.6700188Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:03:49.655678] 2024-04-03T17:09:05.1534770Z 2024-04-03T17:09:05.1536754Z distributed/test_distributed_spawn 6/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.7_d45eb690be470df2_.log 2024-04-03T17:09:05.1575143Z Running 43 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_new_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-04-03T17:09:05.1610983Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2024-04-03T17:09:05.1613406Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2024-04-03T17:09:05.1615449Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2024-04-03T17:09:05.1617432Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2024-04-03T17:09:05.1619438Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2024-04-03T17:09:05.1621448Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2024-04-03T17:09:05.1623658Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup 2024-04-03T17:09:05.1625697Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min 2024-04-03T17:09:05.1627714Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2024-04-03T17:09:05.1629798Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2024-04-03T17:09:05.1631928Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2024-04-03T17:09:05.1634090Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2024-04-03T17:09:05.1636975Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group 2024-04-03T17:09:05.1639180Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2024-04-03T17:09:05.1641223Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2024-04-03T17:09:05.1643184Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2024-04-03T17:09:05.1645222Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2024-04-03T17:09:05.1647323Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2024-04-03T17:09:05.1649180Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2024-04-03T17:09:05.1651151Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2024-04-03T17:09:05.1653274Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping 2024-04-03T17:09:05.1655526Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2024-04-03T17:09:05.1657822Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2024-04-03T17:09:05.1659922Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook 2024-04-03T17:09:05.1661953Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2024-04-03T17:09:05.1663894Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2024-04-03T17:09:05.1666109Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2024-04-03T17:09:05.1668497Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params 2024-04-03T17:09:05.1670893Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2024-04-03T17:09:05.1673140Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2024-04-03T17:09:05.1675576Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states 2024-04-03T17:09:05.1677525Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2024-04-03T17:09:05.1679522Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_new_group 2024-04-03T17:09:05.1681502Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2024-04-03T17:09:05.1683319Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2024-04-03T17:09:05.1685131Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group 2024-04-03T17:09:05.1687023Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2024-04-03T17:09:05.1689025Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2024-04-03T17:09:05.1691274Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2024-04-03T17:09:05.1693585Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2024-04-03T17:09:05.1695577Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2024-04-03T17:09:05.1697441Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2024-04-03T17:09:05.1699550Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-04-03T17:09:05.1700676Z 2024-04-03T17:09:05.1701127Z Running distributed tests for the gloo backend with file init_method 2024-04-03T17:09:05.1703964Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:09:05.157070] 2024-04-03T17:14:20.1487255Z 2024-04-03T17:14:20.1491605Z distributed/test_distributed_spawn 6/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.7_e7ccfbd1e5754c1b_.log 2024-04-03T17:14:20.1530325Z Running 43 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_new_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-04-03T17:14:20.1565971Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2024-04-03T17:14:20.1568411Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2024-04-03T17:14:20.1570447Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2024-04-03T17:14:20.1572476Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2024-04-03T17:14:20.1574633Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2024-04-03T17:14:20.1576689Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2024-04-03T17:14:20.1578753Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup 2024-04-03T17:14:20.1580752Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min 2024-04-03T17:14:20.1582784Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2024-04-03T17:14:20.1584841Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2024-04-03T17:14:20.1586990Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2024-04-03T17:14:20.1589134Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2024-04-03T17:14:20.1591311Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group 2024-04-03T17:14:20.1593521Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2024-04-03T17:14:20.1596151Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2024-04-03T17:14:20.1598104Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2024-04-03T17:14:20.1600313Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2024-04-03T17:14:20.1602274Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2024-04-03T17:14:20.1604126Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2024-04-03T17:14:20.1606076Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2024-04-03T17:14:20.1608207Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping 2024-04-03T17:14:20.1610723Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2024-04-03T17:14:20.1613044Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2024-04-03T17:14:20.1615146Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook 2024-04-03T17:14:20.1617172Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2024-04-03T17:14:20.1619131Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2024-04-03T17:14:20.1621342Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2024-04-03T17:14:20.1623698Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params 2024-04-03T17:14:20.1626107Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2024-04-03T17:14:20.1628524Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2024-04-03T17:14:20.1630510Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states 2024-04-03T17:14:20.1632423Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2024-04-03T17:14:20.1635115Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_new_group 2024-04-03T17:14:20.1637130Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2024-04-03T17:14:20.1638955Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2024-04-03T17:14:20.1640774Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group 2024-04-03T17:14:20.1642678Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2024-04-03T17:14:20.1644675Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2024-04-03T17:14:20.1646912Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2024-04-03T17:14:20.1649236Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2024-04-03T17:14:20.1651250Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2024-04-03T17:14:20.1653281Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2024-04-03T17:14:20.1655221Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-04-03T17:14:20.1656364Z 2024-04-03T17:14:20.1656775Z Running distributed tests for the ucc backend with env init_method 2024-04-03T17:14:20.1659611Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:14:20.151859] 2024-04-03T17:18:57.2804056Z 2024-04-03T17:18:57.2806430Z distributed/test_distributed_spawn 6/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.7_44795233b6c6d9d6_.log 2024-04-03T17:18:57.2845453Z Running 43 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_new_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-04-03T17:18:57.2881119Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2024-04-03T17:18:57.2883559Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2024-04-03T17:18:57.2885586Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2024-04-03T17:18:57.2887762Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2024-04-03T17:18:57.2889793Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2024-04-03T17:18:57.2891829Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2024-04-03T17:18:57.2893865Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup 2024-04-03T17:18:57.2895882Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min 2024-04-03T17:18:57.2897909Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2024-04-03T17:18:57.2899987Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2024-04-03T17:18:57.2902106Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2024-04-03T17:18:57.2904258Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2024-04-03T17:18:57.2906428Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group 2024-04-03T17:18:57.2908644Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2024-04-03T17:18:57.2910667Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2024-04-03T17:18:57.2912796Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2024-04-03T17:18:57.2915399Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2024-04-03T17:18:57.2917375Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2024-04-03T17:18:57.2919213Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2024-04-03T17:18:57.2921179Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2024-04-03T17:18:57.2923305Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping 2024-04-03T17:18:57.2925614Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2024-04-03T17:18:57.2927917Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2024-04-03T17:18:57.2930024Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook 2024-04-03T17:18:57.2932041Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2024-04-03T17:18:57.2933978Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2024-04-03T17:18:57.2936181Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2024-04-03T17:18:57.2938570Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params 2024-04-03T17:18:57.2941111Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2024-04-03T17:18:57.2943374Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2024-04-03T17:18:57.2945331Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states 2024-04-03T17:18:57.2947255Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2024-04-03T17:18:57.2949273Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_new_group 2024-04-03T17:18:57.2951253Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2024-04-03T17:18:57.2953080Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2024-04-03T17:18:57.2955410Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group 2024-04-03T17:18:57.2957323Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2024-04-03T17:18:57.2959297Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2024-04-03T17:18:57.2961546Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2024-04-03T17:18:57.2963861Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2024-04-03T17:18:57.2966103Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2024-04-03T17:18:57.2967963Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2024-04-03T17:18:57.2969909Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-04-03T17:18:57.2971035Z 2024-04-03T17:18:57.2971470Z Running distributed tests for the ucc backend with file init_method 2024-04-03T17:18:57.2974309Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:18:57.283902] 2024-04-03T17:23:34.8586779Z 2024-04-03T17:23:34.8589275Z distributed/test_distributed_spawn 6/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.7_d04a10a5641a7263_.log 2024-04-03T17:23:34.8630678Z Running 43 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_new_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-04-03T17:23:34.8666492Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2024-04-03T17:23:34.8668940Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2024-04-03T17:23:34.8671131Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2024-04-03T17:23:34.8673155Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2024-04-03T17:23:34.8675831Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2024-04-03T17:23:34.8677864Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2024-04-03T17:23:34.8679916Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup 2024-04-03T17:23:34.8681920Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min 2024-04-03T17:23:34.8683942Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2024-04-03T17:23:34.8685997Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2024-04-03T17:23:34.8688130Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2024-04-03T17:23:34.8690268Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2024-04-03T17:23:34.8692434Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group 2024-04-03T17:23:34.8694644Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2024-04-03T17:23:34.8696868Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2024-04-03T17:23:34.8698822Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2024-04-03T17:23:34.8700857Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2024-04-03T17:23:34.8702804Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2024-04-03T17:23:34.8704654Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2024-04-03T17:23:34.8706598Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2024-04-03T17:23:34.8708734Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping 2024-04-03T17:23:34.8711008Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2024-04-03T17:23:34.8713298Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2024-04-03T17:23:34.8716089Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook 2024-04-03T17:23:34.8718110Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2024-04-03T17:23:34.8720065Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2024-04-03T17:23:34.8722401Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2024-04-03T17:23:34.8724849Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params 2024-04-03T17:23:34.8727240Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2024-04-03T17:23:34.8729492Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2024-04-03T17:23:34.8731427Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states 2024-04-03T17:23:34.8733349Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2024-04-03T17:23:34.8735377Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_update_process_group_new_group 2024-04-03T17:23:34.8737361Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2024-04-03T17:23:34.8739160Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2024-04-03T17:23:34.8740965Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group 2024-04-03T17:23:34.8742857Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2024-04-03T17:23:34.8744855Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2024-04-03T17:23:34.8747227Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2024-04-03T17:23:34.8749544Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2024-04-03T17:23:34.8751559Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2024-04-03T17:23:34.8753432Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2024-04-03T17:23:34.8756043Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-04-03T17:23:34.8757186Z 2024-04-03T17:23:34.8757897Z Running distributed/checkpoint/test_fsdp_optim_state 1/1 ... [2024-04-03 17:23:34.861536] 2024-04-03T17:23:34.8760819Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_optim_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:23:34.861996] 2024-04-03T17:23:48.6051383Z 2024-04-03T17:23:48.6053589Z distributed/checkpoint/test_fsdp_optim_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_optim_state_1.1_c354951dd721c0c0_.log 2024-04-03T17:23:48.6057538Z Running 2 items in this shard: test/distributed/checkpoint/test_fsdp_optim_state.py::FsdpOptimStateCheckpoint::test_load_sharded_optimizer_state_dict_pass_planner_False, test/distributed/checkpoint/test_fsdp_optim_state.py::FsdpOptimStateCheckpoint::test_load_sharded_optimizer_state_dict_pass_planner_True 2024-04-03T17:23:48.6059992Z 2024-04-03T17:23:48.6060541Z Running distributed/_tensor/test_api 1/1 ... [2024-04-03 17:23:48.605464] 2024-04-03T17:23:48.6063981Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:23:48.605919] 2024-04-03T17:24:21.8443132Z 2024-04-03T17:24:21.8445002Z distributed/_tensor/test_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_api_1.1_7344f45c4148547c_.log 2024-04-03T17:24:21.8454464Z Running 8 items in this shard: test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_casting, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn_warning, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_meta, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor_errors, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor_uneven_sharding 2024-04-03T17:24:21.8459905Z 2024-04-03T17:24:21.8460573Z Running distributed/_tensor/test_random_ops 1/1 ... [2024-04-03 17:24:21.844699] 2024-04-03T17:24:21.8463356Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_random_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:24:21.845165] 2024-04-03T17:24:55.2310729Z 2024-04-03T17:24:55.2312945Z distributed/_tensor/test_random_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_random_ops_1.1_3c0d1182670c4ac0_.log 2024-04-03T17:24:55.2320045Z Running 7 items in this shard: test/distributed/_tensor/test_random_ops.py::DistTensorRandomInitTest::test_init_ops, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_dropout_1d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_rand_1d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_uniform_2d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_manual_seed, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_meta_tensor_init, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_rng_tracker_init 2024-04-03T17:24:55.2325554Z 2024-04-03T17:24:55.2326290Z Running distributed/checkpoint/test_dtensor_resharding 1/1 ... [2024-04-03 17:24:55.231383] 2024-04-03T17:24:55.2329255Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dtensor_resharding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:24:55.231955] 2024-04-03T17:25:27.6148591Z 2024-04-03T17:25:27.6150832Z distributed/checkpoint/test_dtensor_resharding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dtensor_resharding_1.1_79b4d4b60707006a_.log 2024-04-03T17:25:27.6157852Z Running 5 items in this shard: test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_1d_to_1d_reshard_placement_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_2d_to_2d_reshard_placement_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_1d_to_2d_reshard_mesh_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_2d_to_1d_reshard_mesh_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_dtensor_checkpoint_resharding_with_empty_shard 2024-04-03T17:25:27.6163044Z 2024-04-03T17:25:27.6163814Z Running distributed/_tensor/experimental/test_tp_transform 1/1 ... [2024-04-03 17:25:27.615160] 2024-04-03T17:25:27.6167140Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/experimental/test_tp_transform.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:25:27.615646] 2024-04-03T17:26:01.2524961Z 2024-04-03T17:26:01.2529212Z distributed/_tensor/experimental/test_tp_transform 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.experimental.test_tp_transform_1.1_83395c53a8e854fa_.log 2024-04-03T17:26:01.2537503Z Running 6 items in this shard: test/distributed/_tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_e2e_use_native_funcol_False, test/distributed/_tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_e2e_use_native_funcol_True, test/distributed/_tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_no_bias_use_native_funcol_False, test/distributed/_tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_no_bias_use_native_funcol_True, test/distributed/_tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_with_uncovered_op_use_native_funcol_False, test/distributed/_tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_with_uncovered_op_use_native_funcol_True 2024-04-03T17:26:01.2543486Z 2024-04-03T17:26:01.2544268Z Running distributed/_composable/fsdp/test_fully_shard_autograd 1/1 ... [2024-04-03 17:26:01.252806] 2024-04-03T17:26:01.2547343Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_autograd.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:26:01.253287] 2024-04-03T17:26:34.9396306Z 2024-04-03T17:26:34.9398670Z distributed/_composable/fsdp/test_fully_shard_autograd 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_autograd_1.1_dafd9e0dbb9939e9_.log 2024-04-03T17:26:34.9403827Z Running 3 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardAutograd::test_nontensor_activations, test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardAutograd::test_unused_forward_module, test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardAutograd::test_unused_forward_output 2024-04-03T17:26:34.9406972Z 2024-04-03T17:26:34.9407963Z Running distributed/test_compute_comm_reordering 1/1 ... [2024-04-03 17:26:34.939978] 2024-04-03T17:26:34.9410823Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_compute_comm_reordering.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:26:34.940448] 2024-04-03T17:26:42.5203523Z 2024-04-03T17:26:42.5205616Z distributed/test_compute_comm_reordering 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_compute_comm_reordering_1.1_378fedd2a529c1a4_.log 2024-04-03T17:26:42.5212570Z Running 6 items in this shard: test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_nccl_heuristics, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_raise_comms, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap_custom_runtime_estimation, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits_raise_comms 2024-04-03T17:26:42.5218413Z 2024-04-03T17:26:42.5219212Z Running distributed/tensor/parallel/test_ddp_2d_parallel 1/1 ... [2024-04-03 17:26:42.520685] 2024-04-03T17:26:42.5222210Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_ddp_2d_parallel.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:26:42.521154] 2024-04-03T17:26:49.9508866Z 2024-04-03T17:26:49.9511331Z distributed/tensor/parallel/test_ddp_2d_parallel 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_ddp_2d_parallel_1.1_7e624b7748f90337_.log 2024-04-03T17:26:49.9514512Z Running 1 items in this shard: test/distributed/tensor/parallel/test_ddp_2d_parallel.py::Test2dParallelIntegration::test_2d_ddp_integration_functionality 2024-04-03T17:26:49.9515903Z 2024-04-03T17:26:49.9516553Z Running distributed/checkpoint/test_fsspec 1/1 ... [2024-04-03 17:26:49.951181] 2024-04-03T17:26:49.9522193Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsspec.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:26:49.951695] 2024-04-03T17:26:59.1347186Z 2024-04-03T17:26:59.1349195Z distributed/checkpoint/test_fsspec 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsspec_1.1_d310a05cd22651d1_.log 2024-04-03T17:26:59.1352238Z Running 2 items in this shard: test/distributed/checkpoint/test_fsspec.py::TestFSSpecNoDist::test_fsspec_no_dist, test/distributed/checkpoint/test_fsspec.py::TestFSSpecWithDist::test_fsspec_with_dist 2024-04-03T17:26:59.1354588Z 2024-04-03T17:26:59.1355613Z Running distributed/_tensor/test_experimental_ops 1/1 ... [2024-04-03 17:26:59.135174] 2024-04-03T17:26:59.1362663Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_experimental_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:26:59.135731] 2024-04-03T17:27:16.6387391Z 2024-04-03T17:27:16.6389695Z distributed/_tensor/test_experimental_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_experimental_ops_1.1_ad75fbdc7e9d1cf5_.log 2024-04-03T17:27:16.6393281Z Running 3 items in this shard: test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_bernoulli, test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_nll, test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_slice 2024-04-03T17:27:16.6395702Z 2024-04-03T17:27:16.6396314Z Running distributed/fsdp/test_fsdp_memory 1/1 ... [2024-04-03 17:27:16.639023] 2024-04-03T17:27:16.6400646Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:27:16.639551] 2024-04-03T17:27:34.2424717Z 2024-04-03T17:27:34.2427574Z distributed/fsdp/test_fsdp_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_memory_1.1_4ae9041f2b2dc8af_.log 2024-04-03T17:27:34.2430611Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_memory.py::TestFSDPMemory::test_fsdp_memory_ckpt_ckpt, test/distributed/fsdp/test_fsdp_memory.py::TestFSDPMemory::test_fsdp_memory_ckpt_no_ckpt 2024-04-03T17:27:34.2432283Z 2024-04-03T17:27:34.2432978Z Running distributed/_tensor/test_convolution_ops 1/1 ... [2024-04-03 17:27:34.242794] 2024-04-03T17:27:34.2438083Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_convolution_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:27:34.243246] 2024-04-03T17:28:15.5971920Z 2024-04-03T17:28:15.5974004Z distributed/_tensor/test_convolution_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_convolution_ops_1.1_1dbec8f6ffda35bd_.log 2024-04-03T17:28:15.5978806Z Running 2 items in this shard: test/distributed/_tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_depthwise_convolution, test/distributed/_tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_downsampling_convolution 2024-04-03T17:28:15.5980871Z 2024-04-03T17:28:15.5981561Z Running distributed/_tensor/test_redistribute 1/1 ... [2024-04-03 17:28:15.597505] 2024-04-03T17:28:15.5984418Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_redistribute.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:28:15.597962] 2024-04-03T17:28:56.4047835Z 2024-04-03T17:28:56.4050238Z distributed/_tensor/test_redistribute 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_redistribute_1.1_bd672dde1b5b1d47_.log 2024-04-03T17:28:56.4059241Z Running 10 items in this shard: test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_partial_to_replicate_forward_backward, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_partial_to_shard, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_redistribute_negative_shard_dim, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_redistribute_uneven_sharding, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_local_partial_grad, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_partial, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_replicate_forward_backward, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_shard_forward_backward, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_shard_to_replicate_forward_backward, test/distributed/_tensor/test_redistribute.py::MultiDimRedistributeTest::test_multi_dim_mesh 2024-04-03T17:28:56.4067634Z 2024-04-03T17:28:56.4068238Z Running distributed/test_collective_utils 1/1 ... [2024-04-03 17:28:56.405110] 2024-04-03T17:28:56.4070968Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_collective_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:28:56.405600] 2024-04-03T17:28:59.7405405Z 2024-04-03T17:28:59.7407436Z distributed/test_collective_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_collective_utils_1.1_001224b090664ae4_.log 2024-04-03T17:28:59.7409066Z 2024-04-03T17:28:59.7412223Z Running distributed/tensor/parallel/test_tp_random_state 1/1 ... [2024-04-03 17:28:59.740837] 2024-04-03T17:28:59.7418191Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_random_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:28:59.741321] 2024-04-03T17:29:07.2217564Z 2024-04-03T17:29:07.2220047Z distributed/tensor/parallel/test_tp_random_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_random_state_1.1_41e034a5d8b7a1cb_.log 2024-04-03T17:29:07.2222921Z Running 1 items in this shard: test/distributed/tensor/parallel/test_tp_random_state.py::TensorParallelRandomStateTests::test_model_init 2024-04-03T17:29:07.2224500Z 2024-04-03T17:29:07.2225123Z Running distributed/_tensor/test_math_ops 1/1 ... [2024-04-03 17:29:07.222026] 2024-04-03T17:29:07.2230033Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_math_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:29:07.222490] 2024-04-03T17:29:53.2574946Z 2024-04-03T17:29:53.2579449Z distributed/_tensor/test_math_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_math_ops_1.1_9924f64b0a02ac38_.log 2024-04-03T17:29:53.2585783Z Running 8 items in this shard: test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_bwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_fwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_linear_op_reductions, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_mean, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_nll_loss_and_cross_entropy, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_shard_math_ops, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_softmax_fwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_softmax_with_bwd 2024-04-03T17:29:53.2590780Z 2024-04-03T17:29:53.2591517Z Running distributed/checkpoint/e2e/test_e2e_save_and_load 1/1 ... [2024-04-03 17:29:53.257794] 2024-04-03T17:29:53.2595298Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_e2e_save_and_load.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:29:53.258244] 2024-04-03T17:30:34.9076902Z 2024-04-03T17:30:34.9081341Z distributed/checkpoint/e2e/test_e2e_save_and_load 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_e2e_save_and_load_1.1_2df3ffe0c8fbab1b_.log 2024-04-03T17:30:34.9092141Z Running 10 items in this shard: test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_different_ordered_state_dict_keys, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_async, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_False_model_type0, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_False_model_type1, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_False_model_type2, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_True_model_type0, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_True_model_type1, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_True_model_type2, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_no_dist, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestNoCPU::test_no_cpu 2024-04-03T17:30:34.9100825Z 2024-04-03T17:30:34.9101623Z Running distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 ... [2024-04-03 17:30:34.908032] 2024-04-03T17:30:34.9104717Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:30:34.908494] 2024-04-03T17:30:50.5556085Z 2024-04-03T17:30:50.5558784Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_dict_1.1_53914cc5d59e491a_.log 2024-04-03T17:30:50.5562943Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDict::test_1d_state_dict_save_load, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDict::test_2d_state_dict_save_load 2024-04-03T17:30:50.5565224Z 2024-04-03T17:30:50.5566063Z Running distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 ... [2024-04-03 17:30:50.556045] 2024-04-03T17:30:50.5570045Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:30:50.556514] 2024-04-03T17:31:21.7311746Z 2024-04-03T17:31:21.7314608Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_9e3bec3530bcd5c1_.log 2024-04-03T17:31:21.7322736Z Running 6 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_compute_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_reduce_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_float16_on_one_submodule, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_norm_modules_bf16, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_norm_modules_fp16, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_submodules_with_external_inputs 2024-04-03T17:31:21.7329446Z 2024-04-03T17:31:21.7330198Z Running distributed/checkpoint/test_state_dict_utils 1/1 ... [2024-04-03 17:31:21.731586] 2024-04-03T17:31:21.7333114Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_state_dict_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:31:21.732128] 2024-04-03T17:31:45.2445251Z 2024-04-03T17:31:45.2447434Z distributed/checkpoint/test_state_dict_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_state_dict_utils_1.1_52b84cca89d6a5f4_.log 2024-04-03T17:31:45.2466249Z Running 5 items in this shard: test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_complicated_dict, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_cpu_and_ranks_only, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_create_cpu_state_dict, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_gather_state_dict_dtensor, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_gather_with_cpu_and_ranks_only 2024-04-03T17:31:45.2470507Z 2024-04-03T17:31:45.2471306Z Running distributed/tensor/parallel/test_fsdp_2d_parallel 1/1 ... [2024-04-03 17:31:45.244890] 2024-04-03T17:31:45.2475124Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_fsdp_2d_parallel.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:31:45.245410] 2024-04-03T17:32:21.8893326Z 2024-04-03T17:32:21.8901356Z distributed/tensor/parallel/test_fsdp_2d_parallel 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_fsdp_2d_parallel_1.1_453fccc239e69a0a_.log 2024-04-03T17:32:21.8917602Z Running 9 items in this shard: test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_2d_e2e_training_default, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_2d_e2e_training_not_use_orig_params, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_2d_e2e_training_use_orig_params, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_2d_fsdp_state_enable_extension, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_raise_invalid_tp_composition, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_2d_load_state_dict_is_even_sharded_model_True, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_2d_optim_state_dict_is_even_sharded_model_True, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_2d_state_dict_is_even_sharded_model_True, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_fsdp_2d_extension 2024-04-03T17:32:21.8927399Z 2024-04-03T17:32:21.8927965Z Running distributed/test_fake_pg 1/1 ... [2024-04-03 17:32:21.889807] 2024-04-03T17:32:21.8930596Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_fake_pg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:32:21.890494] 2024-04-03T17:32:26.4646551Z 2024-04-03T17:32:26.4648557Z distributed/test_fake_pg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_fake_pg_1.1_4bc808b8a6003c0a_.log 2024-04-03T17:32:26.4656779Z Running 13 items in this shard: test/distributed/test_fake_pg.py::TestFakePG::test_all_reduce, test/distributed/test_fake_pg.py::TestFakePG::test_allgather, test/distributed/test_fake_pg.py::TestFakePG::test_alltoall, test/distributed/test_fake_pg.py::TestFakePG::test_alltoall_base, test/distributed/test_fake_pg.py::TestFakePG::test_broadcast, test/distributed/test_fake_pg.py::TestFakePG::test_construct_fsdp, test/distributed/test_fake_pg.py::TestFakePG::test_fake_pg_tracing, test/distributed/test_fake_pg.py::TestFakePG::test_fsdp_fake_e2e, test/distributed/test_fake_pg.py::TestFakePG::test_fsdp_tp_fake_e2e, test/distributed/test_fake_pg.py::TestFakePG::test_recv, test/distributed/test_fake_pg.py::TestFakePG::test_reduce_scatter, test/distributed/test_fake_pg.py::TestFakePG::test_scatter, test/distributed/test_fake_pg.py::TestFakePG::test_send 2024-04-03T17:32:26.4663757Z 2024-04-03T17:32:26.4664552Z Running distributed/tensor/parallel/test_parallelize_api 1/1 ... [2024-04-03 17:32:26.465008] 2024-04-03T17:32:26.4667563Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_parallelize_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:32:26.465482] 2024-04-03T17:32:52.4878652Z 2024-04-03T17:32:52.4883184Z distributed/tensor/parallel/test_parallelize_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_parallelize_api_1.1_298d3fc44e7f6a08_.log 2024-04-03T17:32:52.4891250Z Running 6 items in this shard: test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_linear_col_wise_parallel, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_linear_row_wise_parallel, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_mlp_with_module_api, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_mlp_with_module_api_nested, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_prepare_module_input, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_prepare_module_output 2024-04-03T17:32:52.4902421Z 2024-04-03T17:32:52.4904022Z Running distributed/_composable/fully_shard/test_fully_shard_util 1/1 ... [2024-04-03 17:32:52.488166] 2024-04-03T17:32:52.4910235Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_util.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:32:52.488634] 2024-04-03T17:33:00.9705423Z 2024-04-03T17:33:00.9709365Z distributed/_composable/fully_shard/test_fully_shard_util 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_util_1.1_637f1b58e0ddbc2a_.log 2024-04-03T17:33:00.9713927Z Running 1 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_util.py::TestUtils::test_get_sharded_module_tree_with_module_name_to_fqns 2024-04-03T17:33:00.9715785Z 2024-04-03T17:33:00.9716739Z Running distributed/test_c10d_pypg 1/1 ... [2024-04-03 17:33:00.970838] 2024-04-03T17:33:00.9719995Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:33:00.971320] 2024-04-03T17:35:40.3055596Z 2024-04-03T17:35:40.3057482Z distributed/test_c10d_pypg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_pypg_1.1_6fc1928ee185ac1d_.log 2024-04-03T17:35:40.3094380Z Running 42 items in this shard: test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_dataclass_output, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_dataclass_output_unused_param, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_invoke_work_object, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_with_pypg, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_with_pypg_with_grad_views, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_invalid_powerSGD_state, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_sync_batch_norm_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_dataclass_output, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_dataclass_output_unused_param, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_invoke_work_object, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_with_pypg, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_with_pypg_with_grad_views, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_invalid_powerSGD_state, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_sync_batch_norm_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_sync_batch_norm_only_empty_input 2024-04-03T17:35:40.3127149Z 2024-04-03T17:35:40.3127922Z Running distributed/fsdp/test_distributed_checkpoint 1/1 ... [2024-04-03 17:35:40.305956] 2024-04-03T17:35:40.3130849Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:35:40.306461] 2024-04-03T17:35:44.3290472Z 2024-04-03T17:35:44.3292588Z distributed/fsdp/test_distributed_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_distributed_checkpoint_1.1_f1243b6507af6cfd_.log 2024-04-03T17:35:44.3296871Z Running 2 items in this shard: test/distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpoint::test_distributed_checkpoint_state_dict_type0, test/distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpoint::test_distributed_checkpoint_state_dict_type1 2024-04-03T17:35:44.3299155Z 2024-04-03T17:35:44.3299851Z Running distributed/_tensor/test_utils 1/1 ... [2024-04-03 17:35:44.329393] 2024-04-03T17:35:44.3303420Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:35:44.329874] 2024-04-03T17:35:56.0730400Z 2024-04-03T17:35:56.0732415Z distributed/_tensor/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_utils_1.1_e09c6285312fccae_.log 2024-04-03T17:35:56.0736393Z Running 3 items in this shard: test/distributed/_tensor/test_utils.py::UtilTest::test_compute_local_shape_2d_uneven, test/distributed/_tensor/test_utils.py::UtilTest::test_compute_local_shape_and_global_offset_1D, test/distributed/_tensor/test_utils.py::UtilTest::test_compute_local_shape_and_global_offset_2D 2024-04-03T17:35:56.0738794Z 2024-04-03T17:35:56.0739558Z Running distributed/_composable/fsdp/test_fully_shard_comm 1/1 ... [2024-04-03 17:35:56.073274] 2024-04-03T17:35:56.0742542Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_comm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:35:56.073734] 2024-04-03T17:36:36.6226639Z 2024-04-03T17:36:36.6229254Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_comm_1.1_c6eb30ab33bb190c_.log 2024-04-03T17:36:36.6236774Z Running 4 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_all_gather_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_reduce_scatter_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_fully_shard_communication_count, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardBackwardPrefetch::test_fully_shard_backward_prefetch 2024-04-03T17:36:36.6240698Z 2024-04-03T17:36:36.6241628Z Running distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 1/1 ... [2024-04-03 17:36:36.622958] 2024-04-03T17:36:36.6244879Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_model_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:36:36.623621] 2024-04-03T17:37:02.4915333Z 2024-04-03T17:37:02.4917895Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_model_checkpoint_1.1_36c76c429cf71c74_.log 2024-04-03T17:37:02.4925036Z Running 4 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_model_checkpoint.py::TestModelCheckpointing::test_full_state_dict_save_load_mixed_sharding, test/distributed/_composable/fully_shard/test_fully_shard_model_checkpoint.py::TestModelCheckpointing::test_state_dict_save_load_flow, test/distributed/_composable/fully_shard/test_fully_shard_model_checkpoint.py::TestModelCheckpointing::test_state_dict_save_load_root_fully_shard, test/distributed/_composable/fully_shard/test_fully_shard_model_checkpoint.py::TestModelCheckpointing::test_state_dict_save_load_submodule_fully_shard 2024-04-03T17:37:02.4929673Z 2024-04-03T17:37:02.4930330Z Running distributed/checkpoint/test_utils 1/1 ... [2024-04-03 17:37:02.491867] 2024-04-03T17:37:02.4933122Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:37:02.492328] 2024-04-03T17:37:06.5148054Z 2024-04-03T17:37:06.5150030Z distributed/checkpoint/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_utils_1.1_1032277d79a86a08_.log 2024-04-03T17:37:06.5156030Z Running 5 items in this shard: test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_flat_data, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_index_hint_ignored_on_equals, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_index_hint_ignored_on_hash, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_init_convert_offset, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_sharded_tensor_lookup 2024-04-03T17:37:06.5160146Z 2024-04-03T17:37:06.5161127Z Running distributed/_tensor/test_view_ops 1/1 ... [2024-04-03 17:37:06.515109] 2024-04-03T17:37:06.5163902Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_view_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:37:06.515595] 2024-04-03T17:37:21.9647601Z 2024-04-03T17:37:21.9649526Z distributed/_tensor/test_view_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_view_ops_1.1_e90c3601af8ea970_.log 2024-04-03T17:37:21.9653336Z Running 3 items in this shard: test/distributed/_tensor/test_view_ops.py::TestViewOps::test_complex_view_ops, test/distributed/_tensor/test_view_ops.py::TestViewOps::test_view_groups, test/distributed/_tensor/test_view_ops.py::TestViewOps::test_view_ops 2024-04-03T17:37:21.9655368Z 2024-04-03T17:37:21.9656537Z Running distributed/_composable/fully_shard/test_fully_shard_init 1/1 ... [2024-04-03 17:37:21.965046] 2024-04-03T17:37:21.9660100Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:37:21.965516] 2024-04-03T17:37:52.8982171Z 2024-04-03T17:37:52.8986713Z distributed/_composable/fully_shard/test_fully_shard_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_init_1.1_486bb13dbca9b5a9_.log 2024-04-03T17:37:52.8993540Z Running 6 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_device_id, test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_manual_fully_shard, test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_materialize_meta_module, test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_nested_fully_shard_shared_state, test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_policy, test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_sync_module_states 2024-04-03T17:37:52.8999321Z 2024-04-03T17:37:52.8999915Z Running distributed/_shard/test_sharder 1/1 ... [2024-04-03 17:37:52.898494] 2024-04-03T17:37:52.9002638Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:37:52.898963] 2024-04-03T17:38:04.1885195Z 2024-04-03T17:38:04.1887405Z distributed/_shard/test_sharder 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.test_sharder_1.1_4c23b15150b8eff1_.log 2024-04-03T17:38:04.1890316Z Running 2 items in this shard: test/distributed/_shard/test_sharder.py::TestCustomSharder::test_custom_sharder, test/distributed/_shard/test_sharder.py::TestCustomSharder::test_custom_sharder_errors 2024-04-03T17:38:04.1891921Z 2024-04-03T17:38:04.1892531Z Running distributed/fsdp/test_fsdp_overlap 1/1 ... [2024-04-03 17:38:04.188787] 2024-04-03T17:38:04.1897356Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:38:04.189240] 2024-04-03T17:38:52.2417188Z 2024-04-03T17:38:52.2421456Z distributed/fsdp/test_fsdp_overlap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_overlap_1.1_4bda38cddd958f56_.log 2024-04-03T17:38:52.2425103Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_overlap.py::TestForwardOverlapWorldSizeOne::test_forward_overlap, test/distributed/fsdp/test_fsdp_overlap.py::TestForwardOverlapWorldSizeTwo::test_forward_overlap 2024-04-03T17:38:52.2427047Z 2024-04-03T17:38:52.2427610Z Running distributed/test_c10d_logger 1/1 ... [2024-04-03 17:38:52.242055] 2024-04-03T17:38:52.2430282Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:38:52.242514] 2024-04-03T17:39:13.5020228Z 2024-04-03T17:39:13.5022432Z distributed/test_c10d_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_logger_1.1_d2b39a2ca845e49b_.log 2024-04-03T17:39:13.5026160Z Running 3 items in this shard: test/distributed/test_c10d_logger.py::C10dErrorLoggerTest::test_exception_logger, test/distributed/test_c10d_logger.py::C10dErrorLoggerTest::test_get_or_create_logger, test/distributed/test_c10d_logger.py::C10dErrorLoggerTest::test_time_logger 2024-04-03T17:39:13.5028356Z 2024-04-03T17:39:13.5029116Z Running distributed/_shard/sharding_spec/test_sharding_spec 1/1 ... [2024-04-03 17:39:13.502404] 2024-04-03T17:39:13.5034821Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:39:13.502910] 2024-04-03T17:39:29.5543792Z 2024-04-03T17:39:29.5546031Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharding_spec.test_sharding_spec_1.1_0e1a350bf3926547_.log 2024-04-03T17:39:29.5557527Z Running 11 items in this shard: test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_check_overlapping, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_chunked_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_device_placement, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_enumerable_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_chunk_sharding_params, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_chunked_dim_size, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_split_size, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_infer_sharding_spec_from_shards_metadata, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec_shard_tensor, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec_tensor_ctor 2024-04-03T17:39:29.5567014Z 2024-04-03T17:39:29.5567725Z Running distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 ... [2024-04-03 17:39:29.554661] 2024-04-03T17:39:29.5570654Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_dtensor_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:39:29.555116] 2024-04-03T17:40:44.8464884Z 2024-04-03T17:40:44.8467090Z distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_dtensor_state_dict_1.1_17a11afe6953f7e3_.log 2024-04-03T17:40:44.8489781Z Running 15 items in this shard: test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_False_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_False_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_True_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_True_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_False_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_False_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_True_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_True_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_False_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_False_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_True_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_True_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_fsdp_init_with_device_mesh_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_fsdp_init_with_device_mesh_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_raises_warning_or_errors 2024-04-03T17:40:44.8509233Z 2024-04-03T17:40:44.8509978Z Running distributed/fsdp/test_fsdp_tp_integration 1/1 ... [2024-04-03 17:40:44.846798] 2024-04-03T17:40:44.8512847Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:40:44.847261] 2024-04-03T17:41:15.7259946Z 2024-04-03T17:41:15.7262104Z distributed/fsdp/test_fsdp_tp_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_9d0d3afa569f6f6b_.log 2024-04-03T17:41:15.7269338Z Running 6 items in this shard: test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_extension_grad, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload0, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload1, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload0, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload1, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_sync_module_state 2024-04-03T17:41:15.7275697Z 2024-04-03T17:41:15.7276341Z Running distributed/test_multi_threaded_pg 1/1 ... [2024-04-03 17:41:15.726299] 2024-04-03T17:41:15.7279096Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_multi_threaded_pg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:41:15.726775] 2024-04-03T17:41:20.0000860Z 2024-04-03T17:41:20.0003091Z distributed/test_multi_threaded_pg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_multi_threaded_pg_1.1_dfedede533155b3c_.log 2024-04-03T17:41:20.0018185Z Running 19 items in this shard: test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_broadcast_object_list, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_non_zero, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_non_zero_all, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_zero, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_skip, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce_coalesced, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce_ops, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_to_all, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_allgather, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_assert_equal_on_rank, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_broadcast, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_broadcast_object_list, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_bwd_sees_fwd_pg, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_gather, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_reduce_scatter, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_scatter, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_subpg, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_using_pg_from_another_thread 2024-04-03T17:41:20.0032809Z 2024-04-03T17:41:20.0033447Z Running distributed/nn/jit/test_instantiator 1/1 ... [2024-04-03 17:41:20.000430] 2024-04-03T17:41:20.0036561Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:41:20.000915] 2024-04-03T17:41:23.9735793Z 2024-04-03T17:41:23.9737995Z distributed/nn/jit/test_instantiator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.nn.jit.test_instantiator_1.1_cdedd5e02cc57d95_.log 2024-04-03T17:41:23.9742271Z Running 3 items in this shard: test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_get_arg_return_types_from_interface, test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_instantiate_non_scripted_remote_module_template, test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_instantiate_scripted_remote_module_template 2024-04-03T17:41:23.9745124Z 2024-04-03T17:41:23.9745887Z Running distributed/checkpoint/test_file_system_checkpoint 1/1 ... [2024-04-03 17:41:23.973894] 2024-04-03T17:41:23.9749254Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:41:23.974341] 2024-04-03T17:41:54.1021101Z 2024-04-03T17:41:54.1025675Z distributed/checkpoint/test_file_system_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_1.1_a0a683dd5d4824bb_.log 2024-04-03T17:41:54.1033387Z Running 6 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_save_load_bytes, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor 2024-04-03T17:41:54.1039819Z 2024-04-03T17:41:54.1040469Z Running distributed/checkpoint/test_planner 1/1 ... [2024-04-03 17:41:54.102433] 2024-04-03T17:41:54.1043294Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_planner.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:41:54.102889] 2024-04-03T17:41:58.1263187Z 2024-04-03T17:41:58.1265216Z distributed/checkpoint/test_planner 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_planner_1.1_5147e8d00762951a_.log 2024-04-03T17:41:58.1271256Z Running 6 items in this shard: test/distributed/checkpoint/test_planner.py::TestSavePlan::test_global_plan, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_load_with_resharding, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_load_with_world_size_diff_by_one, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_local_load_plan, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_local_plan, test/distributed/checkpoint/test_planner.py::TestPlannerHelpers::test_create_read_item_from_chunks 2024-04-03T17:41:58.1275915Z 2024-04-03T17:41:58.1276549Z Running distributed/fsdp/test_fsdp_apply 1/1 ... [2024-04-03 17:41:58.126660] 2024-04-03T17:41:58.1279285Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:41:58.127159] 2024-04-03T17:42:16.5805746Z 2024-04-03T17:42:16.5810208Z distributed/fsdp/test_fsdp_apply 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_apply_1.1_ddf51abf49085802_.log 2024-04-03T17:42:16.5813758Z Running 3 items in this shard: test/distributed/fsdp/test_fsdp_apply.py::TestApply::test_apply_in_summon_raises_error, test/distributed/fsdp/test_fsdp_apply.py::TestApply::test_nested_module_apply, test/distributed/fsdp/test_fsdp_apply.py::TestApply::test_transformer_module_apply 2024-04-03T17:42:16.5815968Z 2024-04-03T17:42:16.5816643Z Running distributed/pipeline/sync/test_pipeline 1/1 ... [2024-04-03 17:42:16.580989] 2024-04-03T17:42:16.5819531Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_pipeline.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:42:16.581444] 2024-04-03T17:42:19.2022057Z 2024-04-03T17:42:19.2024808Z distributed/pipeline/sync/test_pipeline 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_pipeline_1.1_f9d25d3cdb22d1fd_.log 2024-04-03T17:42:19.2027204Z Running 1 items in this shard: test/distributed/pipeline/sync/test_pipeline.py::test_clock_cycles 2024-04-03T17:42:19.2028095Z 2024-04-03T17:42:19.2028849Z Running distributed/_composable/test_replicate_with_compiler 1/1 ... [2024-04-03 17:42:19.202501] 2024-04-03T17:42:19.2034964Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_replicate_with_compiler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:42:19.202956] 2024-04-03T17:43:45.7105285Z 2024-04-03T17:43:45.7109707Z distributed/_composable/test_replicate_with_compiler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_replicate_with_compiler_1.1_91f365dfc44612bf_.log 2024-04-03T17:43:45.7118307Z Running 9 items in this shard: test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_bucketing_coalesced_op, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_bucketing_concat_op, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_backward_only, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_bf16, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_cpu, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_cpu_no_sync, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_fp16, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_gpu, test/distributed/_composable/test_replicate_with_compiler.py::DDP_TP_Test::test_ddp_tp 2024-04-03T17:43:45.7125276Z 2024-04-03T17:43:45.7125918Z Running distributed/fsdp/test_fsdp_state_dict 1/2 ... [2024-04-03 17:43:45.710805] 2024-04-03T17:43:45.7128737Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '--shard-id=1', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:43:45.711263] 2024-04-03T17:49:54.6603479Z 2024-04-03T17:49:54.6605697Z distributed/fsdp/test_fsdp_state_dict 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_state_dict_1.2_73471e1824b34cc5_.log 2024-04-03T17:49:54.6715159Z Running 78 items in this shard: test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_full_state_dict_missing_unexpected_keys_cleaned, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_rank0_offload_save_load_flow_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_type, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_torch_save_load, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict4GPUs::test_local_state_dict_reshard 2024-04-03T17:49:54.6820743Z 2024-04-03T17:49:54.6821385Z Running distributed/fsdp/test_fsdp_core 2/2 ... [2024-04-03 17:49:54.661368] 2024-04-03T17:49:54.6824309Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '--shard-id=2', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:49:54.661861] 2024-04-03T17:58:25.8542673Z 2024-04-03T17:58:25.8544610Z distributed/fsdp/test_fsdp_core 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_core_2.2_dcb13a4abc4e1c6a_.log 2024-04-03T17:58:25.8585983Z Running 38 items in this shard: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_pre_backward_hook_registration_after_state_dict, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_pre_backward_hook_registration_cuda_first_True, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_register_functions_called_cuda_first_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_register_functions_called_cuda_first_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_register_functions_called_cuda_first_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_register_functions_called_cuda_first_True_mixed_precision_True 2024-04-03T17:58:25.8618409Z 2024-04-03T17:58:25.8619228Z Running distributed/optim/test_apply_optimizer_in_backward 1/1 ... [2024-04-03 17:58:25.858316] 2024-04-03T17:58:25.8622272Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_apply_optimizer_in_backward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:58:25.858833] 2024-04-03T17:58:29.4153227Z 2024-04-03T17:58:29.4156566Z distributed/optim/test_apply_optimizer_in_backward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_apply_optimizer_in_backward_1.1_c897b3667a078c97_.log 2024-04-03T17:58:29.4158666Z 2024-04-03T17:58:29.4159939Z Running distributed/fsdp/test_utils 1/1 ... [2024-04-03 17:58:29.415661] 2024-04-03T17:58:29.4166305Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:58:29.416120] 2024-04-03T17:58:33.4385651Z 2024-04-03T17:58:33.4387936Z distributed/fsdp/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_utils_1.1_a3f6de6ffc6b2fda_.log 2024-04-03T17:58:33.4392379Z Running 5 items in this shard: test/distributed/fsdp/test_utils.py::TestUtils::test_apply_to_tensors_cpu_cuda, test/distributed/fsdp/test_utils.py::TestUtils::test_apply_to_tensors_devices0, test/distributed/fsdp/test_utils.py::TestUtils::test_apply_to_tensors_devices1, test/distributed/fsdp/test_utils.py::TestUtils::test_packed_sequence, test/distributed/fsdp/test_utils.py::TestUtils::test_replace_by_prefix 2024-04-03T17:58:33.4395923Z 2024-04-03T17:58:33.4396606Z Running distributed/pipeline/sync/test_phony 1/1 ... [2024-04-03 17:58:33.438840] 2024-04-03T17:58:33.4399422Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_phony.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:58:33.439291] 2024-04-03T17:58:36.0591091Z 2024-04-03T17:58:36.0593133Z distributed/pipeline/sync/test_phony 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_phony_1.1_ed45acac19d53683_.log 2024-04-03T17:58:36.0597359Z Running 4 items in this shard: test/distributed/pipeline/sync/test_phony.py::test_phony_size, test/distributed/pipeline/sync/test_phony.py::test_phony_requires_grad, test/distributed/pipeline/sync/test_phony.py::test_cached_phony, test/distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function 2024-04-03T17:58:36.0599890Z 2024-04-03T17:58:36.0600557Z Running distributed/test_c10d_object_collectives 1/1 ... [2024-04-03 17:58:36.059402] 2024-04-03T17:58:36.0603742Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:58:36.059900] 2024-04-03T17:59:17.8656185Z 2024-04-03T17:59:17.8660316Z distributed/test_c10d_object_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_object_collectives_1.1_5a8468029324d313_.log 2024-04-03T17:59:17.8668355Z Running 8 items in this shard: test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_all_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_broadcast_object_list, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_scatter_object_list, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_all_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_broadcast_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_scatter_object 2024-04-03T17:59:17.8681118Z 2024-04-03T17:59:17.8682426Z Running distributed/pipeline/sync/test_bugs 1/1 ... [2024-04-03 17:59:17.865893] 2024-04-03T17:59:17.8687964Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_bugs.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:59:17.866357] 2024-04-03T17:59:22.9925281Z 2024-04-03T17:59:22.9927617Z distributed/pipeline/sync/test_bugs 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_bugs_1.1_65d89ef60a1c1fd8_.log 2024-04-03T17:59:22.9932611Z Running 4 items in this shard: test/distributed/pipeline/sync/test_bugs.py::test_python_autograd_function, test/distributed/pipeline/sync/test_bugs.py::test_exception_no_hang, test/distributed/pipeline/sync/test_bugs.py::test_tuple_wait, test/distributed/pipeline/sync/test_bugs.py::test_parallel_randoms 2024-04-03T17:59:22.9935512Z 2024-04-03T17:59:22.9936260Z Running distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 ... [2024-04-03 17:59:22.992814] 2024-04-03T17:59:22.9939183Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_hsdp_dtensor_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:59:22.993285] 2024-04-03T17:59:55.8786852Z 2024-04-03T17:59:55.8790032Z distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_hsdp_dtensor_state_dict_1.1_f13a84b198a09c24_.log 2024-04-03T17:59:55.8803724Z Running 8 items in this shard: test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_False, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_True, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_False, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_True, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_False, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_True, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_hsdp_init_with_device_mesh, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_root_module_is_not_FSDP 2024-04-03T17:59:55.8813277Z 2024-04-03T17:59:55.8813996Z Running distributed/pipeline/sync/skip/test_gpipe 1/1 ... [2024-04-03 17:59:55.878977] 2024-04-03T17:59:55.8816955Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_gpipe.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:59:55.879474] 2024-04-03T18:00:00.4036990Z 2024-04-03T18:00:00.4039118Z distributed/pipeline/sync/skip/test_gpipe 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_gpipe_1.1_36dbbf1b5ff797ce_.log 2024-04-03T18:00:00.4050311Z Running 13 items in this shard: test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip 2024-04-03T18:00:00.4058280Z 2024-04-03T18:00:00.4058893Z Running distributed/algorithms/test_join 1/1 ... [2024-04-03 18:00:00.404213] 2024-04-03T18:00:00.4061999Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/test_join.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:00:00.404758] 2024-04-03T18:00:44.9681346Z 2024-04-03T18:00:44.9683310Z distributed/algorithms/test_join 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.test_join_1.1_0ed745940909f321_.log 2024-04-03T18:00:44.9693270Z Running 9 items in this shard: test/distributed/algorithms/test_join.py::TestJoin::test_join_kwargs, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinable_disable, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinables, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinables_throw, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_disable, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_main_hooks, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_post_hooks, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_throw 2024-04-03T18:00:44.9699035Z 2024-04-03T18:00:44.9699849Z Running distributed/_shard/sharded_tensor/ops/test_embedding 1/1 ... [2024-04-03 18:00:44.968438] 2024-04-03T18:00:44.9702898Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:00:44.968896] 2024-04-03T18:00:56.3583253Z 2024-04-03T18:00:56.3586004Z distributed/_shard/sharded_tensor/ops/test_embedding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_embedding_1.1_165e7556d6db7d41_.log 2024-04-03T18:00:56.3590733Z Running 2 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_embedding.py::TestShardedEmbedding::test_sharded_embedding_colwise, test/distributed/_shard/sharded_tensor/ops/test_embedding.py::TestShardedEmbedding::test_sharded_embedding_rowwise 2024-04-03T18:00:56.3593196Z 2024-04-03T18:00:56.3594047Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 ... [2024-04-03 18:00:56.358515] 2024-04-03T18:00:56.3597402Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:00:56.358978] 2024-04-03T18:01:07.7481720Z 2024-04-03T18:01:07.7484052Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_embedding_bag_1.1_e43996b427076e59_.log 2024-04-03T18:01:07.7488025Z Running 2 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_embedding_bag.py::TestShardedEmbeddingBag::test_sharded_embedding_bag_colwise, test/distributed/_shard/sharded_tensor/ops/test_embedding_bag.py::TestShardedEmbeddingBag::test_sharded_embedding_bag_rowwise 2024-04-03T18:01:07.7490304Z 2024-04-03T18:01:07.7490985Z Running distributed/rpc/cuda/test_tensorpipe_agent 2/2 ... [2024-04-03 18:01:07.748408] 2024-04-03T18:01:07.7493997Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '--shard-id=2', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:01:07.748864] 2024-04-03T18:12:04.9858883Z 2024-04-03T18:12:04.9861021Z distributed/rpc/cuda/test_tensorpipe_agent 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.cuda.test_tensorpipe_agent_2.2_88d51ac41e511ebc_.log 2024-04-03T18:12:04.9922757Z Running 56 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_input_moved_to_cuda_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_input_moved_to_cuda_device_script, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_always, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_except_last, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_never, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_always, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_async_execution_nested_with_cuda_future, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_callback_changes_devices, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_cuda_sparse_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_custom_class_with_cuda_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_int, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_str, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_modify_tensor_inplace, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_replace_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_value_on_bad_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_multi, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_nested_multi, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu_to_gpu_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu_to_gpu_non_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_7, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_8, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_7, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_non_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_to_cpu_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_gpu, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_max_local_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_max_remote_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_many_to_one, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_loop, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_not_timeout, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_remote_response, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_response, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_multi_gpu, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_multi_gpu_self, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_remote, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_devices_option_mismatch, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization5, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_with_unpickleable_attributes, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_tensor_view_as_return_value, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_device_maps_backward_pass 2024-04-03T18:12:04.9978893Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_input_moved_to_cuda_device 2024-04-03T18:12:04.9981284Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_input_moved_to_cuda_device_script 2024-04-03T18:12:04.9983568Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_always 2024-04-03T18:12:04.9985752Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_except_last 2024-04-03T18:12:04.9987941Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_never 2024-04-03T18:12:04.9990097Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_always 2024-04-03T18:12:04.9992645Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_async_execution_nested_with_cuda_future 2024-04-03T18:12:04.9995802Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_callback_changes_devices 2024-04-03T18:12:04.9998416Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_cuda_sparse_tensor 2024-04-03T18:12:05.0001183Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor 2024-04-03T18:12:05.0004027Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_custom_class_with_cuda_tensor 2024-04-03T18:12:05.0006641Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_device 2024-04-03T18:12:05.0009072Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_int 2024-04-03T18:12:05.0011475Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_str 2024-04-03T18:12:05.0013932Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_modify_tensor_inplace 2024-04-03T18:12:05.0016381Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_replace_tensor 2024-04-03T18:12:05.0019053Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_value_on_bad_device 2024-04-03T18:12:05.0021425Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream 2024-04-03T18:12:05.0023713Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_multi 2024-04-03T18:12:05.0026068Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_nested_multi 2024-04-03T18:12:05.0028516Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu_to_gpu_default 2024-04-03T18:12:05.0030997Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu_to_gpu_non_default 2024-04-03T18:12:05.0033446Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_7 2024-04-03T18:12:05.0036421Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_8 2024-04-03T18:12:05.0038834Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_1 2024-04-03T18:12:05.0041262Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_7 2024-04-03T18:12:05.0043682Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_non_default 2024-04-03T18:12:05.0046098Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_to_cpu_default 2024-04-03T18:12:05.0048617Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_gpu 2024-04-03T18:12:05.0051057Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_max_local_device 2024-04-03T18:12:05.0053632Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_max_remote_device 2024-04-03T18:12:05.0056102Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_many_to_one 2024-04-03T18:12:05.0058480Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config 2024-04-03T18:12:05.0060943Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_loop 2024-04-03T18:12:05.0063481Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_not_timeout 2024-04-03T18:12:05.0066093Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_remote_response 2024-04-03T18:12:05.0068666Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_response 2024-04-03T18:12:05.0071112Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_multi_gpu 2024-04-03T18:12:05.0073493Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_multi_gpu_self 2024-04-03T18:12:05.0076625Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_remote 2024-04-03T18:12:05.0078962Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_devices_option_mismatch 2024-04-03T18:12:05.0081425Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization2 2024-04-03T18:12:05.0083958Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization4 2024-04-03T18:12:05.0086448Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization1 2024-04-03T18:12:05.0089041Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization2 2024-04-03T18:12:05.0091515Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization3 2024-04-03T18:12:05.0093960Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization4 2024-04-03T18:12:05.0096393Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization5 2024-04-03T18:12:05.0098829Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization3 2024-04-03T18:12:05.0101292Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization4 2024-04-03T18:12:05.0103885Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization1 2024-04-03T18:12:05.0106361Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization2 2024-04-03T18:12:05.0108816Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization4 2024-04-03T18:12:05.0111287Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_with_unpickleable_attributes 2024-04-03T18:12:05.0113755Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_tensor_view_as_return_value 2024-04-03T18:12:05.0116790Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_device_maps_backward_pass 2024-04-03T18:12:05.0118184Z 2024-04-03T18:12:05.0118802Z Running distributed/test_c10d_common 1/1 ... [2024-04-03 18:12:04.988849] 2024-04-03T18:12:05.0121539Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_common.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:12:04.989310] 2024-04-03T18:14:33.3480175Z 2024-04-03T18:14:33.3484263Z distributed/test_c10d_common 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_common_1.1_f48a6340710d7fff_.log 2024-04-03T18:14:33.3501197Z Running 19 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype, test/distributed/test_c10d_common.py::CommTest::test_debug_level, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle 2024-04-03T18:14:33.3516577Z Running 1 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier 2024-04-03T18:14:33.3518422Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype 2024-04-03T18:14:33.3520380Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype 2024-04-03T18:14:33.3522390Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype 2024-04-03T18:14:33.3524554Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype 2024-04-03T18:14:33.3526291Z Running 1 items in this shard: test/distributed/test_c10d_common.py::CommTest::test_debug_level 2024-04-03T18:14:33.3528003Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr 2024-04-03T18:14:33.3529923Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config 2024-04-03T18:14:33.3531773Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives 2024-04-03T18:14:33.3533670Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name 2024-04-03T18:14:33.3535801Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends 2024-04-03T18:14:33.3537935Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available 2024-04-03T18:14:33.3539816Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv 2024-04-03T18:14:33.3541924Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-04-03T18:14:33.3544325Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-04-03T18:14:33.3546374Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop 2024-04-03T18:14:33.3548173Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable 2024-04-03T18:14:33.3549766Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal 2024-04-03T18:14:33.3551329Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle 2024-04-03T18:14:33.3552236Z 2024-04-03T18:14:33.3552905Z Running distributed/pipeline/sync/test_stream 1/1 ... [2024-04-03 18:14:33.348816] 2024-04-03T18:14:33.3556382Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_stream.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:14:33.349279] 2024-04-03T18:14:37.1713806Z 2024-04-03T18:14:37.1716320Z distributed/pipeline/sync/test_stream 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_stream_1.1_d4c9784d54d159b7_.log 2024-04-03T18:14:37.1731143Z Running 19 items in this shard: test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view 2024-04-03T18:14:37.1744484Z 2024-04-03T18:14:37.1745084Z Running distributed/test_c10d_spawn_nccl 1/1 ... [2024-04-03 18:14:37.171688] 2024-04-03T18:14:37.1747918Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_spawn_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:14:37.172148] 2024-04-03T18:16:00.0859722Z 2024-04-03T18:16:00.0861664Z distributed/test_c10d_spawn_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_nccl_1.1_1968c63c62c31396_.log 2024-04-03T18:16:00.0869619Z Running 9 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather_base, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all_single, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_allreduce, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_broadcast, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter_non_contiguous 2024-04-03T18:16:00.0879583Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather 2024-04-03T18:16:00.0881517Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather_base 2024-04-03T18:16:00.0883426Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all 2024-04-03T18:16:00.0885372Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all_single 2024-04-03T18:16:00.0887282Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_allreduce 2024-04-03T18:16:00.0889131Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_broadcast 2024-04-03T18:16:00.0890952Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce 2024-04-03T18:16:00.0892827Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter 2024-04-03T18:16:00.0894844Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter_non_contiguous 2024-04-03T18:16:00.0896050Z 2024-04-03T18:16:00.0896571Z Running distributed/test_nccl 1/1 ... [2024-04-03 18:16:00.086490] 2024-04-03T18:16:00.0899131Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:16:00.086952] 2024-04-03T18:16:04.6612605Z 2024-04-03T18:16:04.6615072Z distributed/test_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_nccl_1.1_06547e0ea6d09525_.log 2024-04-03T18:16:04.6623035Z Running 12 items in this shard: test/distributed/test_nccl.py::TestNCCLCUDA::test_all_gather_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_gather_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_reduce_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_reduce_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_broadcast_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_broadcast_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_collective_errors_cuda, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_scatter_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_scatter_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_unique_id_cuda 2024-04-03T18:16:04.6629712Z 2024-04-03T18:16:04.6630268Z Running distributed/test_c10d_gloo 1/1 ... [2024-04-03 18:16:04.661546] 2024-04-03T18:16:04.6633021Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:16:04.662004] 2024-04-03T18:33:03.2639809Z 2024-04-03T18:33:03.2643687Z distributed/test_c10d_gloo 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_gloo_1.1_7daf57462ec3fddb_.log 2024-04-03T18:33:03.2762228Z Running 133 items in this shard: test/distributed/test_c10d_gloo.py::RendezvousEnvTest::test_logging_init, test/distributed/test_c10d_gloo.py::TimeoutTest::test_default_store_timeout_gloo, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_into_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_barrier_implies_wait, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_empty_tensors, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_multi_device_constructor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_round_robin, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_round_robin_create_destroy, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_send_recv_all_to_all, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_set_gloo_pg_timeout, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_cuda_dispatched, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_gloo, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_2gpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_4gpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_sharded_tensor, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_save_load_checkpoint, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_optimizer, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_unused_parameters, test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_multi_bucket, test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_single_bucket, test/distributed/test_c10d_gloo.py::ReducerTest::test_single_dtype_single_bucket, test/distributed/test_c10d_gloo.py::CommTest::test_bool_tensors, test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cpu, test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cuda, test/distributed/test_c10d_gloo.py::CommTest::test_gloo_rank_membership, test/distributed/test_c10d_gloo.py::CommTest::test_gloo_warn_not_in_group, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_default, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_subgroup, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_default_pg_gloo, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_gloo_new_group, test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allgather_coalesced, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_monitored_barrier, test/distributed/test_c10d_gloo.py::CompilerTest::test_allgather_work_wait_cpu, test/distributed/test_c10d_gloo.py::CompilerTest::test_allgather_work_wait_gpu, test/distributed/test_c10d_gloo.py::CompilerTest::test_allreduce_work_wait_cpu, test/distributed/test_c10d_gloo.py::CompilerTest::test_allreduce_work_wait_gpu, test/distributed/test_c10d_gloo.py::CompilerTest::test_broadcast_work_wait_cpu, test/distributed/test_c10d_gloo.py::CompilerTest::test_broadcast_work_wait_gpu, test/distributed/test_c10d_gloo.py::CompilerTest::test_consecutive_comm_work_wait_cpu, test/distributed/test_c10d_gloo.py::CompilerTest::test_consecutive_comm_work_wait_gpu, test/distributed/test_c10d_gloo.py::CompilerTest::test_nested_comm_tensor_wrapping, test/distributed/test_c10d_gloo.py::CompilerTest::test_scatter_work_wait_cpu, test/distributed/test_c10d_gloo.py::CompilerTest::test_scatter_work_wait_gpu, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_duplicate_pg, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_sanity_check 2024-04-03T18:33:03.2946151Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::RendezvousEnvTest::test_logging_init 2024-04-03T18:33:03.2949537Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::TimeoutTest::test_default_store_timeout_gloo 2024-04-03T18:33:03.2953071Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics 2024-04-03T18:33:03.2957214Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics_cuda 2024-04-03T18:33:03.2960897Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_checks 2024-04-03T18:33:03.2964589Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_async 2024-04-03T18:33:03.2968475Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_checks 2024-04-03T18:33:03.2972392Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_into_tensor_coalesced 2024-04-03T18:33:03.2976414Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_noncontiguous_input 2024-04-03T18:33:03.2980143Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress 2024-04-03T18:33:03.2983739Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress_cuda 2024-04-03T18:33:03.2987312Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics 2024-04-03T18:33:03.2990909Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics_cuda 2024-04-03T18:33:03.2994986Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_checks 2024-04-03T18:33:03.2998733Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_async 2024-04-03T18:33:03.3002857Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_basics 2024-04-03T18:33:03.3006730Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks 2024-04-03T18:33:03.3010634Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks_cuda 2024-04-03T18:33:03.3014570Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_stress 2024-04-03T18:33:03.3018332Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress 2024-04-03T18:33:03.3021943Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress_cuda 2024-04-03T18:33:03.3025629Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_barrier_implies_wait 2024-04-03T18:33:03.3029312Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics 2024-04-03T18:33:03.3032898Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics_cuda 2024-04-03T18:33:03.3036829Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_checks 2024-04-03T18:33:03.3040361Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress 2024-04-03T18:33:03.3043957Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress_cuda 2024-04-03T18:33:03.3047514Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_empty_tensors 2024-04-03T18:33:03.3051247Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics 2024-04-03T18:33:03.3054841Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics_cuda 2024-04-03T18:33:03.3058393Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_checks 2024-04-03T18:33:03.3062009Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_noncontiguous_input 2024-04-03T18:33:03.3065616Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress 2024-04-03T18:33:03.3069104Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress_cuda 2024-04-03T18:33:03.3072808Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_multi_device_constructor 2024-04-03T18:33:03.3076739Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics 2024-04-03T18:33:03.3080270Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics_cuda 2024-04-03T18:33:03.3083868Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_checks 2024-04-03T18:33:03.3087466Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor 2024-04-03T18:33:03.3091286Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor_coalesced 2024-04-03T18:33:03.3095000Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress 2024-04-03T18:33:03.3098471Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress_cuda 2024-04-03T18:33:03.3101975Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_round_robin 2024-04-03T18:33:03.3105621Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_round_robin_create_destroy 2024-04-03T18:33:03.3109512Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics 2024-04-03T18:33:03.3113071Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics_cuda 2024-04-03T18:33:03.3117138Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_checks 2024-04-03T18:33:03.3120576Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress 2024-04-03T18:33:03.3124063Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress_cuda 2024-04-03T18:33:03.3127656Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_send_recv_all_to_all 2024-04-03T18:33:03.3131250Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_set_gloo_pg_timeout 2024-04-03T18:33:03.3134920Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics 2024-04-03T18:33:03.3138760Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics_cuda 2024-04-03T18:33:03.3142551Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_checks 2024-04-03T18:33:03.3146450Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_cuda_dispatched 2024-04-03T18:33:03.3150423Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output 2024-04-03T18:33:03.3154775Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output_unused_param 2024-04-03T18:33:03.3159338Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2024-04-03T18:33:03.3163735Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2024-04-03T18:33:03.3168298Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2024-04-03T18:33:03.3172852Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2024-04-03T18:33:03.3177544Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False 2024-04-03T18:33:03.3182491Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2024-04-03T18:33:03.3187383Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2024-04-03T18:33:03.3192020Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2024-04-03T18:33:03.3196801Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing 2024-04-03T18:33:03.3201472Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False 2024-04-03T18:33:03.3206326Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2024-04-03T18:33:03.3211244Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False 2024-04-03T18:33:03.3216321Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2024-04-03T18:33:03.3220952Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu 2024-04-03T18:33:03.3225320Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_gloo 2024-04-03T18:33:03.3229677Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once 2024-04-03T18:33:03.3233949Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients 2024-04-03T18:33:03.3238408Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init 2024-04-03T18:33:03.3242758Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type 2024-04-03T18:33:03.3247310Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty 2024-04-03T18:33:03.3251859Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad 2024-04-03T18:33:03.3256375Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view 2024-04-03T18:33:03.3261088Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph 2024-04-03T18:33:03.3265550Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_integer_list 2024-04-03T18:33:03.3270628Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_torch_device_list 2024-04-03T18:33:03.3275477Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_2gpu_module 2024-04-03T18:33:03.3279571Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_4gpu_module 2024-04-03T18:33:03.3283613Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module 2024-04-03T18:33:03.3287889Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module_grad_is_view 2024-04-03T18:33:03.3291979Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output 2024-04-03T18:33:03.3296086Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters 2024-04-03T18:33:03.3300380Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_sharded_tensor 2024-04-03T18:33:03.3304497Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_invalid_powerSGD_state 2024-04-03T18:33:03.3308394Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_save_load_checkpoint 2024-04-03T18:33:03.3312248Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients 2024-04-03T18:33:03.3316611Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view 2024-04-03T18:33:03.3320773Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2024-04-03T18:33:03.3325233Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input 2024-04-03T18:33:03.3328975Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward 2024-04-03T18:33:03.3332292Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_optimizer 2024-04-03T18:33:03.3335894Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_unused_parameters 2024-04-03T18:33:03.3339472Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_multi_bucket 2024-04-03T18:33:03.3342961Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_single_bucket 2024-04-03T18:33:03.3346488Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_single_dtype_single_bucket 2024-04-03T18:33:03.3349796Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_bool_tensors 2024-04-03T18:33:03.3353011Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cpu 2024-04-03T18:33:03.3356816Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cuda 2024-04-03T18:33:03.3360159Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_gloo_rank_membership 2024-04-03T18:33:03.3363404Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_gloo_warn_not_in_group 2024-04-03T18:33:03.3366865Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_default 2024-04-03T18:33:03.3370549Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_subgroup 2024-04-03T18:33:03.3374478Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_default_pg_gloo 2024-04-03T18:33:03.3378049Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_gloo_new_group 2024-04-03T18:33:03.3381424Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_complex 2024-04-03T18:33:03.3384671Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_mismatch 2024-04-03T18:33:03.3388786Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single 2024-04-03T18:33:03.3393334Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allgather_coalesced 2024-04-03T18:33:03.3398192Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced 2024-04-03T18:33:03.3402758Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_collectives 2024-04-03T18:33:03.3407498Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-04-03T18:33:03.3412627Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-04-03T18:33:03.3417534Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_monitored_barrier 2024-04-03T18:33:03.3421531Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_allgather_work_wait_cpu 2024-04-03T18:33:03.3425002Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_allgather_work_wait_gpu 2024-04-03T18:33:03.3428384Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_allreduce_work_wait_cpu 2024-04-03T18:33:03.3432087Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_allreduce_work_wait_gpu 2024-04-03T18:33:03.3435881Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_broadcast_work_wait_cpu 2024-04-03T18:33:03.3439337Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_broadcast_work_wait_gpu 2024-04-03T18:33:03.3442892Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_consecutive_comm_work_wait_cpu 2024-04-03T18:33:03.3446593Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_consecutive_comm_work_wait_gpu 2024-04-03T18:33:03.3450196Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_nested_comm_tensor_wrapping 2024-04-03T18:33:03.3453664Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_scatter_work_wait_cpu 2024-04-03T18:33:03.3457095Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CompilerTest::test_scatter_work_wait_gpu 2024-04-03T18:33:03.3460502Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync 2024-04-03T18:33:03.3464084Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_duplicate_pg 2024-04-03T18:33:03.3467860Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_sanity_check 2024-04-03T18:33:03.3470032Z 2024-04-03T18:33:03.3471343Z Running distributed/elastic/events/lib_test 1/1 ... [2024-04-03 18:33:03.270101] 2024-04-03T18:33:03.3477412Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/events/lib_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:33:03.270632] 2024-04-03T18:33:07.1933253Z 2024-04-03T18:33:07.1935746Z distributed/elastic/events/lib_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.events.lib_test_1.1_85d56d643bc3e184_.log 2024-04-03T18:33:07.1943045Z Running 8 items in this shard: test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_created, test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser, test/distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str 2024-04-03T18:33:07.1948940Z 2024-04-03T18:33:07.1949607Z Running distributed/elastic/metrics/api_test 1/1 ... [2024-04-03 18:33:07.193619] 2024-04-03T18:33:07.1952431Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:33:07.194071] 2024-04-03T18:33:11.0672703Z 2024-04-03T18:33:11.0675293Z distributed/elastic/metrics/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.metrics.api_test_1.1_4163e9da5e8b3af5_.log 2024-04-03T18:33:11.0678958Z Running 3 items in this shard: test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_get_metric_name, test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_inheritance, test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_profile 2024-04-03T18:33:11.0681340Z 2024-04-03T18:33:11.0682379Z Running distributed/elastic/multiprocessing/api_test 1/1 ... [2024-04-03 18:33:11.067640] 2024-04-03T18:33:11.0686134Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:33:11.068140] 2024-04-03T18:33:41.7505677Z 2024-04-03T18:33:41.7507881Z distributed/elastic/multiprocessing/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.multiprocessing.api_test_1.1_1376dbe0a59bde8b_.log 2024-04-03T18:33:41.7540533Z Running 38 items in this shard: test/distributed/elastic/multiprocessing/api_test.py::RunProcResultsTest::test_get_failures, test/distributed/elastic/multiprocessing/api_test.py::RunProcResultsTest::test_is_failed, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_str_bad_input, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_value, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_value_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_args_env_len_mismatch, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_binary_exit, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_binary_incorrect_entrypoint, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_binary_raises, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_function_large_ret_val, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_function_raise, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_function_with_tensor, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_invalid_log_dir, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_multiprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_multiprocessing_context_poll_raises_exception, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_pcontext_wait, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_subprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_to_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_validate_full_rank, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_void_function, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_args_env_len_mismatch, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_binary, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_binary_exit, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_binary_incorrect_entrypoint, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_binary_raises, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_binary_redirect_and_tee, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_function, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_function_large_ret_val, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_function_raise, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_function_with_tensor, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_invalid_log_dir, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_multiprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_multiprocessing_context_poll_raises_exception, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_pcontext_wait, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_subprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_to_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_validate_full_rank, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_void_function 2024-04-03T18:33:41.7571504Z 2024-04-03T18:33:41.7572245Z Running distributed/elastic/timer/local_timer_example 1/1 ... [2024-04-03 18:33:41.751125] 2024-04-03T18:33:41.7575213Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:33:41.751648] 2024-04-03T18:33:55.3471669Z 2024-04-03T18:33:55.3473941Z distributed/elastic/timer/local_timer_example 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.local_timer_example_1.1_bc5a8ed72e1330c9_.log 2024-04-03T18:33:55.3478144Z Running 2 items in this shard: test/distributed/elastic/timer/local_timer_example.py::LocalTimerExample::test_example_start_method_spawn, test/distributed/elastic/timer/local_timer_example.py::LocalTimerExample::test_torch_mp_example 2024-04-03T18:33:55.3480449Z 2024-04-03T18:33:55.3481172Z Running distributed/elastic/timer/local_timer_test 1/1 ... [2024-04-03 18:33:55.347473] 2024-04-03T18:33:55.3484425Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/timer/local_timer_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:33:55.347925] 2024-04-03T18:34:03.7293563Z 2024-04-03T18:34:03.7295790Z distributed/elastic/timer/local_timer_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.local_timer_test_1.1_8fc4b594f8570873_.log 2024-04-03T18:34:03.7307728Z Running 14 items in this shard: test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_client_interaction, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_exception_propagation, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_get_timer_recursive, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_happy_path, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_no_client, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_timer, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get_less_than_size, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get_size, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_acquire_release, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_expired_timers, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_valid_timers, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_watchdog_call_count, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_watchdog_empty_queue 2024-04-03T18:34:03.7318985Z 2024-04-03T18:34:03.7319691Z Running distributed/elastic/utils/distributed_test 1/1 ... [2024-04-03 18:34:03.729608] 2024-04-03T18:34:03.7322644Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/utils/distributed_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:34:03.730059] 2024-04-03T18:34:09.7577140Z 2024-04-03T18:34:09.7579612Z distributed/elastic/utils/distributed_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.distributed_test_1.1_09fef3d43894e4ff_.log 2024-04-03T18:34:09.7587801Z Running 7 items in this shard: test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_multi, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_no_port_multi, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_single_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_timeout_on_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_timeout_on_worker, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_port_already_in_use_on_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_port_already_in_use_on_worker 2024-04-03T18:34:09.7593880Z 2024-04-03T18:34:09.7595201Z Running distributed/elastic/utils/logging_test 1/1 ... [2024-04-03 18:34:09.757956] 2024-04-03T18:34:09.7598112Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:34:09.758415] 2024-04-03T18:34:13.6308516Z 2024-04-03T18:34:13.6310897Z distributed/elastic/utils/logging_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.logging_test_1.1_7a1973c6f43f84c6_.log 2024-04-03T18:34:13.6314782Z Running 2 items in this shard: test/distributed/elastic/utils/logging_test.py::LoggingTest::test_derive_module_name, test/distributed/elastic/utils/logging_test.py::LoggingTest::test_logger_name 2024-04-03T18:34:13.6317278Z 2024-04-03T18:34:13.6317943Z Running distributed/elastic/utils/util_test 1/1 ... [2024-04-03 18:34:13.631177] 2024-04-03T18:34:13.6321402Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/utils/util_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:34:13.631676] 2024-04-03T18:34:17.6043792Z 2024-04-03T18:34:17.6046088Z distributed/elastic/utils/util_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.util_test_1.1_f29dec163b3eb601_.log 2024-04-03T18:34:17.6052058Z Running 7 items in this shard: test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_get_all_rank_0, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_get_all_rank_n, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_synchronize, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_custom_name, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_different, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_none 2024-04-03T18:34:17.6056593Z 2024-04-03T18:34:18.2431543Z 2024-04-03T18:34:18.2432262Z real 130m11.837s 2024-04-03T18:34:18.2432794Z user 224m45.419s 2024-04-03T18:34:18.2433230Z sys 102m51.169s 2024-04-03T18:34:18.2433640Z + assert_git_not_dirty 2024-04-03T18:34:18.2435098Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-04-03T18:34:18.2435975Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-04-03T18:34:18.2438666Z ++ git status --porcelain 2024-04-03T18:34:18.2439750Z ++ grep -v '?? third_party' 2024-04-03T18:34:19.7859398Z ++ true 2024-04-03T18:34:19.7862954Z + git_status= 2024-04-03T18:34:19.7863566Z + [[ -n '' ]] 2024-04-03T18:34:19.7864234Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-04-03T18:34:19.7864893Z + [[ 2 == 1 ]] 2024-04-03T18:34:19.7865296Z + [[ 2 == 1 ]] 2024-04-03T18:34:19.7951261Z ##[group]Run cat test/**/*_toprint.log || true 2024-04-03T18:34:19.7952018Z cat test/**/*_toprint.log || true 2024-04-03T18:34:19.7962523Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:34:19.7963245Z env: 2024-04-03T18:34:19.7963632Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:34:19.7964254Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:34:19.7965308Z DOCKER_CONTAINER_ID: 7033b537529c546187ef7c776250a79cad9cc38d4987347f20cdbf86b60e2c90 2024-04-03T18:34:19.7966225Z ##[endgroup] 2024-04-03T18:34:19.8049311Z cat: test/**/*_toprint.log: No such file or directory 2024-04-03T18:34:19.8094971Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-04-03T18:34:19.8095685Z kill "$MONITOR_SCRIPT_PID" 2024-04-03T18:34:19.8104773Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:34:19.8105511Z env: 2024-04-03T18:34:19.8105907Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:34:19.8106537Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:34:19.8107598Z DOCKER_CONTAINER_ID: 7033b537529c546187ef7c776250a79cad9cc38d4987347f20cdbf86b60e2c90 2024-04-03T18:34:19.8108547Z MONITOR_SCRIPT_PID: 90207 2024-04-03T18:34:19.8109031Z ##[endgroup] 2024-04-03T18:34:19.8357425Z Prepare all required actions 2024-04-03T18:34:19.8358111Z Getting action download info 2024-04-03T18:34:19.9651179Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-04-03T18:34:20.1656759Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-04-03T18:34:20.1657424Z with: 2024-04-03T18:34:20.1658135Z file-suffix: test-distributed-2-3-linux.8xlarge.nvidia.gpu_23402706124 2024-04-03T18:34:20.1659040Z s3-bucket: gha-artifacts 2024-04-03T18:34:20.1659510Z env: 2024-04-03T18:34:20.1659901Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:34:20.1660779Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:34:20.1661822Z DOCKER_CONTAINER_ID: 7033b537529c546187ef7c776250a79cad9cc38d4987347f20cdbf86b60e2c90 2024-04-03T18:34:20.1662751Z ##[endgroup] 2024-04-03T18:34:20.1701140Z ##[group]Run # Remove any previous test jsons if they exist 2024-04-03T18:34:20.1702101Z # Remove any previous test jsons if they exist 2024-04-03T18:34:20.1702818Z rm -f test-jsons-*.zip 2024-04-03T18:34:20.1703565Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-04-03T18:34:20.1713154Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:34:20.1713876Z env: 2024-04-03T18:34:20.1714815Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:34:20.1715539Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:34:20.1716597Z DOCKER_CONTAINER_ID: 7033b537529c546187ef7c776250a79cad9cc38d4987347f20cdbf86b60e2c90 2024-04-03T18:34:20.1717820Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_23402706124 2024-04-03T18:34:20.1718676Z ##[endgroup] 2024-04-03T18:34:20.2049285Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-04-03T18:34:20.2092782Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-04-03T18:34:20.2093772Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-04-03T18:34:20.2102159Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-04-03T18:34:20.2103276Z adding: test/.pytorch-slow-tests.json (deflated 59%) 2024-04-03T18:34:20.2113402Z adding: test/.pytorch-disabled-tests.json (deflated 84%) 2024-04-03T18:34:20.2169481Z ##[group]Run # Remove any previous test reports if they exist 2024-04-03T18:34:20.2170417Z # Remove any previous test reports if they exist 2024-04-03T18:34:20.2171172Z rm -f test-reports-*.zip 2024-04-03T18:34:20.2172019Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-04-03T18:34:20.2181455Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:34:20.2182166Z env: 2024-04-03T18:34:20.2182549Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:34:20.2183193Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:34:20.2184228Z DOCKER_CONTAINER_ID: 7033b537529c546187ef7c776250a79cad9cc38d4987347f20cdbf86b60e2c90 2024-04-03T18:34:20.2185465Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_23402706124 2024-04-03T18:34:20.2186320Z ##[endgroup] 2024-04-03T18:34:20.2507953Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_nested_dict/distributed.checkpoint.test_nested_dict-6436a21ca54ae6a9.xml (deflated 51%) 2024-04-03T18:34:20.2510491Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor/distributed._tensor.test_dtensor-04a197eb2142134e.xml (deflated 93%) 2024-04-03T18:34:20.2513081Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_optim_state/distributed.checkpoint.test_fsdp_optim_state-7143f24e73d67445.xml (deflated 53%) 2024-04-03T18:34:20.2516294Z adding: test/test-reports/python-pytest/distributed._tensor.test_api/distributed._tensor.test_api-36fadc29f736bd8b.xml (deflated 85%) 2024-04-03T18:34:20.2518635Z adding: test/test-reports/python-pytest/distributed._tensor.test_random_ops/distributed._tensor.test_random_ops-9c90627bdfe2c120.xml (deflated 73%) 2024-04-03T18:34:20.2521517Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dtensor_resharding/distributed.checkpoint.test_dtensor_resharding-c9b939653d443a33.xml (deflated 71%) 2024-04-03T18:34:20.2524465Z adding: test/test-reports/python-pytest/distributed._tensor.experimental.test_tp_transform/distributed._tensor.experimental.test_tp_transform-5b66f97b6d51b830.xml (deflated 75%) 2024-04-03T18:34:20.2527475Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_autograd/distributed._composable.fsdp.test_fully_shard_autograd-233ffb9db3b3a805.xml (deflated 58%) 2024-04-03T18:34:20.2530305Z adding: test/test-reports/python-pytest/distributed.test_compute_comm_reordering/distributed.test_compute_comm_reordering-2b6ec7ffd4d815db.xml (deflated 81%) 2024-04-03T18:34:20.2533270Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_ddp_2d_parallel/distributed.tensor.parallel.test_ddp_2d_parallel-5a88947b3ea819d3.xml (deflated 51%) 2024-04-03T18:34:20.2535999Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsspec/distributed.checkpoint.test_fsspec-4bfea7a70b54560d.xml (deflated 56%) 2024-04-03T18:34:20.2538571Z adding: test/test-reports/python-pytest/distributed._tensor.test_experimental_ops/distributed._tensor.test_experimental_ops-9baf12e7911471db.xml (deflated 57%) 2024-04-03T18:34:20.2541086Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_memory/distributed.fsdp.test_fsdp_memory-b90a85b9a2238152.xml (deflated 51%) 2024-04-03T18:34:20.2543584Z adding: test/test-reports/python-pytest/distributed._tensor.test_convolution_ops/distributed._tensor.test_convolution_ops-0dbfa518ceb17f26.xml (deflated 51%) 2024-04-03T18:34:20.2546142Z adding: test/test-reports/python-pytest/distributed._tensor.test_redistribute/distributed._tensor.test_redistribute-a459f54ce8d50c5d.xml (deflated 86%) 2024-04-03T18:34:20.2548898Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_random_state/distributed.tensor.parallel.test_tp_random_state-6afe6bfb7411bf09.xml (deflated 51%) 2024-04-03T18:34:20.2551536Z adding: test/test-reports/python-pytest/distributed._tensor.test_math_ops/distributed._tensor.test_math_ops-c2c589428bf01f0f.xml (deflated 72%) 2024-04-03T18:34:20.2554644Z adding: test/test-reports/python-pytest/distributed.checkpoint.e2e.test_e2e_save_and_load/distributed.checkpoint.e2e.test_e2e_save_and_load-5d0c194490b11dce.xml (deflated 86%) 2024-04-03T18:34:20.2557761Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_state_dict/distributed._composable.fsdp.test_fully_shard_state_dict-188db4aee1a926de.xml (deflated 52%) 2024-04-03T18:34:20.2560949Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_mixed_precision/distributed._composable.fsdp.test_fully_shard_mixed_precision-776a2c59f95a11e4.xml (deflated 71%) 2024-04-03T18:34:20.2563963Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_state_dict_utils/distributed.checkpoint.test_state_dict_utils-7055dcde5a1393b8.xml (deflated 74%) 2024-04-03T18:34:20.2566840Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_fsdp_2d_parallel/distributed.tensor.parallel.test_fsdp_2d_parallel-4f7ebc70d8a77df4.xml (deflated 86%) 2024-04-03T18:34:20.2569362Z adding: test/test-reports/python-pytest/distributed.test_fake_pg/distributed.test_fake_pg-be97d90230db9c89.xml (deflated 79%) 2024-04-03T18:34:20.2571956Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_parallelize_api/distributed.tensor.parallel.test_parallelize_api-81bf51e82d91da28.xml (deflated 83%) 2024-04-03T18:34:20.2574999Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_util/distributed._composable.fully_shard.test_fully_shard_util-25249ade8d1d9d60.xml (deflated 39%) 2024-04-03T18:34:20.2577615Z adding: test/test-reports/python-pytest/distributed.test_c10d_pypg/distributed.test_c10d_pypg-8f5184f1498f708c.xml (deflated 90%) 2024-04-03T18:34:20.2580240Z adding: test/test-reports/python-pytest/distributed.fsdp.test_distributed_checkpoint/distributed.fsdp.test_distributed_checkpoint-87b98384f0a02984.xml (deflated 76%) 2024-04-03T18:34:20.2582787Z adding: test/test-reports/python-pytest/distributed._tensor.test_utils/distributed._tensor.test_utils-7fcded5f0e17f62f.xml (deflated 69%) 2024-04-03T18:34:20.2585409Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_comm/distributed._composable.fsdp.test_fully_shard_comm-cc4566e149fecb02.xml (deflated 62%) 2024-04-03T18:34:20.2588616Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_model_checkpoint/distributed._composable.fully_shard.test_fully_shard_model_checkpoint-0609cfda792de188.xml (deflated 67%) 2024-04-03T18:34:20.2591677Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_utils/distributed.checkpoint.test_utils-8c0d443116d9f335.xml (deflated 68%) 2024-04-03T18:34:20.2594091Z adding: test/test-reports/python-pytest/distributed._tensor.test_view_ops/distributed._tensor.test_view_ops-0099c8621d48f2dd.xml (deflated 66%) 2024-04-03T18:34:20.2597433Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_init/distributed._composable.fully_shard.test_fully_shard_init-ef0b4ab4bb51f93e.xml (deflated 71%) 2024-04-03T18:34:20.2600134Z adding: test/test-reports/python-pytest/distributed._shard.test_sharder/distributed._shard.test_sharder-d439ffeb93f18337.xml (deflated 65%) 2024-04-03T18:34:20.2602511Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_overlap/distributed.fsdp.test_fsdp_overlap-c11db62ae3943e43.xml (deflated 51%) 2024-04-03T18:34:20.2604876Z adding: test/test-reports/python-pytest/distributed.test_c10d_logger/distributed.test_c10d_logger-da5a5c64c4bc151f.xml (deflated 56%) 2024-04-03T18:34:20.2607495Z adding: test/test-reports/python-pytest/distributed._shard.sharding_spec.test_sharding_spec/distributed._shard.sharding_spec.test_sharding_spec-bab257ddb6380331.xml (deflated 80%) 2024-04-03T18:34:20.2610340Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_dtensor_state_dict/distributed.fsdp.test_fsdp_dtensor_state_dict-5a38289e2bd375b6.xml (deflated 86%) 2024-04-03T18:34:20.2613041Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_tp_integration/distributed.fsdp.test_fsdp_tp_integration-5b516cb24b3c852d.xml (deflated 83%) 2024-04-03T18:34:20.2615619Z adding: test/test-reports/python-pytest/distributed.test_multi_threaded_pg/distributed.test_multi_threaded_pg-f1e9204379bc6fd3.xml (deflated 79%) 2024-04-03T18:34:20.2618124Z adding: test/test-reports/python-pytest/distributed.nn.jit.test_instantiator/distributed.nn.jit.test_instantiator-4cbed902d6d1f5ba.xml (deflated 61%) 2024-04-03T18:34:20.2620906Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_file_system_checkpoint/distributed.checkpoint.test_file_system_checkpoint-834ebce1ca18ee9e.xml (deflated 68%) 2024-04-03T18:34:20.2623656Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_planner/distributed.checkpoint.test_planner-562c7749f10813d5.xml (deflated 63%) 2024-04-03T18:34:20.2626099Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_apply/distributed.fsdp.test_fsdp_apply-378ea8165f1d7ce1.xml (deflated 56%) 2024-04-03T18:34:20.2628638Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_pipeline/distributed.pipeline.sync.test_pipeline-a872260412d3a0e4.xml (deflated 38%) 2024-04-03T18:34:20.2631471Z adding: test/test-reports/python-pytest/distributed._composable.test_replicate_with_compiler/distributed._composable.test_replicate_with_compiler-c01009541d77160e.xml (deflated 83%) 2024-04-03T18:34:20.2634935Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_state_dict/distributed.fsdp.test_fsdp_state_dict-a278247d4e7148a7.xml (deflated 94%) 2024-04-03T18:34:20.2637652Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-1f03ca983e0e15ae.xml (deflated 88%) 2024-04-03T18:34:20.2639984Z adding: test/test-reports/python-pytest/distributed.fsdp.test_utils/distributed.fsdp.test_utils-abf7b10837143340.xml (deflated 68%) 2024-04-03T18:34:20.2642409Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_phony/distributed.pipeline.sync.test_phony-218280dc67b71e47.xml (deflated 64%) 2024-04-03T18:34:20.2645009Z adding: test/test-reports/python-pytest/distributed.test_c10d_object_collectives/distributed.test_c10d_object_collectives-71deb56ed46e9a88.xml (deflated 77%) 2024-04-03T18:34:20.2647557Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_bugs/distributed.pipeline.sync.test_bugs-83909a54b4ca2baa.xml (deflated 54%) 2024-04-03T18:34:20.2650368Z adding: test/test-reports/python-pytest/distributed.fsdp.test_hsdp_dtensor_state_dict/distributed.fsdp.test_hsdp_dtensor_state_dict-9e7bc4aa8263ada9.xml (deflated 86%) 2024-04-03T18:34:20.2653152Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_gpipe/distributed.pipeline.sync.skip.test_gpipe-3a71c249832cbf0a.xml (deflated 80%) 2024-04-03T18:34:20.2656243Z adding: test/test-reports/python-pytest/distributed.algorithms.test_join/distributed.algorithms.test_join-7512bb08f6cc2585.xml (deflated 77%) 2024-04-03T18:34:20.2659464Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_embedding/distributed._shard.sharded_tensor.ops.test_embedding-199430e6120622fa.xml (deflated 66%) 2024-04-03T18:34:20.2662902Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_embedding_bag/distributed._shard.sharded_tensor.ops.test_embedding_bag-1cd6d14d26b496d0.xml (deflated 67%) 2024-04-03T18:34:20.2666148Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-36acfedf8eebdcf7.xml (deflated 37%) 2024-04-03T18:34:20.2669334Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-7afc9a8cc08b133a.xml (deflated 37%) 2024-04-03T18:34:20.2672573Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-8a5823c681528c33.xml (deflated 47%) 2024-04-03T18:34:20.2675924Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a38c3b06843ccf85.xml (deflated 47%) 2024-04-03T18:34:20.2678664Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a4875c5434581b1b.xml (deflated 47%) 2024-04-03T18:34:20.2681392Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-18ce327be08bdc00.xml (deflated 47%) 2024-04-03T18:34:20.2684126Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-8574dba42f01e6e9.xml (deflated 38%) 2024-04-03T18:34:20.2686854Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4c10f9a626528b4d.xml (deflated 38%) 2024-04-03T18:34:20.2689565Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e1fabcec425e0df2.xml (deflated 39%) 2024-04-03T18:34:20.2692292Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5e1d7f917fcf6d93.xml (deflated 39%) 2024-04-03T18:34:20.2695028Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a074bbb32b1a6a3c.xml (deflated 39%) 2024-04-03T18:34:20.2697757Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9ee4ce8f6bf3dece.xml (deflated 39%) 2024-04-03T18:34:20.2700632Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4fe4c823acac44cf.xml (deflated 38%) 2024-04-03T18:34:20.2703397Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-98ebcdef6ba33bd0.xml (deflated 38%) 2024-04-03T18:34:20.2706122Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0d6fbe66e3b022ad.xml (deflated 39%) 2024-04-03T18:34:20.2708856Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-8fe489daefbb60b1.xml (deflated 39%) 2024-04-03T18:34:20.2711703Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1404d94a4485c95d.xml (deflated 38%) 2024-04-03T18:34:20.2715008Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b3965b3c8e817330.xml (deflated 38%) 2024-04-03T18:34:20.2717809Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-f18f272411d4b782.xml (deflated 38%) 2024-04-03T18:34:20.2720537Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5cfcc1e1bfec6282.xml (deflated 39%) 2024-04-03T18:34:20.2723244Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-bda580088a74aa68.xml (deflated 38%) 2024-04-03T18:34:20.2726217Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ff5eefc1d5346f45.xml (deflated 38%) 2024-04-03T18:34:20.2729432Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-35a04e2751f8f7ab.xml (deflated 38%) 2024-04-03T18:34:20.2732640Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-14bd26f8ae1315bc.xml (deflated 38%) 2024-04-03T18:34:20.2735861Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d39c391efc11c9e4.xml (deflated 38%) 2024-04-03T18:34:20.2738823Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b7d35021b4c86f5a.xml (deflated 38%) 2024-04-03T18:34:20.2741779Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-859e92a393f3abac.xml (deflated 38%) 2024-04-03T18:34:20.2744513Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a4779b8f58c77c4d.xml (deflated 39%) 2024-04-03T18:34:20.2747234Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1f6fae526461b55e.xml (deflated 38%) 2024-04-03T18:34:20.2749929Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-3044812b9a45c969.xml (deflated 39%) 2024-04-03T18:34:20.2752654Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-cce57f6243684916.xml (deflated 39%) 2024-04-03T18:34:20.2755947Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-52a24e4598f55686.xml (deflated 38%) 2024-04-03T18:34:20.2758917Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-def5f62074b4d2a5.xml (deflated 38%) 2024-04-03T18:34:20.2762045Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-f896e76a5fcd8f1b.xml (deflated 38%) 2024-04-03T18:34:20.2764785Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e780ec210e8fdac2.xml (deflated 38%) 2024-04-03T18:34:20.2767503Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-912cf48d909e7041.xml (deflated 38%) 2024-04-03T18:34:20.2770675Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5a1cb7bd93d2dbdc.xml (deflated 38%) 2024-04-03T18:34:20.2773594Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9159606cb189e57f.xml (deflated 38%) 2024-04-03T18:34:20.2776297Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ecc45771ff15ad99.xml (deflated 38%) 2024-04-03T18:34:20.2779464Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-18343236ac578f15.xml (deflated 38%) 2024-04-03T18:34:20.2782585Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5095f8023b991f36.xml (deflated 38%) 2024-04-03T18:34:20.2785320Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-683cd79357678c32.xml (deflated 37%) 2024-04-03T18:34:20.2788042Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c49d9597b5dcdec3.xml (deflated 37%) 2024-04-03T18:34:20.2790769Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2bece421b50b7dfe.xml (deflated 39%) 2024-04-03T18:34:20.2793502Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-27cb9a525f4ea1b2.xml (deflated 38%) 2024-04-03T18:34:20.2797001Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a3a352e3355beead.xml (deflated 39%) 2024-04-03T18:34:20.2799720Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4ac649eef886120f.xml (deflated 38%) 2024-04-03T18:34:20.2802452Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-6760065b981f98bc.xml (deflated 38%) 2024-04-03T18:34:20.2805168Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-45f5aec43b574ac4.xml (deflated 38%) 2024-04-03T18:34:20.2807961Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c49ae0d66900a1e0.xml (deflated 38%) 2024-04-03T18:34:20.2810673Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-dcc8c32b911f127b.xml (deflated 38%) 2024-04-03T18:34:20.2813399Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-dcd00dbdcffdc6a1.xml (deflated 38%) 2024-04-03T18:34:20.2816167Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0ed9f70872333bee.xml (deflated 38%) 2024-04-03T18:34:20.2818884Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b99433e190794663.xml (deflated 39%) 2024-04-03T18:34:20.2821748Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c8b44946f0423f19.xml (deflated 39%) 2024-04-03T18:34:20.2824483Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ddc312b421e4aa29.xml (deflated 48%) 2024-04-03T18:34:20.2826953Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-a03b112133da84fa.xml (deflated 38%) 2024-04-03T18:34:20.2829190Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-031877a9134db0e9.xml (deflated 37%) 2024-04-03T18:34:20.2831549Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-ea1681adadd9e4e6.xml (deflated 37%) 2024-04-03T18:34:20.2833761Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-2c7470e0c77a8d40.xml (deflated 37%) 2024-04-03T18:34:20.2836536Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-f0e04bdf1b167569.xml (deflated 38%) 2024-04-03T18:34:20.2838766Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-b80c7015140c56e7.xml (deflated 38%) 2024-04-03T18:34:20.2840998Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-992cdf78fb86d9f2.xml (deflated 37%) 2024-04-03T18:34:20.2843211Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-56859a90cbef1383.xml (deflated 37%) 2024-04-03T18:34:20.2845421Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-587856370b3e8b4f.xml (deflated 37%) 2024-04-03T18:34:20.2847642Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-a8207f8c29cc2c8c.xml (deflated 37%) 2024-04-03T18:34:20.2849848Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-e0932d6c90a6eae8.xml (deflated 37%) 2024-04-03T18:34:20.2852067Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-963297a8bbcf0fd8.xml (deflated 37%) 2024-04-03T18:34:20.2854301Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-7d09b358baeab5c1.xml (deflated 37%) 2024-04-03T18:34:20.2856519Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-8971f9832ef0952b.xml (deflated 37%) 2024-04-03T18:34:20.2858721Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-d88e156452cc7e49.xml (deflated 37%) 2024-04-03T18:34:20.2860960Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-06ffd3887e773433.xml (deflated 38%) 2024-04-03T18:34:20.2863185Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-630b2670584a6f1a.xml (deflated 38%) 2024-04-03T18:34:20.2865415Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-f20b071ec39cbaa6.xml (deflated 38%) 2024-04-03T18:34:20.2867618Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-606c90b2d82770b7.xml (deflated 38%) 2024-04-03T18:34:20.2870035Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_stream/distributed.pipeline.sync.test_stream-299744f70d6426f3.xml (deflated 86%) 2024-04-03T18:34:20.2872520Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-29bb274c9b85a6ba.xml (deflated 35%) 2024-04-03T18:34:20.2875440Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-fccf94a27e2910b6.xml (deflated 35%) 2024-04-03T18:34:20.2877951Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-d22f916ecba0127c.xml (deflated 36%) 2024-04-03T18:34:20.2880322Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-4e76dedf26e94130.xml (deflated 36%) 2024-04-03T18:34:20.2882683Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-8be2813b0b5fdcb5.xml (deflated 35%) 2024-04-03T18:34:20.2885020Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-cf28957110a3518b.xml (deflated 35%) 2024-04-03T18:34:20.2887347Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-83d2e09aeb6a8057.xml (deflated 35%) 2024-04-03T18:34:20.2889833Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-268bf047b3e0d561.xml (deflated 35%) 2024-04-03T18:34:20.2892177Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-8dc2494b2bf08090.xml (deflated 35%) 2024-04-03T18:34:20.2894364Z adding: test/test-reports/python-pytest/distributed.test_nccl/distributed.test_nccl-6bc0b6d387cad052.xml (deflated 81%) 2024-04-03T18:34:20.2896615Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-80ffc0319921986f.xml (deflated 38%) 2024-04-03T18:34:20.2898798Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2c7b9df6bf46e2d6.xml (deflated 39%) 2024-04-03T18:34:20.2900983Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d1d9250b10af7f5a.xml (deflated 37%) 2024-04-03T18:34:20.2903158Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e549d00d8504ee80.xml (deflated 37%) 2024-04-03T18:34:20.2905306Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a90fd2c8033020f1.xml (deflated 37%) 2024-04-03T18:34:20.2907474Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9638c45c0cc5e7ad.xml (deflated 37%) 2024-04-03T18:34:20.2909653Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-aa3a5b223363e81c.xml (deflated 37%) 2024-04-03T18:34:20.2911814Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-088f606dcdb1eb86.xml (deflated 37%) 2024-04-03T18:34:20.2913964Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7209ce201ad9adba.xml (deflated 37%) 2024-04-03T18:34:20.2916985Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9579dd7880ad8be3.xml (deflated 37%) 2024-04-03T18:34:20.2919199Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b08c709d66f3f7d2.xml (deflated 37%) 2024-04-03T18:34:20.2921409Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-4a3e67058d03b243.xml (deflated 37%) 2024-04-03T18:34:20.2923572Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-cff1d4449b0fd24a.xml (deflated 37%) 2024-04-03T18:34:20.2925730Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a62366083819497d.xml (deflated 37%) 2024-04-03T18:34:20.2927901Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e04b2f00c0b149ff.xml (deflated 38%) 2024-04-03T18:34:20.2930073Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-be210a124f2e7842.xml (deflated 38%) 2024-04-03T18:34:20.2932235Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-46f7d3241284510c.xml (deflated 38%) 2024-04-03T18:34:20.2934608Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9806ee22ac4663b6.xml (deflated 38%) 2024-04-03T18:34:20.2936803Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7d9394d45a36d1b1.xml (deflated 38%) 2024-04-03T18:34:20.2938977Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a15269cf72d59789.xml (deflated 38%) 2024-04-03T18:34:20.2941120Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-8866a1da7ca4999f.xml (deflated 38%) 2024-04-03T18:34:20.2943276Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a46d5420ff554740.xml (deflated 38%) 2024-04-03T18:34:20.2945591Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-0d487cca5fe600fe.xml (deflated 38%) 2024-04-03T18:34:20.2947749Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-eb70bfebdd766d74.xml (deflated 38%) 2024-04-03T18:34:20.2949927Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a5e12e8eca73a776.xml (deflated 37%) 2024-04-03T18:34:20.2952088Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-66be9f002ab2e071.xml (deflated 38%) 2024-04-03T18:34:20.2954578Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-13478f5befb8e020.xml (deflated 38%) 2024-04-03T18:34:20.2956749Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-062cf3a9070a58d7.xml (deflated 38%) 2024-04-03T18:34:20.2959042Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-f8aca599c427df37.xml (deflated 38%) 2024-04-03T18:34:20.2961735Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9374f4ccdb5d1799.xml (deflated 38%) 2024-04-03T18:34:20.2963982Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9183fff0fc2b0c56.xml (deflated 37%) 2024-04-03T18:34:20.2966148Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-50c79a6a08f545ff.xml (deflated 38%) 2024-04-03T18:34:20.2968299Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-449f15f7c10111ea.xml (deflated 38%) 2024-04-03T18:34:20.2970445Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-31b2e3eca0d84136.xml (deflated 38%) 2024-04-03T18:34:20.2972609Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-4f29324d203d6161.xml (deflated 37%) 2024-04-03T18:34:20.2974769Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-439b23f6e80a5e6a.xml (deflated 38%) 2024-04-03T18:34:20.2976922Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-634e9ed32499bbd6.xml (deflated 38%) 2024-04-03T18:34:20.2979082Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b4fae237ab3401fb.xml (deflated 37%) 2024-04-03T18:34:20.2981240Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a6915c1f9faa2785.xml (deflated 38%) 2024-04-03T18:34:20.2983405Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-27701b470643a3a5.xml (deflated 37%) 2024-04-03T18:34:20.2985529Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-f909d29b83497625.xml (deflated 38%) 2024-04-03T18:34:20.2987703Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-706bb6f0d3f5bf46.xml (deflated 38%) 2024-04-03T18:34:20.2990119Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7ed208af5b7072e0.xml (deflated 38%) 2024-04-03T18:34:20.2992424Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2210418b496622c7.xml (deflated 38%) 2024-04-03T18:34:20.2995559Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-42237432a92ee129.xml (deflated 38%) 2024-04-03T18:34:20.2997703Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-380961f57462848d.xml (deflated 37%) 2024-04-03T18:34:20.2999863Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7a050d53a5e4cc26.xml (deflated 37%) 2024-04-03T18:34:20.3002022Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5cdd7cc7dcc43343.xml (deflated 38%) 2024-04-03T18:34:20.3004330Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9d28a9f994a7e355.xml (deflated 47%) 2024-04-03T18:34:20.3006508Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-68a9b84e15c59b00.xml (deflated 38%) 2024-04-03T18:34:20.3008715Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-51c56dead2f8fd7e.xml (deflated 59%) 2024-04-03T18:34:20.3010919Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6dd28a1756c2ae3c.xml (deflated 37%) 2024-04-03T18:34:20.3013073Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-0e20182b623372f5.xml (deflated 37%) 2024-04-03T18:34:20.3015218Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-927c276405d61e8e.xml (deflated 37%) 2024-04-03T18:34:20.3017448Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7b207ae6d9f3a027.xml (deflated 38%) 2024-04-03T18:34:20.3019636Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-dbe95694495781b2.xml (deflated 39%) 2024-04-03T18:34:20.3021786Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-619ae939446b820b.xml (deflated 39%) 2024-04-03T18:34:20.3023939Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7fcad49742fc6cea.xml (deflated 38%) 2024-04-03T18:34:20.3026101Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-0b54e3f23f58e8c0.xml (deflated 38%) 2024-04-03T18:34:20.3028264Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-40635036564f0dc8.xml (deflated 38%) 2024-04-03T18:34:20.3030403Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-71ac3d6046628108.xml (deflated 38%) 2024-04-03T18:34:20.3032561Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-632f69d9f2342633.xml (deflated 37%) 2024-04-03T18:34:20.3035013Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-51cad8d102cb4368.xml (deflated 37%) 2024-04-03T18:34:20.3037185Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-34f9804478cd8cf6.xml (deflated 38%) 2024-04-03T18:34:20.3039326Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-c0b184371ce3dd89.xml (deflated 38%) 2024-04-03T18:34:20.3041494Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-3d4b25e7d0837c8b.xml (deflated 38%) 2024-04-03T18:34:20.3043659Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-585a4f9ee7f23faa.xml (deflated 38%) 2024-04-03T18:34:20.3045815Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-4823f5ec4921e28b.xml (deflated 38%) 2024-04-03T18:34:20.3048112Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5a42688ae7feed75.xml (deflated 37%) 2024-04-03T18:34:20.3050316Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-fcd66fa2b7cfd455.xml (deflated 38%) 2024-04-03T18:34:20.3052478Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-018b9ba0a3c56f1e.xml (deflated 38%) 2024-04-03T18:34:20.3054618Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b7dbb309610197ff.xml (deflated 38%) 2024-04-03T18:34:20.3056776Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-628902913129600e.xml (deflated 38%) 2024-04-03T18:34:20.3059048Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-edd733e6a6278e12.xml (deflated 38%) 2024-04-03T18:34:20.3061209Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-51c200dee710474a.xml (deflated 38%) 2024-04-03T18:34:20.3063354Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-705b7a90ccaf1433.xml (deflated 38%) 2024-04-03T18:34:20.3065501Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-321a92c8286b7506.xml (deflated 39%) 2024-04-03T18:34:20.3067661Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-cdee08141e98b14c.xml (deflated 38%) 2024-04-03T18:34:20.3069824Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2b55c208afdaa4cc.xml (deflated 38%) 2024-04-03T18:34:20.3071975Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-8c99ab5e00be7972.xml (deflated 38%) 2024-04-03T18:34:20.3074452Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7214cbfcc84a3abe.xml (deflated 38%) 2024-04-03T18:34:20.3076653Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-aa5fd2bdceaeddbc.xml (deflated 38%) 2024-04-03T18:34:20.3078825Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a4b195a6c4e37b48.xml (deflated 50%) 2024-04-03T18:34:20.3080972Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d9df95d28dfdf335.xml (deflated 50%) 2024-04-03T18:34:20.3083133Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2d9ce55b0f86e698.xml (deflated 38%) 2024-04-03T18:34:20.3085296Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-089e91289cc421e7.xml (deflated 38%) 2024-04-03T18:34:20.3087453Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e47f96e76a9757d8.xml (deflated 38%) 2024-04-03T18:34:20.3089593Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-4388d5e63e9133ac.xml (deflated 38%) 2024-04-03T18:34:20.3091744Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7e4d708593111432.xml (deflated 38%) 2024-04-03T18:34:20.3093890Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-86d8e177c6196448.xml (deflated 38%) 2024-04-03T18:34:20.3096681Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-63731a7a02837b63.xml (deflated 38%) 2024-04-03T18:34:20.3098844Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-0ecc895e1c9bd31a.xml (deflated 38%) 2024-04-03T18:34:20.3101006Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-976f3560b24d5087.xml (deflated 38%) 2024-04-03T18:34:20.3103162Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-23e4af825815c10d.xml (deflated 38%) 2024-04-03T18:34:20.3105465Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a42f69fc518db38e.xml (deflated 38%) 2024-04-03T18:34:20.3107656Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-38db4d2abdf3cedb.xml (deflated 36%) 2024-04-03T18:34:20.3109826Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-460d8ae37a140f5d.xml (deflated 36%) 2024-04-03T18:34:20.3111975Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-55b764732e8dc70e.xml (deflated 36%) 2024-04-03T18:34:20.3114549Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a937ab03ea46abcf.xml (deflated 36%) 2024-04-03T18:34:20.3116762Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-07409d7ceb34512f.xml (deflated 35%) 2024-04-03T18:34:20.3118918Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2417195c4f55cb24.xml (deflated 37%) 2024-04-03T18:34:20.3121080Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-c82e4d40734b4380.xml (deflated 37%) 2024-04-03T18:34:20.3123220Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2ae32c0baa556f5c.xml (deflated 38%) 2024-04-03T18:34:20.3125384Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-cb06ed6b25cef572.xml (deflated 38%) 2024-04-03T18:34:20.3127556Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-aa99fbe2e114037d.xml (deflated 38%) 2024-04-03T18:34:20.3129723Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-cb6f4ef944107962.xml (deflated 38%) 2024-04-03T18:34:20.3131874Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d4aba6befe701683.xml (deflated 38%) 2024-04-03T18:34:20.3134037Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7274e41316ec0772.xml (deflated 49%) 2024-04-03T18:34:20.3136192Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-090b2ceaa97cb9d5.xml (deflated 38%) 2024-04-03T18:34:20.3138364Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-ad139ed5208859ab.xml (deflated 38%) 2024-04-03T18:34:20.3140507Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5a0532c8a40ef436.xml (deflated 37%) 2024-04-03T18:34:20.3142665Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-39810aa8f4e5c45b.xml (deflated 37%) 2024-04-03T18:34:20.3144824Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7b8a9ad498c67718.xml (deflated 37%) 2024-04-03T18:34:20.3146966Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9b90f1cc681f1f93.xml (deflated 36%) 2024-04-03T18:34:20.3149131Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-27ee4e967d6ef0a8.xml (deflated 36%) 2024-04-03T18:34:20.3151637Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-58bcb3e83456f03f.xml (deflated 38%) 2024-04-03T18:34:20.3153829Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6c0f4fa1ef8f299b.xml (deflated 37%) 2024-04-03T18:34:20.3156330Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-178b643333d5fa5e.xml (deflated 37%) 2024-04-03T18:34:20.3158494Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-ddc933bce1a85020.xml (deflated 36%) 2024-04-03T18:34:20.3160803Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e37b816f1b3bbe28.xml (deflated 37%) 2024-04-03T18:34:20.3162982Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-36fe227b9342757b.xml (deflated 37%) 2024-04-03T18:34:20.3165112Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e023b56b5d885189.xml (deflated 37%) 2024-04-03T18:34:20.3167270Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9e178c91ac718b4c.xml (deflated 37%) 2024-04-03T18:34:20.3169427Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b7bfe194dc69a70b.xml (deflated 37%) 2024-04-03T18:34:20.3171706Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a200646854d821e1.xml (deflated 37%) 2024-04-03T18:34:20.3173853Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e7b8af04d92d9d78.xml (deflated 38%) 2024-04-03T18:34:20.3176024Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-0d5cd8a7fe2dbdab.xml (deflated 37%) 2024-04-03T18:34:20.3178183Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6f75748098079078.xml (deflated 38%) 2024-04-03T18:34:20.3180344Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6e4be9289d6a8650.xml (deflated 37%) 2024-04-03T18:34:20.3182491Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-84dd1ba8a97a94ff.xml (deflated 37%) 2024-04-03T18:34:20.3184655Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-69e581fb97ad1414.xml (deflated 37%) 2024-04-03T18:34:20.3186815Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-bb9d58830bc3d736.xml (deflated 37%) 2024-04-03T18:34:20.3188984Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-224b74f127532971.xml (deflated 37%) 2024-04-03T18:34:20.3191333Z adding: test/test-reports/python-pytest/distributed.elastic.events.lib_test/distributed.elastic.events.lib_test-0f2cc5a0726fbef7.xml (deflated 75%) 2024-04-03T18:34:20.3193917Z adding: test/test-reports/python-pytest/distributed.elastic.metrics.api_test/distributed.elastic.metrics.api_test-4f6e14f0dcf70c06.xml (deflated 60%) 2024-04-03T18:34:20.3196989Z adding: test/test-reports/python-pytest/distributed.elastic.multiprocessing.api_test/distributed.elastic.multiprocessing.api_test-63577873ef5ceace.xml (deflated 89%) 2024-04-03T18:34:20.3199895Z adding: test/test-reports/python-pytest/distributed.elastic.timer.local_timer_example/distributed.elastic.timer.local_timer_example-fe6561dffb86464d.xml (deflated 49%) 2024-04-03T18:34:20.3202693Z adding: test/test-reports/python-pytest/distributed.elastic.timer.local_timer_test/distributed.elastic.timer.local_timer_test-c378fef11d5632bd.xml (deflated 80%) 2024-04-03T18:34:20.3205484Z adding: test/test-reports/python-pytest/distributed.elastic.utils.distributed_test/distributed.elastic.utils.distributed_test-886e9faa63886d26.xml (deflated 77%) 2024-04-03T18:34:20.3208214Z adding: test/test-reports/python-pytest/distributed.elastic.utils.logging_test/distributed.elastic.utils.logging_test-79450d8eeb369036.xml (deflated 51%) 2024-04-03T18:34:20.3210820Z adding: test/test-reports/python-pytest/distributed.elastic.utils.util_test/distributed.elastic.utils.util_test-cffc76ce80e55b42.xml (deflated 74%) 2024-04-03T18:34:20.3213263Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-617812cdd9b5c34f.xml (deflated 37%) 2024-04-03T18:34:20.3215722Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a8e11fdbbccc6c8e.xml (deflated 38%) 2024-04-03T18:34:20.3218222Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-071dc9dcd805e138.xml (deflated 43%) 2024-04-03T18:34:20.3220601Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e6e6534c0f164e78.xml (deflated 43%) 2024-04-03T18:34:20.3222932Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-44c54e4f5a17d8ff.xml (deflated 43%) 2024-04-03T18:34:20.3225280Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a6a9ab26ed24d903.xml (deflated 38%) 2024-04-03T18:34:20.3227763Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d0f639af38f3a3e8.xml (deflated 38%) 2024-04-03T18:34:20.3230115Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3c130f3ebfff5276.xml (deflated 38%) 2024-04-03T18:34:20.3232452Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-85395a8d330b2872.xml (deflated 44%) 2024-04-03T18:34:20.3235120Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5b4629608293aa7e.xml (deflated 48%) 2024-04-03T18:34:20.3237477Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ec4d50c02b52258f.xml (deflated 43%) 2024-04-03T18:34:20.3239819Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-66ad1243a79470ed.xml (deflated 47%) 2024-04-03T18:34:20.3242154Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e155bc080ba93dee.xml (deflated 38%) 2024-04-03T18:34:20.3244515Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4bd6a5d1af3ed13f.xml (deflated 37%) 2024-04-03T18:34:20.3247087Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-75a936b1469cc316.xml (deflated 37%) 2024-04-03T18:34:20.3249446Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5394f748e4a91d9b.xml (deflated 42%) 2024-04-03T18:34:20.3251762Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-407bf8e9847846da.xml (deflated 42%) 2024-04-03T18:34:20.3254489Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a512eb276f7cb611.xml (deflated 38%) 2024-04-03T18:34:20.3256862Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f6efd0b8a1fb601b.xml (deflated 37%) 2024-04-03T18:34:20.3259223Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0ebab004eb46a028.xml (deflated 38%) 2024-04-03T18:34:20.3261552Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-41bf28803760542f.xml (deflated 38%) 2024-04-03T18:34:20.3263892Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-034f12bb1b826819.xml (deflated 43%) 2024-04-03T18:34:20.3266234Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d74e5853622e9e0.xml (deflated 43%) 2024-04-03T18:34:20.3268551Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-36e83944e177c035.xml (deflated 44%) 2024-04-03T18:34:20.3270889Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9fb29d554b46b814.xml (deflated 38%) 2024-04-03T18:34:20.3273219Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b700153136dd6f83.xml (deflated 38%) 2024-04-03T18:34:20.3276021Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f27b062fad0874b3.xml (deflated 43%) 2024-04-03T18:34:20.3278377Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ca435c1ed40fff2.xml (deflated 38%) 2024-04-03T18:34:20.3280723Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a3f86269fd9f5ff.xml (deflated 37%) 2024-04-03T18:34:20.3283070Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6cc5c0db23ad39b.xml (deflated 43%) 2024-04-03T18:34:20.3285567Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6acaff9ad1534bc5.xml (deflated 43%) 2024-04-03T18:34:20.3287887Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7bac367032666620.xml (deflated 43%) 2024-04-03T18:34:20.3290235Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aed333ae7d68cbef.xml (deflated 38%) 2024-04-03T18:34:20.3292587Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e48beae72f01b0e0.xml (deflated 45%) 2024-04-03T18:34:20.3294931Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-56bafd9ae91a3f7d.xml (deflated 38%) 2024-04-03T18:34:20.3297257Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-61597db465ce8f4c.xml (deflated 37%) 2024-04-03T18:34:20.3299606Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-adf269d12d53a4f6.xml (deflated 39%) 2024-04-03T18:34:20.3301950Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c05238ac82eba3c.xml (deflated 43%) 2024-04-03T18:34:20.3304304Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1cc9c107b8bc606a.xml (deflated 43%) 2024-04-03T18:34:20.3306629Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0220f3e1f80a49c3.xml (deflated 43%) 2024-04-03T18:34:20.3308965Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bf211ff48dc456d4.xml (deflated 38%) 2024-04-03T18:34:20.3311309Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6f99432287c337a.xml (deflated 38%) 2024-04-03T18:34:20.3313659Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-176dd490e09e5bae.xml (deflated 38%) 2024-04-03T18:34:20.3316345Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3b2cf520616ccd20.xml (deflated 44%) 2024-04-03T18:34:20.3318685Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c46b35e9019e1282.xml (deflated 49%) 2024-04-03T18:34:20.3321024Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4a9d7de33d353a56.xml (deflated 43%) 2024-04-03T18:34:20.3323366Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-58a619f5f2ea924f.xml (deflated 47%) 2024-04-03T18:34:20.3325689Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6e05bffccce6366.xml (deflated 38%) 2024-04-03T18:34:20.3328038Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a432ac25111434b4.xml (deflated 38%) 2024-04-03T18:34:20.3330512Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-65d8fef4ec2219d8.xml (deflated 37%) 2024-04-03T18:34:20.3332880Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1d4f73adbcbf62c4.xml (deflated 42%) 2024-04-03T18:34:20.3335202Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d7af1a49706009af.xml (deflated 42%) 2024-04-03T18:34:20.3337537Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-59b5852aea4280ac.xml (deflated 38%) 2024-04-03T18:34:20.3339875Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ab5d18d2166e469c.xml (deflated 37%) 2024-04-03T18:34:20.3342347Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-19302c6d0093476e.xml (deflated 39%) 2024-04-03T18:34:20.3344682Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dbee37bca09ed73b.xml (deflated 38%) 2024-04-03T18:34:20.3347029Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-294633bf839ecb9b.xml (deflated 43%) 2024-04-03T18:34:20.3349373Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b6dcaa143c5008a5.xml (deflated 43%) 2024-04-03T18:34:20.3351725Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-569d5f1a1e80fc60.xml (deflated 44%) 2024-04-03T18:34:20.3354041Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-54417bd090265ddb.xml (deflated 38%) 2024-04-03T18:34:20.3356721Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0850db63d139e6c9.xml (deflated 38%) 2024-04-03T18:34:20.3359074Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1ef3620abb73eaaf.xml (deflated 43%) 2024-04-03T18:34:20.3361423Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-400e426f0b7fedc0.xml (deflated 38%) 2024-04-03T18:34:20.3363750Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6f8cfc8acaaeae5f.xml (deflated 37%) 2024-04-03T18:34:20.3366091Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3139407f54358897.xml (deflated 43%) 2024-04-03T18:34:20.3368420Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2aca1c129658b045.xml (deflated 43%) 2024-04-03T18:34:20.3370765Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2d4d51ee50788bde.xml (deflated 43%) 2024-04-03T18:34:20.3373098Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e511aa6af38c1ca.xml (deflated 38%) 2024-04-03T18:34:20.3375448Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e6dd5e7531f00f98.xml (deflated 45%) 2024-04-03T18:34:20.3377790Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6af3a820c6f5f10e.xml (deflated 38%) 2024-04-03T18:34:20.3380131Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aa63e6436cc452ff.xml (deflated 38%) 2024-04-03T18:34:20.3382457Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b0d3c335272e101e.xml (deflated 43%) 2024-04-03T18:34:20.3384806Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cc53f83040318957.xml (deflated 39%) 2024-04-03T18:34:20.3387271Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-91687bb6695664ee.xml (deflated 46%) 2024-04-03T18:34:20.3389630Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b7c20e1acc53c0fe.xml (deflated 43%) 2024-04-03T18:34:20.3391953Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f848468c6fd47bc3.xml (deflated 37%) 2024-04-03T18:34:20.3394658Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-249b100525f259ed.xml (deflated 37%) 2024-04-03T18:34:20.3397012Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf1534734eb9f24e.xml (deflated 44%) 2024-04-03T18:34:20.3399487Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cc46df1f16add7c6.xml (deflated 44%) 2024-04-03T18:34:20.3401819Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a644edec00791e1.xml (deflated 43%) 2024-04-03T18:34:20.3404154Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-153526715450ecf3.xml (deflated 49%) 2024-04-03T18:34:20.3406489Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fda6667625de8671.xml (deflated 44%) 2024-04-03T18:34:20.3408828Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4a6f6f123964a89d.xml (deflated 44%) 2024-04-03T18:34:20.3411160Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-49b2f4ac4145f840.xml (deflated 44%) 2024-04-03T18:34:20.3413498Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b42fbfec20884747.xml (deflated 43%) 2024-04-03T18:34:20.3415892Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-715a0438ede433d4.xml (deflated 44%) 2024-04-03T18:34:20.3418234Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3f946ce83dd99b2f.xml (deflated 37%) 2024-04-03T18:34:20.3420557Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a1d58eafba81e7ea.xml (deflated 38%) 2024-04-03T18:34:20.3422909Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1fdb23da31d36abd.xml (deflated 43%) 2024-04-03T18:34:20.3425269Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3b8fd0a0d72588a1.xml (deflated 38%) 2024-04-03T18:34:20.3427603Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-36b00d4738c1c8e5.xml (deflated 37%) 2024-04-03T18:34:20.3429921Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-462793a5736fcfa6.xml (deflated 37%) 2024-04-03T18:34:20.3432261Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd8794bc75d798e4.xml (deflated 37%) 2024-04-03T18:34:20.3434897Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c7bbf2818ad0e421.xml (deflated 38%) 2024-04-03T18:34:20.3437258Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fce22a6dec2fb7a0.xml (deflated 37%) 2024-04-03T18:34:20.3439589Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-53e7ad058ed22851.xml (deflated 48%) 2024-04-03T18:34:20.3441932Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5fa1f7ca03af0560.xml (deflated 38%) 2024-04-03T18:34:20.3444433Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-78fbd2b162a0b643.xml (deflated 38%) 2024-04-03T18:34:20.3446808Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ffc45d581ebf5b6.xml (deflated 38%) 2024-04-03T18:34:20.3449240Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e0430a806347511.xml (deflated 38%) 2024-04-03T18:34:20.3451975Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d4177ef150c6db17.xml (deflated 38%) 2024-04-03T18:34:20.3454487Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c8524e64080e0bbe.xml (deflated 49%) 2024-04-03T18:34:20.3456831Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c5c2befc577e7cdf.xml (deflated 43%) 2024-04-03T18:34:20.3459176Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f5d5194ac5c6d50a.xml (deflated 43%) 2024-04-03T18:34:20.3461527Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-05c5d74c66dbbdb8.xml (deflated 38%) 2024-04-03T18:34:20.3463878Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-52d07a647464c453.xml (deflated 43%) 2024-04-03T18:34:20.3466217Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c81f2b2e878f089.xml (deflated 38%) 2024-04-03T18:34:20.3468653Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d4d7542e5da13830.xml (deflated 38%) 2024-04-03T18:34:20.3471384Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-78a60cd4f75a5bb3.xml (deflated 43%) 2024-04-03T18:34:20.3473752Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-91b3e5e3d85ae351.xml (deflated 37%) 2024-04-03T18:34:20.3476492Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-67405323daedf39a.xml (deflated 43%) 2024-04-03T18:34:20.3478810Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4e8871b2a5db62b.xml (deflated 43%) 2024-04-03T18:34:20.3481159Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2b152a814a5a3075.xml (deflated 37%) 2024-04-03T18:34:20.3483513Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b1c38d01b956056a.xml (deflated 39%) 2024-04-03T18:34:20.3485866Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e180e31e53fd6258.xml (deflated 43%) 2024-04-03T18:34:20.3488201Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2bd75ddc1c1fe6fe.xml (deflated 39%) 2024-04-03T18:34:20.3490550Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aad89315adb8732c.xml (deflated 46%) 2024-04-03T18:34:20.3492903Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b335b756101bd28.xml (deflated 43%) 2024-04-03T18:34:20.3495226Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b3cb49a4218e64fb.xml (deflated 38%) 2024-04-03T18:34:20.3497577Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-816acd4e5dd00119.xml (deflated 37%) 2024-04-03T18:34:20.3500077Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0ea3c6ffc9fa0a3a.xml (deflated 44%) 2024-04-03T18:34:20.3502450Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e6aad2bcd3c785c.xml (deflated 45%) 2024-04-03T18:34:20.3504776Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a4516ed026599997.xml (deflated 43%) 2024-04-03T18:34:20.3507116Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f91abf88764ad59f.xml (deflated 48%) 2024-04-03T18:34:20.3509458Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-62db93c8bf123d6f.xml (deflated 45%) 2024-04-03T18:34:20.3511934Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-87fe08a097aeae7d.xml (deflated 44%) 2024-04-03T18:34:20.3514595Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ac2ff2300b173482.xml (deflated 44%) 2024-04-03T18:34:20.3517033Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ee6bda4ad02e407d.xml (deflated 43%) 2024-04-03T18:34:20.3519378Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-98adc599a490f175.xml (deflated 44%) 2024-04-03T18:34:20.3521713Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-304089f112e1dad7.xml (deflated 38%) 2024-04-03T18:34:20.3524029Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-001e5fff7bd019b6.xml (deflated 38%) 2024-04-03T18:34:20.3526373Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-40d10051b8258328.xml (deflated 43%) 2024-04-03T18:34:20.3528718Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eae0ee504a08cd3e.xml (deflated 38%) 2024-04-03T18:34:20.3531065Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aae966308ded09dd.xml (deflated 37%) 2024-04-03T18:34:20.3533392Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2b6d6d1c0817eff7.xml (deflated 37%) 2024-04-03T18:34:20.3535731Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-443423a4a5c51501.xml (deflated 37%) 2024-04-03T18:34:20.3538065Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4a8fc6606c0d722a.xml (deflated 38%) 2024-04-03T18:34:20.3540415Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e3daf4539ed0fa9.xml (deflated 37%) 2024-04-03T18:34:20.3542748Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ddd246b5ad6a1620.xml (deflated 48%) 2024-04-03T18:34:20.3545098Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4aff44cc819c5225.xml (deflated 38%) 2024-04-03T18:34:20.3547440Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-096631d0fc80e29e.xml (deflated 38%) 2024-04-03T18:34:20.3549780Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-083a0bd09506cdd7.xml (deflated 38%) 2024-04-03T18:34:20.3552095Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0ba20504889c1d43.xml (deflated 38%) 2024-04-03T18:34:20.3555015Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b182bf1d1f926ddf.xml (deflated 38%) 2024-04-03T18:34:20.3557556Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c6cbb1e41c2236dc.xml (deflated 49%) 2024-04-03T18:34:20.3559933Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e96985c5bc88d20e.xml (deflated 43%) 2024-04-03T18:34:20.3562254Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-077b8a05b43ae23c.xml (deflated 43%) 2024-04-03T18:34:20.3564600Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24ea5ec11c98c2e3.xml (deflated 38%) 2024-04-03T18:34:20.3566945Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d5e4cf98c45be599.xml (deflated 43%) 2024-04-03T18:34:20.3569419Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5a1ac4ee910ba4e.xml (deflated 37%) 2024-04-03T18:34:20.3571757Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c2ba1c275893f37b.xml (deflated 38%) 2024-04-03T18:34:20.3574093Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a16e3c536d674008.xml (deflated 43%) 2024-04-03T18:34:20.3576440Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d13b34fc48bad7f3.xml (deflated 37%) 2024-04-03T18:34:20.3578791Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0697d863e73da65b.xml (deflated 43%) 2024-04-03T18:34:20.3581109Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-308750b7af83aa86.xml (deflated 43%) 2024-04-03T18:34:20.3583439Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-62626af0b23ee747.xml (deflated 37%) 2024-04-03T18:34:20.3585777Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b57f61336f1fd5b.xml (deflated 38%) 2024-04-03T18:34:20.3588113Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a5b8a525102db313.xml (deflated 39%) 2024-04-03T18:34:20.3590430Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-de86433c57ad3804.xml (deflated 37%) 2024-04-03T18:34:20.3592773Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f6f6d79844a6e2ae.xml (deflated 37%) 2024-04-03T18:34:20.3595593Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb1f89b83bbfa934.xml (deflated 38%) 2024-04-03T18:34:20.3597931Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1640da7842577947.xml (deflated 43%) 2024-04-03T18:34:20.3600252Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fecf24777702060f.xml (deflated 45%) 2024-04-03T18:34:20.3602585Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-115ef8395047e206.xml (deflated 44%) 2024-04-03T18:34:20.3604918Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-19dc3361e43fd842.xml (deflated 44%) 2024-04-03T18:34:20.3607247Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c90331e759497d7.xml (deflated 42%) 2024-04-03T18:34:20.3609575Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c3e6cde4fd967432.xml (deflated 37%) 2024-04-03T18:34:20.3611918Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b25ac76e2834fa4d.xml (deflated 47%) 2024-04-03T18:34:20.3614396Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3dbb4efe63277a24.xml (deflated 47%) 2024-04-03T18:34:20.3616801Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0798edb72ce4e28d.xml (deflated 37%) 2024-04-03T18:34:20.3619116Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e36088eef626e39a.xml (deflated 37%) 2024-04-03T18:34:20.3621452Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c935864b3630c01a.xml (deflated 38%) 2024-04-03T18:34:20.3623920Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e41b1889e4fe6db.xml (deflated 42%) 2024-04-03T18:34:20.3626261Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f2aea519af61146e.xml (deflated 38%) 2024-04-03T18:34:20.3628594Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f115e1adf3d70e21.xml (deflated 37%) 2024-04-03T18:34:20.3630937Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-603cfa40e44f93e4.xml (deflated 38%) 2024-04-03T18:34:20.3633278Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cae368102e72a922.xml (deflated 38%) 2024-04-03T18:34:20.3635915Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0560a69da762c8b0.xml (deflated 38%) 2024-04-03T18:34:20.3638233Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51999a4d190cb751.xml (deflated 38%) 2024-04-03T18:34:20.3640572Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6a52ac871f2fc9ad.xml (deflated 37%) 2024-04-03T18:34:20.3642925Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f02b13016553f14a.xml (deflated 45%) 2024-04-03T18:34:20.3645260Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f73540d2391bbe1.xml (deflated 45%) 2024-04-03T18:34:20.3647583Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d1c3e886eb706504.xml (deflated 43%) 2024-04-03T18:34:20.3649916Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ba287ad6b0721daa.xml (deflated 37%) 2024-04-03T18:34:20.3652277Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ce95a7e2c87c785.xml (deflated 37%) 2024-04-03T18:34:20.3654632Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a6039e9fce59b3d1.xml (deflated 38%) 2024-04-03T18:34:20.3657396Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-48443f0fdf57679a.xml (deflated 49%) 2024-04-03T18:34:20.3659731Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a37619507a75be6.xml (deflated 38%) 2024-04-03T18:34:20.3662081Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9ce8f0e967bfa2ff.xml (deflated 45%) 2024-04-03T18:34:20.3664434Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cac7ca8f0c723e6b.xml (deflated 38%) 2024-04-03T18:34:20.3666765Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-47229e30a5121645.xml (deflated 42%) 2024-04-03T18:34:20.3669232Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6c0d110384a851ee.xml (deflated 37%) 2024-04-03T18:34:20.3671577Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3236198dc939b112.xml (deflated 38%) 2024-04-03T18:34:20.3696379Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a8c2a38d5a9a24b3.xml (deflated 37%) 2024-04-03T18:34:20.3698774Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7faa842957e31153.xml (deflated 37%) 2024-04-03T18:34:20.3701112Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c8db380607e937d0.xml (deflated 38%) 2024-04-03T18:34:20.3703713Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e043708f0ed6619d.xml (deflated 43%) 2024-04-03T18:34:20.3706074Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bc0b874b46bdbd9c.xml (deflated 45%) 2024-04-03T18:34:20.3708447Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bb3b8e6ffd0e21b7.xml (deflated 44%) 2024-04-03T18:34:20.3710789Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5bcdb47fdde68e0c.xml (deflated 44%) 2024-04-03T18:34:20.3713145Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f3700f21cab04723.xml (deflated 41%) 2024-04-03T18:34:20.3715824Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-36d5baae82e28688.xml (deflated 38%) 2024-04-03T18:34:20.3718185Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5bebe992e6824ec3.xml (deflated 47%) 2024-04-03T18:34:20.3720523Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7d336d75e1bc9bf6.xml (deflated 47%) 2024-04-03T18:34:20.3722869Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-59c4d70e97329c74.xml (deflated 37%) 2024-04-03T18:34:20.3725200Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b33a9cc6f57785f4.xml (deflated 38%) 2024-04-03T18:34:20.3727526Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cda46baba7cd27fa.xml (deflated 38%) 2024-04-03T18:34:20.3729850Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-53b54130512b1293.xml (deflated 42%) 2024-04-03T18:34:20.3732190Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ac73f2429261caff.xml (deflated 38%) 2024-04-03T18:34:20.3734552Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b260b0b85e463a6d.xml (deflated 38%) 2024-04-03T18:34:20.3736877Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b5b1c23e182e8fca.xml (deflated 39%) 2024-04-03T18:34:20.3739191Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-84e4fb29c54f95d7.xml (deflated 38%) 2024-04-03T18:34:20.3741534Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-94cfeba6aad9596c.xml (deflated 38%) 2024-04-03T18:34:20.3743880Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6f9f8070fbf93529.xml (deflated 38%) 2024-04-03T18:34:20.3746200Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-47a465cc12b06c0b.xml (deflated 37%) 2024-04-03T18:34:20.3748644Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9fa74aecf8590fd6.xml (deflated 45%) 2024-04-03T18:34:20.3751007Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3a748fae54801ad5.xml (deflated 45%) 2024-04-03T18:34:20.3753355Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b0c80f456be8b11.xml (deflated 43%) 2024-04-03T18:34:20.3755937Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-01a980194dd3913e.xml (deflated 38%) 2024-04-03T18:34:20.3758257Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2f83d53089dcede6.xml (deflated 37%) 2024-04-03T18:34:20.3760751Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a1ffe8f0a6dfb55e.xml (deflated 38%) 2024-04-03T18:34:20.3763105Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-28fdd11f4e909645.xml (deflated 49%) 2024-04-03T18:34:20.3765449Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4bbc190bb7184175.xml (deflated 38%) 2024-04-03T18:34:20.3767764Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0a18741e03da913c.xml (deflated 45%) 2024-04-03T18:34:20.3770102Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b11fd2eb91c07857.xml (deflated 38%) 2024-04-03T18:34:20.3772451Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0919342988233c9e.xml (deflated 41%) 2024-04-03T18:34:20.3774782Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-afadce9d1f650045.xml (deflated 39%) 2024-04-03T18:34:20.3777126Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a83a2fdfdc8e1d8b.xml (deflated 43%) 2024-04-03T18:34:20.3779471Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-57479a03763907fd.xml (deflated 39%) 2024-04-03T18:34:20.3781809Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0d4425cf7f003251.xml (deflated 37%) 2024-04-03T18:34:20.3784149Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8ae4ba3e7e4602af.xml (deflated 38%) 2024-04-03T18:34:20.3786486Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e9deebb2f33e52a8.xml (deflated 45%) 2024-04-03T18:34:20.3788829Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6c6538245449dc82.xml (deflated 37%) 2024-04-03T18:34:20.3791165Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89b71b3da7030088.xml (deflated 38%) 2024-04-03T18:34:20.3793503Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b254a62f57c1d25f.xml (deflated 38%) 2024-04-03T18:34:20.3796142Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4b2089b3bd839664.xml (deflated 38%) 2024-04-03T18:34:20.3798486Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6dfef6691a669aff.xml (deflated 45%) 2024-04-03T18:34:20.3800840Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-429fa2763bd54cc7.xml (deflated 44%) 2024-04-03T18:34:20.3803181Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2fecb8f9d672d965.xml (deflated 44%) 2024-04-03T18:34:20.3805623Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-38717e0084d22f7f.xml (deflated 44%) 2024-04-03T18:34:20.3807976Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fc515850225b3e88.xml (deflated 37%) 2024-04-03T18:34:20.3810311Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30420f8a0ca8176a.xml (deflated 38%) 2024-04-03T18:34:20.3812628Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4b24c54e9f544d4a.xml (deflated 42%) 2024-04-03T18:34:20.3815161Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30c8d8966a114046.xml (deflated 38%) 2024-04-03T18:34:20.3817538Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fc491522d2ded5cb.xml (deflated 49%) 2024-04-03T18:34:20.3819888Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f4dd1b6ec7592850.xml (deflated 38%) 2024-04-03T18:34:20.3822216Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b9e2a5f00c77de7.xml (deflated 37%) 2024-04-03T18:34:20.3824558Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-289cf57785f9c19b.xml (deflated 37%) 2024-04-03T18:34:20.3826904Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d7608b87a0bb3c63.xml (deflated 37%) 2024-04-03T18:34:20.3829248Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6e03a6173597e047.xml (deflated 38%) 2024-04-03T18:34:20.3831558Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-878f664fb532c722.xml (deflated 38%) 2024-04-03T18:34:20.3833903Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6edbfac7f42706ff.xml (deflated 49%) 2024-04-03T18:34:20.3836514Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-88da312bcf8b684e.xml (deflated 38%) 2024-04-03T18:34:20.3838869Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d3df322a50e2090e.xml (deflated 38%) 2024-04-03T18:34:20.3841179Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-903d541b05cba792.xml (deflated 38%) 2024-04-03T18:34:20.3843522Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-36d9f318a278bf28.xml (deflated 37%) 2024-04-03T18:34:20.3845850Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-85d0bf39df534346.xml (deflated 38%) 2024-04-03T18:34:20.3848196Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ed4bc76825897c9.xml (deflated 49%) 2024-04-03T18:34:20.3850511Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aa5fb8f10113466f.xml (deflated 43%) 2024-04-03T18:34:20.3852850Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e9c1f30f4b733882.xml (deflated 38%) 2024-04-03T18:34:20.3855186Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b53b190a3cd7d0a.xml (deflated 43%) 2024-04-03T18:34:20.3857541Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d22ee17f020a2030.xml (deflated 49%) 2024-04-03T18:34:20.3859989Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-88412d24cf9aa1dd.xml (deflated 37%) 2024-04-03T18:34:20.3862340Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b6e884079fc20864.xml (deflated 45%) 2024-04-03T18:34:20.3864675Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-60b8db23895336a5.xml (deflated 43%) 2024-04-03T18:34:20.3867016Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1d0f4e87b9c57db1.xml (deflated 37%) 2024-04-03T18:34:20.3869332Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-069108dc4ca87444.xml (deflated 49%) 2024-04-03T18:34:20.3871792Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-18313d74b7dc1b93.xml (deflated 49%) 2024-04-03T18:34:20.3874132Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0aa49df7e58b720c.xml (deflated 38%) 2024-04-03T18:34:20.3876729Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39999e1d87dff427.xml (deflated 38%) 2024-04-03T18:34:20.3879043Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51ee204b6c7c1314.xml (deflated 43%) 2024-04-03T18:34:20.3881373Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-595563083a00ab86.xml (deflated 39%) 2024-04-03T18:34:20.3883708Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a046b61b6cfbf6c7.xml (deflated 38%) 2024-04-03T18:34:20.3886064Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d96fcd04ab371c6f.xml (deflated 38%) 2024-04-03T18:34:20.3888389Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-333597cb96e91c3e.xml (deflated 45%) 2024-04-03T18:34:20.3890734Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a8cd4e7c330b44f.xml (deflated 37%) 2024-04-03T18:34:20.3893083Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e393b26d8d3c0d74.xml (deflated 38%) 2024-04-03T18:34:20.3895430Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3119bd1c32871fba.xml (deflated 38%) 2024-04-03T18:34:20.3897749Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c8e63c5434666af8.xml (deflated 38%) 2024-04-03T18:34:20.3900101Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9fa352aa22f4f140.xml (deflated 44%) 2024-04-03T18:34:20.3902445Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7426c51cbf365bbf.xml (deflated 44%) 2024-04-03T18:34:20.3904794Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8fd6e7efaa2b26c3.xml (deflated 44%) 2024-04-03T18:34:20.3907126Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dcd9b6b2d83791d0.xml (deflated 44%) 2024-04-03T18:34:20.3909477Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bafba252d78f29b8.xml (deflated 38%) 2024-04-03T18:34:20.3911823Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34603de45de40da0.xml (deflated 38%) 2024-04-03T18:34:20.3914403Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9ecfd7b245ea16e4.xml (deflated 42%) 2024-04-03T18:34:20.3916928Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3b3fe6e353093845.xml (deflated 38%) 2024-04-03T18:34:20.3919289Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51507aaf3a237728.xml (deflated 49%) 2024-04-03T18:34:20.3921626Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ba17a6dc438775b4.xml (deflated 38%) 2024-04-03T18:34:20.3923977Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e43be56c9b60fc3.xml (deflated 37%) 2024-04-03T18:34:20.3926306Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-48b9b659911686fc.xml (deflated 37%) 2024-04-03T18:34:20.3928776Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-76fb88d4f94e1a1b.xml (deflated 37%) 2024-04-03T18:34:20.3931119Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-33020ea63c23b5d8.xml (deflated 38%) 2024-04-03T18:34:20.3933459Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1196e55d902a5fb.xml (deflated 38%) 2024-04-03T18:34:20.3935776Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1ee5adb965455872.xml (deflated 49%) 2024-04-03T18:34:20.3938107Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fe62127773a81e61.xml (deflated 38%) 2024-04-03T18:34:20.3940434Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f477836eae7fe9e4.xml (deflated 38%) 2024-04-03T18:34:20.3942777Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3822a6ae81670579.xml (deflated 38%) 2024-04-03T18:34:20.3945097Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-919fe55f1674705b.xml (deflated 38%) 2024-04-03T18:34:20.3947427Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-67530234e0fea977.xml (deflated 38%) 2024-04-03T18:34:20.3949770Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2bf2bdaf50c0fcec.xml (deflated 49%) 2024-04-03T18:34:20.3952108Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6c0f53f61689f074.xml (deflated 43%) 2024-04-03T18:34:20.3954669Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2dfe3602f1fd35fd.xml (deflated 38%) 2024-04-03T18:34:20.3957036Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-179dac6e44201eac.xml (deflated 43%) 2024-04-03T18:34:20.3959386Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6c8908237cccd87.xml (deflated 49%) 2024-04-03T18:34:20.3961724Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1dea769c369b41c0.xml (deflated 38%) 2024-04-03T18:34:20.3964051Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-558ce4f0cb3ea6ea.xml (deflated 45%) 2024-04-03T18:34:20.3966401Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c98abdadd3c4341.xml (deflated 43%) 2024-04-03T18:34:20.3968745Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f387c000d604fc46.xml (deflated 37%) 2024-04-03T18:34:20.3971074Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-33635bd2dfd42f73.xml (deflated 49%) 2024-04-03T18:34:20.3973514Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5443b7815ac0268a.xml (deflated 49%) 2024-04-03T18:34:20.3975871Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1aac6c12556249d2.xml (deflated 38%) 2024-04-03T18:34:20.3978212Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4eb36d2ab259d09.xml (deflated 38%) 2024-04-03T18:34:20.3980553Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a84791a209c472da.xml (deflated 39%) 2024-04-03T18:34:20.3983000Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6ae73e82e8290af0.xml (deflated 37%) 2024-04-03T18:34:20.3985333Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-285b36a905a38c3e.xml (deflated 38%) 2024-04-03T18:34:20.3987676Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9d29867f055f7b43.xml (deflated 38%) 2024-04-03T18:34:20.3989995Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9543f05ac258ab9b.xml (deflated 43%) 2024-04-03T18:34:20.3992299Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e97492c294d015c.xml (deflated 45%) 2024-04-03T18:34:20.3994916Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e388c4cf8c07d170.xml (deflated 44%) 2024-04-03T18:34:20.3997282Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ef7c6cdcc55385e4.xml (deflated 44%) 2024-04-03T18:34:20.3999614Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f1ec8d359af2ec9.xml (deflated 41%) 2024-04-03T18:34:20.4001931Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c6608da6b262dc9.xml (deflated 37%) 2024-04-03T18:34:20.4004261Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1233df87fc662843.xml (deflated 47%) 2024-04-03T18:34:20.4006588Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a0ffb9d84458afa.xml (deflated 47%) 2024-04-03T18:34:20.4008911Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96c1deb1e5242987.xml (deflated 37%) 2024-04-03T18:34:20.4011233Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e72a9fa1e705cba.xml (deflated 37%) 2024-04-03T18:34:20.4013560Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-29cfb2fbce873b7e.xml (deflated 38%) 2024-04-03T18:34:20.4015941Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-258c82f671e0fe89.xml (deflated 42%) 2024-04-03T18:34:20.4018268Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8166c1b80e00b228.xml (deflated 38%) 2024-04-03T18:34:20.4020583Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b7b524b1d8cc0bee.xml (deflated 38%) 2024-04-03T18:34:20.4022909Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-02a25e77212be963.xml (deflated 38%) 2024-04-03T18:34:20.4025247Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af5e18abb71c2109.xml (deflated 38%) 2024-04-03T18:34:20.4027563Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1d423bcb07549f66.xml (deflated 38%) 2024-04-03T18:34:20.4030004Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cba66ee775522614.xml (deflated 38%) 2024-04-03T18:34:20.4032348Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-453b4d5bb8ef6a60.xml (deflated 44%) 2024-04-03T18:34:20.4034916Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3286959499fecf43.xml (deflated 45%) 2024-04-03T18:34:20.4037233Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e164a297ef98e18.xml (deflated 45%) 2024-04-03T18:34:20.4039697Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e56aaeca5bfc898.xml (deflated 43%) 2024-04-03T18:34:20.4042025Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7e632b30ec43f5c6.xml (deflated 38%) 2024-04-03T18:34:20.4044356Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8d8f41d7295320f8.xml (deflated 37%) 2024-04-03T18:34:20.4046669Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d880dbb79f5ebf30.xml (deflated 37%) 2024-04-03T18:34:20.4049003Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e2d714165eae639.xml (deflated 49%) 2024-04-03T18:34:20.4051324Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e1661c0051c6142.xml (deflated 38%) 2024-04-03T18:34:20.4053656Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c04e72fc94a17ad.xml (deflated 45%) 2024-04-03T18:34:20.4055951Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-52843e9d89775159.xml (deflated 37%) 2024-04-03T18:34:20.4058275Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0f6f7849e82e3b1f.xml (deflated 41%) 2024-04-03T18:34:20.4061115Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89ccad97e6a56c5e.xml (deflated 38%) 2024-04-03T18:34:20.4063466Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bdbcc0e742ea5400.xml (deflated 38%) 2024-04-03T18:34:20.4065776Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1485a847376e4df2.xml (deflated 37%) 2024-04-03T18:34:20.4068108Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6e61640bb4df0af3.xml (deflated 38%) 2024-04-03T18:34:20.4070445Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6fd2773c5ed2ffb2.xml (deflated 38%) 2024-04-03T18:34:20.4072772Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-daa408c8be302376.xml (deflated 43%) 2024-04-03T18:34:20.4075336Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-255caff10ec6145d.xml (deflated 45%) 2024-04-03T18:34:20.4077682Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ed02822da92fecc7.xml (deflated 44%) 2024-04-03T18:34:20.4080009Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a32f82fc61ce7cfc.xml (deflated 44%) 2024-04-03T18:34:20.4082350Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc618d502af57b7d.xml (deflated 41%) 2024-04-03T18:34:20.4084793Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cc2fbd917110b4c3.xml (deflated 37%) 2024-04-03T18:34:20.4087132Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7e82a6876005921e.xml (deflated 47%) 2024-04-03T18:34:20.4089456Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-948d2e0518af9136.xml (deflated 48%) 2024-04-03T18:34:20.4091780Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b3224e3fbb455ebf.xml (deflated 37%) 2024-04-03T18:34:20.4094084Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5979cf5b13ecd70f.xml (deflated 38%) 2024-04-03T18:34:20.4096560Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70ebbb6ceeb05f57.xml (deflated 38%) 2024-04-03T18:34:20.4098899Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-afc80122acba30ef.xml (deflated 42%) 2024-04-03T18:34:20.4101240Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-be3a8d9ea47ae684.xml (deflated 38%) 2024-04-03T18:34:20.4103546Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-93433817f511d20c.xml (deflated 38%) 2024-04-03T18:34:20.4105858Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-989f94eec0f70023.xml (deflated 38%) 2024-04-03T18:34:20.4108179Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f74eae184239226c.xml (deflated 38%) 2024-04-03T18:34:20.4110516Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b004889bb7ae9374.xml (deflated 38%) 2024-04-03T18:34:20.4112825Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-10afd83914e309af.xml (deflated 38%) 2024-04-03T18:34:20.4115503Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8c427f9c25f746a3.xml (deflated 44%) 2024-04-03T18:34:20.4117873Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39a072c614378b85.xml (deflated 45%) 2024-04-03T18:34:20.4120196Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fe19750b861d5329.xml (deflated 45%) 2024-04-03T18:34:20.4122501Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ee98e0ffa227ed4d.xml (deflated 43%) 2024-04-03T18:34:20.4124847Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-79007df9a77e563d.xml (deflated 38%) 2024-04-03T18:34:20.4127175Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f3b0ce8dcdf7869a.xml (deflated 37%) 2024-04-03T18:34:20.4129503Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f96f4d45016c95ac.xml (deflated 37%) 2024-04-03T18:34:20.4131809Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1cb64d869fb8ec10.xml (deflated 49%) 2024-04-03T18:34:20.4134141Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4468589e1f51fef5.xml (deflated 38%) 2024-04-03T18:34:20.4136465Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1bc23f8a0a46c53b.xml (deflated 45%) 2024-04-03T18:34:20.4138806Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-729e8e074dc9a6bf.xml (deflated 38%) 2024-04-03T18:34:20.4141227Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a019c148d91cbefb.xml (deflated 41%) 2024-04-03T18:34:20.4143579Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6117746affbe37ed.xml (deflated 39%) 2024-04-03T18:34:20.4145903Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1a96ed85629ed5ef.xml (deflated 43%) 2024-04-03T18:34:20.4148235Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-80ac4123b4c831fe.xml (deflated 39%) 2024-04-03T18:34:20.4150552Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7bcb2b0101c898a1.xml (deflated 45%) 2024-04-03T18:34:20.4153008Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7d447f1df0cf9ded.xml (deflated 38%) 2024-04-03T18:34:20.4155589Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-afd4e490fe781f0b.xml (deflated 45%) 2024-04-03T18:34:20.4157914Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7304f3782393c3b9.xml (deflated 37%) 2024-04-03T18:34:20.4160551Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-49104d435234f4bc.xml (deflated 44%) 2024-04-03T18:34:20.4162937Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ff364f2b29712446.xml (deflated 44%) 2024-04-03T18:34:20.4165254Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-293e747aa98992ab.xml (deflated 38%) 2024-04-03T18:34:20.4167575Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a082904b16b742bf.xml (deflated 45%) 2024-04-03T18:34:20.4169882Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3b30c80a28ceabe.xml (deflated 44%) 2024-04-03T18:34:20.4172198Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-678290c41f421612.xml (deflated 44%) 2024-04-03T18:34:20.4174512Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cba8859bb0ec2318.xml (deflated 44%) 2024-04-03T18:34:20.4176821Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c515899b289b7a07.xml (deflated 43%) 2024-04-03T18:34:20.4179113Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15525996cdb49c73.xml (deflated 44%) 2024-04-03T18:34:20.4181434Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2b34cd43087567fe.xml (deflated 42%) 2024-04-03T18:34:20.4183750Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b5c35b02a085d945.xml (deflated 44%) 2024-04-03T18:34:20.4186068Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-90a3dd65ca1f8018.xml (deflated 49%) 2024-04-03T18:34:20.4188359Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3981b0fc1d21966f.xml (deflated 38%) 2024-04-03T18:34:20.4190679Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e7f30d069d589b31.xml (deflated 37%) 2024-04-03T18:34:20.4193000Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0073465adab5d35d.xml (deflated 37%) 2024-04-03T18:34:20.4195642Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-73735c3efa310041.xml (deflated 37%) 2024-04-03T18:34:20.4198087Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a4c1c5d0eef1825c.xml (deflated 38%) 2024-04-03T18:34:20.4200430Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2fa9b077383a12f8.xml (deflated 38%) 2024-04-03T18:34:20.4202746Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-207777498c4f4685.xml (deflated 49%) 2024-04-03T18:34:20.4205050Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8c1efa127843d814.xml (deflated 38%) 2024-04-03T18:34:20.4207350Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a74895e396fb3137.xml (deflated 38%) 2024-04-03T18:34:20.4209786Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-79b84c4767a7293a.xml (deflated 38%) 2024-04-03T18:34:20.4212110Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51ffd7833e76f795.xml (deflated 38%) 2024-04-03T18:34:20.4214424Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6a4482fbefdd3cce.xml (deflated 38%) 2024-04-03T18:34:20.4216763Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-790baa4a24cc7929.xml (deflated 49%) 2024-04-03T18:34:20.4219089Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0d86d3df27fceb86.xml (deflated 43%) 2024-04-03T18:34:20.4221435Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b403eb5f3ac878fd.xml (deflated 43%) 2024-04-03T18:34:20.4223761Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fc1637a5dd905c77.xml (deflated 43%) 2024-04-03T18:34:20.4226065Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15a970ca579ed369.xml (deflated 43%) 2024-04-03T18:34:20.4228375Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4db0a707960f7e47.xml (deflated 43%) 2024-04-03T18:34:20.4230699Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e8c7c1fce8265cac.xml (deflated 45%) 2024-04-03T18:34:20.4233025Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a4244b97ca23edda.xml (deflated 42%) 2024-04-03T18:34:20.4235649Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c982779a6da50051.xml (deflated 37%) 2024-04-03T18:34:20.4237965Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b4a6147aa15ff8d1.xml (deflated 49%) 2024-04-03T18:34:20.4240293Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b1a19ea6f29eb2a5.xml (deflated 49%) 2024-04-03T18:34:20.4242614Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9092473283060fc6.xml (deflated 45%) 2024-04-03T18:34:20.4244908Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3294bc1226ef6742.xml (deflated 38%) 2024-04-03T18:34:20.4247232Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39a902bd1fcb707c.xml (deflated 43%) 2024-04-03T18:34:20.4249566Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dd289f9415879d45.xml (deflated 39%) 2024-04-03T18:34:20.4251893Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f30c3f3d3b2ef1b.xml (deflated 45%) 2024-04-03T18:34:20.4254321Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e98b3c95a8047252.xml (deflated 38%) 2024-04-03T18:34:20.4256663Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ba113da0f4a1a14b.xml (deflated 45%) 2024-04-03T18:34:20.4258990Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-37d55accd49dcde6.xml (deflated 38%) 2024-04-03T18:34:20.4261309Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7607e7c809928ec2.xml (deflated 44%) 2024-04-03T18:34:20.4263732Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15d34590aa33537e.xml (deflated 44%) 2024-04-03T18:34:20.4266051Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-668914f2cc61d37e.xml (deflated 38%) 2024-04-03T18:34:20.4268379Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6bb7dedc3a577d31.xml (deflated 44%) 2024-04-03T18:34:20.4270697Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0e25a913f7048e9e.xml (deflated 45%) 2024-04-03T18:34:20.4273014Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c874057e5008d9fb.xml (deflated 44%) 2024-04-03T18:34:20.4275580Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-930528ddc74a33b2.xml (deflated 44%) 2024-04-03T18:34:20.4277924Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6b25e6f81ad054a4.xml (deflated 43%) 2024-04-03T18:34:20.4280216Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c661641989553454.xml (deflated 44%) 2024-04-03T18:34:20.4282543Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2ed7958b10aef7a3.xml (deflated 42%) 2024-04-03T18:34:20.4284868Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2309b102a196cfda.xml (deflated 45%) 2024-04-03T18:34:20.4287196Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-944572cdc3af17c8.xml (deflated 49%) 2024-04-03T18:34:20.4289494Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6425c9d91fb61b3d.xml (deflated 38%) 2024-04-03T18:34:20.4291821Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-07cafc3276a4946f.xml (deflated 37%) 2024-04-03T18:34:20.4294151Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d7be4e8aebabbe2e.xml (deflated 37%) 2024-04-03T18:34:20.4296499Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-abf90f7fe53cbdad.xml (deflated 37%) 2024-04-03T18:34:20.4298814Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ff3fdbf099e23045.xml (deflated 38%) 2024-04-03T18:34:20.4301142Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-90bbeeb961676541.xml (deflated 38%) 2024-04-03T18:34:20.4303463Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f38d42e31292924f.xml (deflated 49%) 2024-04-03T18:34:20.4305791Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-403052a213543f0e.xml (deflated 37%) 2024-04-03T18:34:20.4308093Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-08fcdd219deb668b.xml (deflated 38%) 2024-04-03T18:34:20.4310558Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9861d0d6b47f3813.xml (deflated 38%) 2024-04-03T18:34:20.4312904Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-86275dd91acb87a2.xml (deflated 38%) 2024-04-03T18:34:20.4315483Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4b7d7bfede225a70.xml (deflated 38%) 2024-04-03T18:34:20.4317832Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4be2ba85cad8ab43.xml (deflated 49%) 2024-04-03T18:34:20.4320297Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f10a7409e50b701e.xml (deflated 43%) 2024-04-03T18:34:20.4322619Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89482e369e98b536.xml (deflated 43%) 2024-04-03T18:34:20.4324952Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2f12896d2095ee17.xml (deflated 43%) 2024-04-03T18:34:20.4327257Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-510e35f2ead96937.xml (deflated 43%) 2024-04-03T18:34:20.4329574Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-66ab31baae27e5ec.xml (deflated 43%) 2024-04-03T18:34:20.4331902Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca98f48e96d27011.xml (deflated 45%) 2024-04-03T18:34:20.4334243Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b347f27d349623a3.xml (deflated 43%) 2024-04-03T18:34:20.4336547Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-319a253e064e10c6.xml (deflated 37%) 2024-04-03T18:34:20.4338870Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e38658265d602dfa.xml (deflated 49%) 2024-04-03T18:34:20.4341201Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ba4e4c18d67c949f.xml (deflated 49%) 2024-04-03T18:34:20.4343534Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c73b0eef40ed648.xml (deflated 45%) 2024-04-03T18:34:20.4380465Z ##[group]Run # Remove any previous test reports if they exist 2024-04-03T18:34:20.4381400Z # Remove any previous test reports if they exist 2024-04-03T18:34:20.4382129Z rm -f logs-*.zip 2024-04-03T18:34:20.4383087Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-04-03T18:34:20.4384222Z # so check to see if the file exists first 2024-04-03T18:34:20.4384923Z if [ -f 'usage_log.txt' ]; then 2024-04-03T18:34:20.4385675Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-04-03T18:34:20.4386364Z fi 2024-04-03T18:34:20.4386855Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-04-03T18:34:20.4387672Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-04-03T18:34:20.4388368Z fi 2024-04-03T18:34:20.4397668Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:34:20.4398360Z env: 2024-04-03T18:34:20.4398738Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:34:20.4399378Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:34:20.4400419Z DOCKER_CONTAINER_ID: 7033b537529c546187ef7c776250a79cad9cc38d4987347f20cdbf86b60e2c90 2024-04-03T18:34:20.4401642Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_23402706124 2024-04-03T18:34:20.4402494Z ##[endgroup] 2024-04-03T18:34:20.5027051Z adding: usage_log.txt (deflated 96%) 2024-04-03T18:34:20.5342098Z adding: test/test-reports/distributed.fsdp.test_fsdp_memory_1.1_4ae9041f2b2dc8af_.log (deflated 74%) 2024-04-03T18:34:20.5343914Z adding: test/test-reports/distributed.checkpoint.test_nested_dict_1.1_3521e461febaea11_.log (deflated 59%) 2024-04-03T18:34:20.5345620Z adding: test/test-reports/distributed.test_collective_utils_1.1_001224b090664ae4_.log (stored 0%) 2024-04-03T18:34:20.5347246Z adding: test/test-reports/distributed._tensor.test_dtensor_1.1_4e5b12d772640e2b_.log (deflated 92%) 2024-04-03T18:34:20.5348941Z adding: test/test-reports/distributed._tensor.test_convolution_ops_1.1_1dbec8f6ffda35bd_.log (deflated 59%) 2024-04-03T18:34:20.5350671Z adding: test/test-reports/distributed._spmd.test_transformation_1.1_e661aec4dca40fac_.log (stored 0%) 2024-04-03T18:34:20.5352496Z adding: test/test-reports/distributed._tensor.test_math_ops_1.1_9924f64b0a02ac38_.log (deflated 70%) 2024-04-03T18:34:20.5354517Z adding: test/test-reports/distributed.test_distributed_spawn_3.7_c8bbdcd6db047027_.log (deflated 12%) 2024-04-03T18:34:20.5356155Z adding: test/test-reports/distributed.test_fake_pg_1.1_4bc808b8a6003c0a_.log (deflated 72%) 2024-04-03T18:34:20.5357760Z adding: test/test-reports/distributed.test_distributed_spawn_3.7_bd3266d02baf3957_.log (deflated 12%) 2024-04-03T18:34:20.5359384Z adding: test/test-reports/distributed.test_c10d_pypg_1.1_6fc1928ee185ac1d_.log (deflated 89%) 2024-04-03T18:34:20.5362583Z adding: test/test-reports/distributed.test_distributed_spawn_3.7_a938b6a40ce37f58_.log (deflated 94%) 2024-04-03T18:34:20.5364237Z adding: test/test-reports/distributed._tensor.test_utils_1.1_e09c6285312fccae_.log (deflated 59%) 2024-04-03T18:34:20.5371928Z adding: test/test-reports/distributed.test_distributed_spawn_3.7_905aac7628a68065_.log (deflated 94%) 2024-04-03T18:34:20.5373617Z adding: test/test-reports/distributed.checkpoint.test_utils_1.1_1032277d79a86a08_.log (deflated 64%) 2024-04-03T18:34:20.5383426Z adding: test/test-reports/distributed.test_distributed_spawn_3.7_39f019b06941c4a0_.log (deflated 93%) 2024-04-03T18:34:20.5385109Z adding: test/test-reports/distributed._tensor.test_view_ops_1.1_e90c3601af8ea970_.log (deflated 59%) 2024-04-03T18:34:20.5395210Z adding: test/test-reports/distributed.test_distributed_spawn_3.7_2b5384d13697351a_.log (deflated 93%) 2024-04-03T18:34:20.5396861Z adding: test/test-reports/distributed._shard.test_sharder_1.1_4c23b15150b8eff1_.log (deflated 58%) 2024-04-03T18:34:20.5408697Z adding: test/test-reports/distributed.test_distributed_spawn_3.7_3ef940ee4f2485e3_.log (deflated 93%) 2024-04-03T18:34:20.5410585Z adding: test/test-reports/distributed.fsdp.test_fsdp_overlap_1.1_4bda38cddd958f56_.log (deflated 80%) 2024-04-03T18:34:20.5423088Z adding: test/test-reports/distributed.test_distributed_spawn_3.7_861b207ce5857cae_.log (deflated 93%) 2024-04-03T18:34:20.5424718Z adding: test/test-reports/distributed.test_c10d_logger_1.1_d2b39a2ca845e49b_.log (deflated 60%) 2024-04-03T18:34:20.5426342Z adding: test/test-reports/distributed.test_distributed_spawn_6.7_3e03558aad63f984_.log (deflated 12%) 2024-04-03T18:34:20.5428023Z adding: test/test-reports/distributed.test_multi_threaded_pg_1.1_dfedede533155b3c_.log (deflated 85%) 2024-04-03T18:34:20.5429695Z adding: test/test-reports/distributed.test_distributed_spawn_6.7_b7df6e87edaa0a76_.log (deflated 12%) 2024-04-03T18:34:20.5431360Z adding: test/test-reports/distributed.fsdp.test_fsdp_apply_1.1_ddf51abf49085802_.log (deflated 81%) 2024-04-03T18:34:20.5440374Z adding: test/test-reports/distributed.test_distributed_spawn_6.7_31a190479c17e3c1_.log (deflated 94%) 2024-04-03T18:34:20.5544225Z adding: test/test-reports/distributed.fsdp.test_fsdp_core_2.2_dcb13a4abc4e1c6a_.log (deflated 97%) 2024-04-03T18:34:20.5555523Z adding: test/test-reports/distributed.test_distributed_spawn_6.7_f14d43229be38a32_.log (deflated 94%) 2024-04-03T18:34:20.5557172Z adding: test/test-reports/distributed.fsdp.test_utils_1.1_a3f6de6ffc6b2fda_.log (deflated 64%) 2024-04-03T18:34:20.5568111Z adding: test/test-reports/distributed.test_distributed_spawn_6.7_d45eb690be470df2_.log (deflated 94%) 2024-04-03T18:34:20.5569830Z adding: test/test-reports/distributed.algorithms.test_join_1.1_0ed745940909f321_.log (deflated 72%) 2024-04-03T18:34:20.5580115Z adding: test/test-reports/distributed.test_distributed_spawn_6.7_e7ccfbd1e5754c1b_.log (deflated 94%) 2024-04-03T18:34:20.5584074Z adding: test/test-reports/distributed.test_c10d_common_1.1_f48a6340710d7fff_.log (deflated 94%) 2024-04-03T18:34:20.5595817Z adding: test/test-reports/distributed.test_distributed_spawn_6.7_44795233b6c6d9d6_.log (deflated 93%) 2024-04-03T18:34:20.5597684Z adding: test/test-reports/distributed.test_c10d_spawn_nccl_1.1_1968c63c62c31396_.log (deflated 92%) 2024-04-03T18:34:20.5610416Z adding: test/test-reports/distributed.test_distributed_spawn_6.7_d04a10a5641a7263_.log (deflated 93%) 2024-04-03T18:34:20.5612622Z adding: test/test-reports/distributed.tensor.parallel.test_ddp_2d_parallel_1.1_7e624b7748f90337_.log (deflated 55%) 2024-04-03T18:34:20.5614751Z adding: test/test-reports/distributed.checkpoint.test_fsdp_optim_state_1.1_c354951dd721c0c0_.log (deflated 80%) 2024-04-03T18:34:20.5616754Z adding: test/test-reports/distributed.test_nccl_1.1_06547e0ea6d09525_.log (deflated 74%) 2024-04-03T18:34:20.5618411Z adding: test/test-reports/distributed._tensor.test_api_1.1_7344f45c4148547c_.log (deflated 73%) 2024-04-03T18:34:20.5645938Z adding: test/test-reports/distributed.test_c10d_gloo_1.1_7daf57462ec3fddb_.log (deflated 96%) 2024-04-03T18:34:20.5647574Z adding: test/test-reports/distributed._tensor.test_random_ops_1.1_3c0d1182670c4ac0_.log (deflated 69%) 2024-04-03T18:34:20.5649289Z adding: test/test-reports/distributed.checkpoint.test_fsspec_1.1_d310a05cd22651d1_.log (deflated 70%) 2024-04-03T18:34:20.5651107Z adding: test/test-reports/distributed.checkpoint.test_dtensor_resharding_1.1_79b4d4b60707006a_.log (deflated 82%) 2024-04-03T18:34:20.5652953Z adding: test/test-reports/distributed._tensor.test_experimental_ops_1.1_ad75fbdc7e9d1cf5_.log (deflated 79%) 2024-04-03T18:34:20.5654831Z adding: test/test-reports/distributed._tensor.experimental.test_tp_transform_1.1_83395c53a8e854fa_.log (deflated 78%) 2024-04-03T18:34:20.5656685Z adding: test/test-reports/distributed.test_compute_comm_reordering_1.1_378fedd2a529c1a4_.log (deflated 73%) 2024-04-03T18:34:20.5658577Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_autograd_1.1_dafd9e0dbb9939e9_.log (deflated 66%) 2024-04-03T18:34:20.5660433Z adding: test/test-reports/distributed._tensor.test_redistribute_1.1_bd672dde1b5b1d47_.log (deflated 77%) 2024-04-03T18:34:20.5662294Z adding: test/test-reports/distributed.tensor.parallel.test_tp_random_state_1.1_41e034a5d8b7a1cb_.log (deflated 55%) 2024-04-03T18:34:20.5664269Z adding: test/test-reports/distributed.checkpoint.e2e.test_e2e_save_and_load_1.1_2df3ffe0c8fbab1b_.log (deflated 80%) 2024-04-03T18:34:20.5666261Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_state_dict_1.1_53914cc5d59e491a_.log (deflated 62%) 2024-04-03T18:34:20.5668317Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_9e3bec3530bcd5c1_.log (deflated 73%) 2024-04-03T18:34:20.5670289Z adding: test/test-reports/distributed.checkpoint.test_state_dict_utils_1.1_52b84cca89d6a5f4_.log (deflated 68%) 2024-04-03T18:34:20.5672205Z adding: test/test-reports/distributed.tensor.parallel.test_fsdp_2d_parallel_1.1_453fccc239e69a0a_.log (deflated 78%) 2024-04-03T18:34:20.5674644Z adding: test/test-reports/distributed.tensor.parallel.test_parallelize_api_1.1_298d3fc44e7f6a08_.log (deflated 73%) 2024-04-03T18:34:20.5676766Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_util_1.1_637f1b58e0ddbc2a_.log (deflated 65%) 2024-04-03T18:34:20.5678716Z adding: test/test-reports/distributed.fsdp.test_distributed_checkpoint_1.1_f1243b6507af6cfd_.log (deflated 65%) 2024-04-03T18:34:20.5680800Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_comm_1.1_c6eb30ab33bb190c_.log (deflated 66%) 2024-04-03T18:34:20.5682915Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_model_checkpoint_1.1_36c76c429cf71c74_.log (deflated 86%) 2024-04-03T18:34:20.5685008Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_init_1.1_486bb13dbca9b5a9_.log (deflated 88%) 2024-04-03T18:34:20.5686993Z adding: test/test-reports/distributed._shard.sharding_spec.test_sharding_spec_1.1_0e1a350bf3926547_.log (deflated 79%) 2024-04-03T18:34:20.5688890Z adding: test/test-reports/distributed.fsdp.test_fsdp_dtensor_state_dict_1.1_17a11afe6953f7e3_.log (deflated 86%) 2024-04-03T18:34:20.5690877Z adding: test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_9d0d3afa569f6f6b_.log (deflated 75%) 2024-04-03T18:34:20.5692626Z adding: test/test-reports/distributed.nn.jit.test_instantiator_1.1_cdedd5e02cc57d95_.log (deflated 63%) 2024-04-03T18:34:20.5694495Z adding: test/test-reports/distributed.checkpoint.test_file_system_checkpoint_1.1_a0a683dd5d4824bb_.log (deflated 88%) 2024-04-03T18:34:20.5696324Z adding: test/test-reports/distributed.checkpoint.test_planner_1.1_5147e8d00762951a_.log (deflated 65%) 2024-04-03T18:34:20.5698111Z adding: test/test-reports/distributed.pipeline.sync.test_pipeline_1.1_f9d25d3cdb22d1fd_.log (deflated 53%) 2024-04-03T18:34:20.5699980Z adding: test/test-reports/distributed._composable.test_replicate_with_compiler_1.1_91f365dfc44612bf_.log (deflated 85%) 2024-04-03T18:34:20.5701814Z adding: test/test-reports/distributed.fsdp.test_fsdp_state_dict_1.2_73471e1824b34cc5_.log (deflated 94%) 2024-04-03T18:34:20.5703620Z adding: test/test-reports/distributed.optim.test_apply_optimizer_in_backward_1.1_c897b3667a078c97_.log (stored 0%) 2024-04-03T18:34:20.5705425Z adding: test/test-reports/distributed.pipeline.sync.test_phony_1.1_ed45acac19d53683_.log (deflated 60%) 2024-04-03T18:34:20.5707184Z adding: test/test-reports/distributed.test_c10d_object_collectives_1.1_5a8468029324d313_.log (deflated 74%) 2024-04-03T18:34:20.5708927Z adding: test/test-reports/distributed.pipeline.sync.test_bugs_1.1_65d89ef60a1c1fd8_.log (deflated 59%) 2024-04-03T18:34:20.5710729Z adding: test/test-reports/distributed.fsdp.test_hsdp_dtensor_state_dict_1.1_f13a84b198a09c24_.log (deflated 79%) 2024-04-03T18:34:20.5712579Z adding: test/test-reports/distributed.pipeline.sync.skip.test_gpipe_1.1_36dbbf1b5ff797ce_.log (deflated 73%) 2024-04-03T18:34:20.5715002Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_embedding_1.1_165e7556d6db7d41_.log (deflated 61%) 2024-04-03T18:34:20.5717063Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_embedding_bag_1.1_e43996b427076e59_.log (deflated 62%) 2024-04-03T18:34:20.5719031Z adding: test/test-reports/distributed.rpc.cuda.test_tensorpipe_agent_2.2_88d51ac41e511ebc_.log (deflated 98%) 2024-04-03T18:34:20.5721113Z adding: test/test-reports/distributed.pipeline.sync.test_stream_1.1_d4c9784d54d159b7_.log (deflated 79%) 2024-04-03T18:34:20.5722898Z adding: test/test-reports/distributed.elastic.events.lib_test_1.1_85d56d643bc3e184_.log (deflated 72%) 2024-04-03T18:34:20.5724645Z adding: test/test-reports/distributed.elastic.metrics.api_test_1.1_4163e9da5e8b3af5_.log (deflated 62%) 2024-04-03T18:34:20.5726483Z adding: test/test-reports/distributed.elastic.multiprocessing.api_test_1.1_1376dbe0a59bde8b_.log (deflated 89%) 2024-04-03T18:34:20.5728397Z adding: test/test-reports/distributed.elastic.timer.local_timer_example_1.1_bc5a8ed72e1330c9_.log (deflated 68%) 2024-04-03T18:34:20.5730275Z adding: test/test-reports/distributed.elastic.timer.local_timer_test_1.1_8fc4b594f8570873_.log (deflated 77%) 2024-04-03T18:34:20.5732147Z adding: test/test-reports/distributed.elastic.utils.distributed_test_1.1_09fef3d43894e4ff_.log (deflated 72%) 2024-04-03T18:34:20.5734205Z adding: test/test-reports/distributed.elastic.utils.logging_test_1.1_7a1973c6f43f84c6_.log (deflated 58%) 2024-04-03T18:34:20.5736185Z adding: test/test-reports/distributed.elastic.utils.util_test_1.1_f29dec163b3eb601_.log (deflated 69%) 2024-04-03T18:34:20.5811434Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-04-03T18:34:20.5812043Z with: 2024-04-03T18:34:20.5812428Z s3-bucket: gha-artifacts 2024-04-03T18:34:20.5813029Z s3-prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T18:34:20.5813665Z retention-days: 14 2024-04-03T18:34:20.5814130Z if-no-files-found: warn 2024-04-03T18:34:20.5814629Z path: test-jsons-*.zip 2024-04-03T18:34:20.5815088Z name: artifact 2024-04-03T18:34:20.5815565Z region: us-east-1 2024-04-03T18:34:20.5815992Z env: 2024-04-03T18:34:20.5816349Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:34:20.5817181Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:34:20.5818225Z DOCKER_CONTAINER_ID: 7033b537529c546187ef7c776250a79cad9cc38d4987347f20cdbf86b60e2c90 2024-04-03T18:34:20.5819118Z ##[endgroup] 2024-04-03T18:34:21.0923163Z NOTE: s3-prefix specified, ignoring name parameter 2024-04-03T18:34:21.0924232Z With the provided path, there will be 1 file uploaded 2024-04-03T18:34:21.0925108Z Uploading to s3 prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T18:34:21.2525243Z Starting upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_23402706124.zip 2024-04-03T18:34:21.4753427Z Finished upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_23402706124.zip 2024-04-03T18:34:21.4947442Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-04-03T18:34:21.4948052Z with: 2024-04-03T18:34:21.4948437Z s3-bucket: gha-artifacts 2024-04-03T18:34:21.4949035Z s3-prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T18:34:21.4949706Z retention-days: 14 2024-04-03T18:34:21.4950178Z if-no-files-found: error 2024-04-03T18:34:21.4950687Z path: test-reports-*.zip 2024-04-03T18:34:21.4951156Z name: artifact 2024-04-03T18:34:21.4951578Z region: us-east-1 2024-04-03T18:34:21.4951990Z env: 2024-04-03T18:34:21.4952347Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:34:21.4952997Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:34:21.4954050Z DOCKER_CONTAINER_ID: 7033b537529c546187ef7c776250a79cad9cc38d4987347f20cdbf86b60e2c90 2024-04-03T18:34:21.4955254Z ##[endgroup] 2024-04-03T18:34:21.9920273Z NOTE: s3-prefix specified, ignoring name parameter 2024-04-03T18:34:21.9921322Z With the provided path, there will be 1 file uploaded 2024-04-03T18:34:21.9922195Z Uploading to s3 prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T18:34:21.9976589Z Starting upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_23402706124.zip 2024-04-03T18:34:22.5338822Z Finished upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_23402706124.zip 2024-04-03T18:34:22.5534990Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-04-03T18:34:22.5535606Z with: 2024-04-03T18:34:22.5536007Z s3-bucket: gha-artifacts 2024-04-03T18:34:22.5536612Z s3-prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T18:34:22.5537267Z retention-days: 14 2024-04-03T18:34:22.5537762Z if-no-files-found: ignore 2024-04-03T18:34:22.5538277Z path: logs-*.zip 2024-04-03T18:34:22.5538695Z name: artifact 2024-04-03T18:34:22.5539123Z region: us-east-1 2024-04-03T18:34:22.5539543Z env: 2024-04-03T18:34:22.5539912Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:34:22.5540563Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:34:22.5541620Z DOCKER_CONTAINER_ID: 7033b537529c546187ef7c776250a79cad9cc38d4987347f20cdbf86b60e2c90 2024-04-03T18:34:22.5542518Z ##[endgroup] 2024-04-03T18:34:23.0597564Z NOTE: s3-prefix specified, ignoring name parameter 2024-04-03T18:34:23.0598746Z With the provided path, there will be 1 file uploaded 2024-04-03T18:34:23.0599673Z Uploading to s3 prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T18:34:23.0652971Z Starting upload of logs-test-distributed-2-3-linux.8xlarge.nvidia.gpu_23402706124.zip 2024-04-03T18:34:23.3064282Z Finished upload of logs-test-distributed-2-3-linux.8xlarge.nvidia.gpu_23402706124.zip 2024-04-03T18:34:23.3265435Z ##[group]Run # shellcheck disable=SC2156 2024-04-03T18:34:23.3266144Z # shellcheck disable=SC2156 2024-04-03T18:34:23.3267302Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-04-03T18:34:23.3277984Z shell: /usr/bin/bash -e {0} 2024-04-03T18:34:23.3278468Z env: 2024-04-03T18:34:23.3278851Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:34:23.3279493Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:34:23.3280538Z DOCKER_CONTAINER_ID: 7033b537529c546187ef7c776250a79cad9cc38d4987347f20cdbf86b60e2c90 2024-04-03T18:34:23.3281468Z ##[endgroup] 2024-04-03T18:34:23.6448122Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-04-03T18:34:23.6448960Z with: 2024-04-03T18:34:23.6449420Z env: 2024-04-03T18:34:23.6449843Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:34:23.6450500Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:34:23.6451576Z DOCKER_CONTAINER_ID: 7033b537529c546187ef7c776250a79cad9cc38d4987347f20cdbf86b60e2c90 2024-04-03T18:34:23.6452508Z ##[endgroup] 2024-04-03T18:34:23.6516542Z ##[group]Run set -eou pipefail 2024-04-03T18:34:23.6517096Z set -eou pipefail 2024-04-03T18:34:23.6517599Z  2024-04-03T18:34:23.6518319Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-04-03T18:34:23.6519223Z for _ in $(seq 1440); do 2024-04-03T18:34:23.6519876Z  # Break if no ssh session exists anymore 2024-04-03T18:34:23.6520563Z  if [ "$(who)" = "" ]; then 2024-04-03T18:34:23.6521104Z  break 2024-04-03T18:34:23.6521580Z  fi 2024-04-03T18:34:23.6521988Z  echo "." 2024-04-03T18:34:23.6522416Z  sleep 5 2024-04-03T18:34:23.6522839Z done 2024-04-03T18:34:23.6532145Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:34:23.6532841Z env: 2024-04-03T18:34:23.6533195Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:34:23.6533823Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:34:23.6534850Z DOCKER_CONTAINER_ID: 7033b537529c546187ef7c776250a79cad9cc38d4987347f20cdbf86b60e2c90 2024-04-03T18:34:23.6535731Z ##[endgroup] 2024-04-03T18:34:23.6564439Z Holding runner for 2 hours until all ssh sessions have logged out 2024-04-03T18:34:23.6638970Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T18:34:23.6640072Z # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T18:34:23.6640926Z # shellcheck disable=SC2046 2024-04-03T18:34:23.6641592Z docker stop $(docker ps -q) || true 2024-04-03T18:34:23.6642268Z # Prune all of the docker images 2024-04-03T18:34:23.6642906Z docker system prune -af 2024-04-03T18:34:23.6651901Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:34:23.6652614Z env: 2024-04-03T18:34:23.6652985Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:34:23.6653631Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:34:23.6654686Z DOCKER_CONTAINER_ID: 7033b537529c546187ef7c776250a79cad9cc38d4987347f20cdbf86b60e2c90 2024-04-03T18:34:23.6655588Z ##[endgroup] 2024-04-03T18:34:23.9989366Z 7033b537529c 2024-04-03T18:34:24.6310819Z Deleted Containers: 2024-04-03T18:34:24.6311614Z 7033b537529c546187ef7c776250a79cad9cc38d4987347f20cdbf86b60e2c90 2024-04-03T18:34:24.6312242Z 2024-04-03T18:34:32.1699406Z Deleted Images: 2024-04-03T18:34:32.1701269Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T18:34:32.1705278Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9@sha256:68e0ca2e38253fb3bd10c84d380560df0602da1b16bccdfd7d4dd7ac22529759 2024-04-03T18:34:32.1709355Z deleted: sha256:f0aa5113e491b8f8ef3b51fc066bce798422ecc2bdeb4be54a8a4efa6914052e 2024-04-03T18:34:32.1711663Z deleted: sha256:c454db9d5e4e4efa0edf7c73a701618617c23c0f03a17d5e9c5f49ccbbfa0e73 2024-04-03T18:34:32.1713846Z deleted: sha256:b2dd02706fb0cdd8a4420189b04471a7739f76c16bb6c9e36747a62daaf28ad9 2024-04-03T18:34:32.1716483Z deleted: sha256:3e52b95dec2563c40922e1bfd58e5ec9813906f6958f9e481f3dad044b00c734 2024-04-03T18:34:32.1718689Z deleted: sha256:eeeb44278fe18f54afa2e7db8fd0210ec20683601770a2353cb53c27f7a1b5bc 2024-04-03T18:34:32.1720590Z deleted: sha256:3de3ee3c1c2cccb5001455ea6adcb75600111cc3fd52f8f80c11c495d6ee4e57 2024-04-03T18:34:32.1722841Z deleted: sha256:d6bf411cddfbc13bf76f9472a8b37a35ba76ffdccd5a2e21b14829368e18d3f7 2024-04-03T18:34:32.1725631Z deleted: sha256:8613c4bc44191f5e1cc622290ed8b516510d1ab876c25ccd377c36f51c2773f5 2024-04-03T18:34:32.1728258Z deleted: sha256:c3b2fbe550b9b298db4f89162757d368522e9cfede30c563e85dbf4440beb582 2024-04-03T18:34:32.1730657Z deleted: sha256:7a2ba3e1861525f15dcbdea7d5f2b39e6d14cc6a1a4d3a4669496c7a5c3a2d75 2024-04-03T18:34:32.1733027Z deleted: sha256:09e9c5c1b4f18382df5cddefe5c9b0dc27e5679be209c1e8622765d97df05bd2 2024-04-03T18:34:32.1735436Z deleted: sha256:3afff7cf2c94a194113068301442afd3f5a1b5c921841f28093ed5947ff17bf9 2024-04-03T18:34:32.1737567Z deleted: sha256:01421ca8d6d1e66f01d50067086730b4100f9a1966a2266332540cfd2fa67288 2024-04-03T18:34:32.1738762Z deleted: sha256:9d8429c72401fd7217175b98df195cdbe1dd4ff878348104c6a0be2c8fe87f07 2024-04-03T18:34:32.1740028Z deleted: sha256:b0ab639903fffd49ad6bda7fdf2fcd209dcbfce6f84ef28b9dac3dae321bb025 2024-04-03T18:34:32.1741303Z deleted: sha256:e1e2ba6eb5c7694469bb6fbd6820df749e354c265ab0b261960dec6a7a3a28be 2024-04-03T18:34:32.1742548Z deleted: sha256:9722a85cb366e43d512471e05004f0bba6c5a2c2dd592ba038ca03d96e348d92 2024-04-03T18:34:32.1743752Z deleted: sha256:59105bb9683ece65c0c62952242c8f582486ef48e62af263bdadbc6f61608fad 2024-04-03T18:34:32.1744976Z deleted: sha256:053162ab6c481eb96b5f661978e688dcc6bee984e7ec68e2d6ce6296bc891738 2024-04-03T18:34:32.1746200Z deleted: sha256:aa95ee7dea81036a66f9398d7a92020ea53c0de98fea9690a848c4887242c204 2024-04-03T18:34:32.1747403Z deleted: sha256:b00f90cd5628b07e6c16d09e2bdd195897085b1fd7c583800717b13fdb2bddba 2024-04-03T18:34:32.1748992Z deleted: sha256:d8764b2eab188f5b3c2c6573294c0e7d25cabe9e2206b51a1089bee5fdc9a099 2024-04-03T18:34:32.1750530Z deleted: sha256:2f02a119b01d7d4f2da3ae04e62eb82c6862630ad151910586b7d5fd54e3dd20 2024-04-03T18:34:32.1751772Z deleted: sha256:bb8ac4a8069a057d8ce651918a3f3a37f69cf070660ae23904dc9cd9675da101 2024-04-03T18:34:32.1753003Z deleted: sha256:20129be101ad623cea8a076f51abe873d22e6a1b9182dcffa89aefda79b6a7ec 2024-04-03T18:34:32.1754565Z deleted: sha256:ddc263e2d5206bdf8e3face17284fd07e61bedefc7820f3d15e43754688f56f6 2024-04-03T18:34:32.1755837Z deleted: sha256:331c5b9483ad593f5aa15565c0c1f483b7827a1ef011a7c0d9d40658f7e43a11 2024-04-03T18:34:32.1757046Z deleted: sha256:e2746ff342dbe1a36a1388a0bb33bc12e69b1627af1a7a13ffa6231db1645bd3 2024-04-03T18:34:32.1758293Z deleted: sha256:d2af2292b7bf3bd83e33811bdfceb98e128b901af3f13071e17ca8bb9a852a6f 2024-04-03T18:34:32.1759538Z deleted: sha256:a2ec7109ed947976900ebfb586b3d4e2552f6af6ce04fa04adda236eee7550ef 2024-04-03T18:34:32.1760758Z deleted: sha256:6b40bbee3bc6087e5c14ba5d43abde7796d62653a46181c6f87aa635f9ade9ac 2024-04-03T18:34:32.1761996Z deleted: sha256:377d27777d5b7ccd2c4a239e28d1d2ced864e35163e3ffc9131aee6b50ebfd44 2024-04-03T18:34:32.1763221Z deleted: sha256:139d166e407995d29a3d1c469c6af0eccb96895ec95721e668d7d1a96f1601e6 2024-04-03T18:34:32.1764397Z deleted: sha256:9e29d5432f1d387203719074a0d079463fc945d166c6fe9168342acd94d25565 2024-04-03T18:34:32.1765605Z deleted: sha256:7c66ee17d9810d8b654318ddadf226dc224d7ca4adbb64287aa4e1d378369d40 2024-04-03T18:34:32.1766867Z deleted: sha256:ca79b52f3eae23be700ee8aedcb485040d1918aacb1dcac027ff07a5e30d49e5 2024-04-03T18:34:32.1768126Z deleted: sha256:7fd608199fde74f258f5ff0cdb5708bbbb269c40c238f0941357570ceb3c0f97 2024-04-03T18:34:32.1769336Z deleted: sha256:953c2071c9b50052113d1dcbf64eaf51cd11e8dfa56e581b412bc075a784fb30 2024-04-03T18:34:32.1770541Z deleted: sha256:e397941ce9440cfbb931d0384718d8689325a0bca0c603e02412c5d28dc91891 2024-04-03T18:34:32.1771742Z deleted: sha256:a2f0692c36460bc69d0253b25333b5b17eb59f9f0cbc5325329b3823ac81025d 2024-04-03T18:34:32.1772916Z deleted: sha256:53a2bb00ca64b3972528438023e17333f36c3b85db48605df0f1111c08b6a91c 2024-04-03T18:34:32.1774098Z deleted: sha256:350185224c8f26322214dfdf6f1f72776da8088d3168d6f83723c98445e43f10 2024-04-03T18:34:32.1775288Z deleted: sha256:76385dca1ebd668f08aad474882b95d21679544054579c76ad79776a75b83e22 2024-04-03T18:34:32.1776498Z deleted: sha256:b11bcb556f21ebd78f10a9125d7f534ea59f86e81a006d815962a83602b0e662 2024-04-03T18:34:32.1777907Z deleted: sha256:7c6e21fb86aabd7cccb7531f8e7f4fa39607bf4136679b174638238ba2036339 2024-04-03T18:34:32.1779304Z deleted: sha256:49b69334db1af3decf2111f3474f9a7a76aeadafd34b647465bf35baf4496632 2024-04-03T18:34:32.1780535Z deleted: sha256:d6830305dc4ef0d8627ddc0fe0f1e49f5304333b7151a30b2aed904945e6a68f 2024-04-03T18:34:32.1781739Z deleted: sha256:21e3145f7deb8e4c8e4c359f495779f0b2a6ffd8a54ad879ec0bb6bcdbed9001 2024-04-03T18:34:32.1782961Z deleted: sha256:1c978ef9d890a352bfc23641fa000d332f042527032227c9715f297ce15a9d3a 2024-04-03T18:34:32.1784154Z deleted: sha256:5e2571ab32a3976828a3cf45d33bf651d60949687f8a81f3970c51c90a5a4a1d 2024-04-03T18:34:32.1785321Z deleted: sha256:547e1709b8e6365cc024934a2d46fe0b4029f4195b372a5ca63522747b09f9f5 2024-04-03T18:34:32.1786513Z deleted: sha256:d6f91263e804bd96920276a41b0538e01c33bf03b3ab32c4175588b5a9b5c217 2024-04-03T18:34:32.1787702Z deleted: sha256:ac317a820242b9e6c255f8065538f07519c66bbb5b40f313434cccb99ed34e56 2024-04-03T18:34:32.1788929Z deleted: sha256:6787bc9a8b82b4efce75e9e6d725ce95ff1e03a4f53768f96d420c5c0d5904d4 2024-04-03T18:34:32.1790146Z deleted: sha256:6a19981c4f76113029dbbcc0d87cdf2d324ec8d370b70ffb9d64623c804cfc3e 2024-04-03T18:34:32.1791370Z deleted: sha256:90ee54386e1cfb202860b4f78bfa7bb0db9722c0b8cc055c9f578130f09363d3 2024-04-03T18:34:32.1792584Z deleted: sha256:e039079492e83156c78b02baa6b4a38218c1c1529f7aec4dece116017320584d 2024-04-03T18:34:32.1793776Z deleted: sha256:488705e52ca1ed434cf0c3ee439eca1262115a69eb7492de82b7610234e2f6c5 2024-04-03T18:34:32.1795277Z deleted: sha256:5c567c50a43b1644e3211ef0e28dc73aa22a25d12bb1346308746375c9efbcfd 2024-04-03T18:34:32.1796477Z deleted: sha256:7c59d78b7f784e56d846633c9da755128c52d841ac70c649169b80efc9861e94 2024-04-03T18:34:32.1797643Z deleted: sha256:0702e7a9df3e7459b15b0f3d87c1029524539618e4676908b5934b923ed199ea 2024-04-03T18:34:32.1798829Z deleted: sha256:57a6f7d803492e9f768503b021ed8248e4d78d575185dd1938e57d18245667ad 2024-04-03T18:34:32.1800020Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-04-03T18:34:32.1801271Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-04-03T18:34:32.1802480Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-04-03T18:34:32.1803686Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-04-03T18:34:32.1804920Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-04-03T18:34:32.1806145Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-04-03T18:34:32.1807382Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-04-03T18:34:32.1808590Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-04-03T18:34:32.1809800Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-04-03T18:34:32.1811024Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-04-03T18:34:32.1812234Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-04-03T18:34:32.1812980Z 2024-04-03T18:34:32.1813208Z Total reclaimed space: 25.63GB 2024-04-03T18:34:32.1912146Z Post job cleanup. 2024-04-03T18:34:32.1990095Z Post job cleanup. 2024-04-03T18:34:32.3320681Z [command]/usr/bin/git version 2024-04-03T18:34:32.3381375Z git version 2.40.1 2024-04-03T18:34:32.3452591Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/18a36a62-35bc-4db9-a10b-9a1e8c330dfa' before making global git config changes 2024-04-03T18:34:32.3454439Z Adding repository directory to the temporary git global config as a safe directory 2024-04-03T18:34:32.3461312Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T18:34:32.3508896Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-04-03T18:34:32.3550101Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-04-03T18:34:32.3909559Z Entering 'android/libs/fbjni' 2024-04-03T18:34:32.3968721Z Entering 'third_party/FP16' 2024-04-03T18:34:32.4026484Z Entering 'third_party/FXdiv' 2024-04-03T18:34:32.4085916Z Entering 'third_party/NNPACK' 2024-04-03T18:34:32.4144398Z Entering 'third_party/QNNPACK' 2024-04-03T18:34:32.4206020Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T18:34:32.4266666Z Entering 'third_party/XNNPACK' 2024-04-03T18:34:32.4348846Z Entering 'third_party/benchmark' 2024-04-03T18:34:32.4407045Z Entering 'third_party/cpuinfo' 2024-04-03T18:34:32.4467336Z Entering 'third_party/cub' 2024-04-03T18:34:32.4526158Z Entering 'third_party/cudnn_frontend' 2024-04-03T18:34:32.4585095Z Entering 'third_party/cutlass' 2024-04-03T18:34:32.4653097Z Entering 'third_party/eigen' 2024-04-03T18:34:32.4715260Z Entering 'third_party/fbgemm' 2024-04-03T18:34:32.4772210Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T18:34:32.4829790Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T18:34:32.4887221Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T18:34:32.4951945Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T18:34:32.5008259Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T18:34:32.5066023Z Entering 'third_party/flatbuffers' 2024-04-03T18:34:32.5128763Z Entering 'third_party/fmt' 2024-04-03T18:34:32.5187511Z Entering 'third_party/foxi' 2024-04-03T18:34:32.5245291Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T18:34:32.5305803Z Entering 'third_party/gloo' 2024-04-03T18:34:32.5365952Z Entering 'third_party/googletest' 2024-04-03T18:34:32.5424456Z Entering 'third_party/ideep' 2024-04-03T18:34:32.5482135Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T18:34:32.5547664Z Entering 'third_party/ios-cmake' 2024-04-03T18:34:32.5605201Z Entering 'third_party/ittapi' 2024-04-03T18:34:32.5663668Z Entering 'third_party/kineto' 2024-04-03T18:34:32.5722365Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T18:34:32.5781747Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T18:34:32.5842946Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T18:34:32.5899813Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T18:34:32.5958446Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T18:34:32.6014497Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T18:34:32.6074619Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T18:34:32.6131590Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T18:34:32.6188628Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T18:34:32.6249608Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T18:34:32.6308209Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T18:34:32.6366034Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T18:34:32.6424151Z Entering 'third_party/mimalloc' 2024-04-03T18:34:32.6482528Z Entering 'third_party/nccl/nccl' 2024-04-03T18:34:32.6540402Z Entering 'third_party/neon2sse' 2024-04-03T18:34:32.6599352Z Entering 'third_party/nlohmann' 2024-04-03T18:34:32.6658399Z Entering 'third_party/onnx' 2024-04-03T18:34:32.6737784Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T18:34:32.6797409Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T18:34:32.6856288Z Entering 'third_party/onnx-tensorrt' 2024-04-03T18:34:32.6913795Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T18:34:32.6976286Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T18:34:32.7035526Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T18:34:32.7092351Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T18:34:32.7156439Z Entering 'third_party/pocketfft' 2024-04-03T18:34:32.7214692Z Entering 'third_party/protobuf' 2024-04-03T18:34:32.7278092Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T18:34:32.7334607Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T18:34:32.7394551Z Entering 'third_party/psimd' 2024-04-03T18:34:32.7452902Z Entering 'third_party/pthreadpool' 2024-04-03T18:34:32.7510606Z Entering 'third_party/pybind11' 2024-04-03T18:34:32.7569695Z Entering 'third_party/python-peachpy' 2024-04-03T18:34:32.7627857Z Entering 'third_party/sleef' 2024-04-03T18:34:32.7687236Z Entering 'third_party/tbb' 2024-04-03T18:34:32.7748356Z Entering 'third_party/tensorpipe' 2024-04-03T18:34:32.7807041Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T18:34:32.7864952Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T18:34:32.7921545Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T18:34:32.7979657Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T18:34:32.8034968Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T18:34:32.8095205Z Entering 'third_party/zstd' 2024-04-03T18:34:32.8175289Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-04-03T18:34:32.8203960Z http.https://github.com/.extraheader 2024-04-03T18:34:32.8217354Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-04-03T18:34:32.8258865Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-04-03T18:34:32.8612003Z Entering 'android/libs/fbjni' 2024-04-03T18:34:32.8648935Z http.https://github.com/.extraheader 2024-04-03T18:34:32.8687634Z Entering 'third_party/FP16' 2024-04-03T18:34:32.8725125Z http.https://github.com/.extraheader 2024-04-03T18:34:32.8763774Z Entering 'third_party/FXdiv' 2024-04-03T18:34:32.8800767Z http.https://github.com/.extraheader 2024-04-03T18:34:32.8839059Z Entering 'third_party/NNPACK' 2024-04-03T18:34:32.8876250Z http.https://github.com/.extraheader 2024-04-03T18:34:32.8914429Z Entering 'third_party/QNNPACK' 2024-04-03T18:34:32.8951518Z http.https://github.com/.extraheader 2024-04-03T18:34:32.8989744Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T18:34:32.9027526Z http.https://github.com/.extraheader 2024-04-03T18:34:32.9066687Z Entering 'third_party/XNNPACK' 2024-04-03T18:34:32.9104009Z http.https://github.com/.extraheader 2024-04-03T18:34:32.9163382Z Entering 'third_party/benchmark' 2024-04-03T18:34:32.9201567Z http.https://github.com/.extraheader 2024-04-03T18:34:32.9240786Z Entering 'third_party/cpuinfo' 2024-04-03T18:34:32.9277965Z http.https://github.com/.extraheader 2024-04-03T18:34:32.9316846Z Entering 'third_party/cub' 2024-04-03T18:34:32.9353811Z http.https://github.com/.extraheader 2024-04-03T18:34:32.9393542Z Entering 'third_party/cudnn_frontend' 2024-04-03T18:34:32.9431728Z http.https://github.com/.extraheader 2024-04-03T18:34:32.9469971Z Entering 'third_party/cutlass' 2024-04-03T18:34:32.9506835Z http.https://github.com/.extraheader 2024-04-03T18:34:32.9555700Z Entering 'third_party/eigen' 2024-04-03T18:34:32.9591452Z http.https://github.com/.extraheader 2024-04-03T18:34:32.9633222Z Entering 'third_party/fbgemm' 2024-04-03T18:34:32.9669953Z http.https://github.com/.extraheader 2024-04-03T18:34:32.9709547Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T18:34:32.9745874Z http.https://github.com/.extraheader 2024-04-03T18:34:32.9784257Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T18:34:32.9820675Z http.https://github.com/.extraheader 2024-04-03T18:34:32.9859604Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T18:34:32.9895814Z http.https://github.com/.extraheader 2024-04-03T18:34:32.9941646Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T18:34:32.9978440Z http.https://github.com/.extraheader 2024-04-03T18:34:33.0017842Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T18:34:33.0053768Z http.https://github.com/.extraheader 2024-04-03T18:34:33.0093358Z Entering 'third_party/flatbuffers' 2024-04-03T18:34:33.0131433Z http.https://github.com/.extraheader 2024-04-03T18:34:33.0172623Z Entering 'third_party/fmt' 2024-04-03T18:34:33.0209627Z http.https://github.com/.extraheader 2024-04-03T18:34:33.0249252Z Entering 'third_party/foxi' 2024-04-03T18:34:33.0286178Z http.https://github.com/.extraheader 2024-04-03T18:34:33.0324018Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T18:34:33.0361181Z http.https://github.com/.extraheader 2024-04-03T18:34:33.0400622Z Entering 'third_party/gloo' 2024-04-03T18:34:33.0437382Z http.https://github.com/.extraheader 2024-04-03T18:34:33.0475135Z Entering 'third_party/googletest' 2024-04-03T18:34:33.0512154Z http.https://github.com/.extraheader 2024-04-03T18:34:33.0551767Z Entering 'third_party/ideep' 2024-04-03T18:34:33.0589200Z http.https://github.com/.extraheader 2024-04-03T18:34:33.0626932Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T18:34:33.0664288Z http.https://github.com/.extraheader 2024-04-03T18:34:33.0711791Z Entering 'third_party/ios-cmake' 2024-04-03T18:34:33.0749857Z http.https://github.com/.extraheader 2024-04-03T18:34:33.0788297Z Entering 'third_party/ittapi' 2024-04-03T18:34:33.0825788Z http.https://github.com/.extraheader 2024-04-03T18:34:33.0865087Z Entering 'third_party/kineto' 2024-04-03T18:34:33.0902804Z http.https://github.com/.extraheader 2024-04-03T18:34:33.0943768Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T18:34:33.0980691Z http.https://github.com/.extraheader 2024-04-03T18:34:33.1020944Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T18:34:33.1057624Z http.https://github.com/.extraheader 2024-04-03T18:34:33.1098908Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T18:34:33.1138014Z http.https://github.com/.extraheader 2024-04-03T18:34:33.1177772Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T18:34:33.1214436Z http.https://github.com/.extraheader 2024-04-03T18:34:33.1253403Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T18:34:33.1289403Z http.https://github.com/.extraheader 2024-04-03T18:34:33.1327057Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T18:34:33.1364432Z http.https://github.com/.extraheader 2024-04-03T18:34:33.1406561Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T18:34:33.1443584Z http.https://github.com/.extraheader 2024-04-03T18:34:33.1483321Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T18:34:33.1520597Z http.https://github.com/.extraheader 2024-04-03T18:34:33.1561477Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T18:34:33.1598479Z http.https://github.com/.extraheader 2024-04-03T18:34:33.1639737Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T18:34:33.1676303Z http.https://github.com/.extraheader 2024-04-03T18:34:33.1718429Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T18:34:33.1757658Z http.https://github.com/.extraheader 2024-04-03T18:34:33.1798369Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T18:34:33.1836053Z http.https://github.com/.extraheader 2024-04-03T18:34:33.1878288Z Entering 'third_party/mimalloc' 2024-04-03T18:34:33.1917170Z http.https://github.com/.extraheader 2024-04-03T18:34:33.1957700Z Entering 'third_party/nccl/nccl' 2024-04-03T18:34:33.1995575Z http.https://github.com/.extraheader 2024-04-03T18:34:33.2035604Z Entering 'third_party/neon2sse' 2024-04-03T18:34:33.2073705Z http.https://github.com/.extraheader 2024-04-03T18:34:33.2113060Z Entering 'third_party/nlohmann' 2024-04-03T18:34:33.2152050Z http.https://github.com/.extraheader 2024-04-03T18:34:33.2193716Z Entering 'third_party/onnx' 2024-04-03T18:34:33.2231915Z http.https://github.com/.extraheader 2024-04-03T18:34:33.2293116Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T18:34:33.2330753Z http.https://github.com/.extraheader 2024-04-03T18:34:33.2372545Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T18:34:33.2410418Z http.https://github.com/.extraheader 2024-04-03T18:34:33.2452398Z Entering 'third_party/onnx-tensorrt' 2024-04-03T18:34:33.2490418Z http.https://github.com/.extraheader 2024-04-03T18:34:33.2528225Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T18:34:33.2566243Z http.https://github.com/.extraheader 2024-04-03T18:34:33.2611772Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T18:34:33.2649072Z http.https://github.com/.extraheader 2024-04-03T18:34:33.2688221Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T18:34:33.2725415Z http.https://github.com/.extraheader 2024-04-03T18:34:33.2764838Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T18:34:33.2801349Z http.https://github.com/.extraheader 2024-04-03T18:34:33.2846382Z Entering 'third_party/pocketfft' 2024-04-03T18:34:33.2884659Z http.https://github.com/.extraheader 2024-04-03T18:34:33.2924103Z Entering 'third_party/protobuf' 2024-04-03T18:34:33.2962573Z http.https://github.com/.extraheader 2024-04-03T18:34:33.3005416Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T18:34:33.3042582Z http.https://github.com/.extraheader 2024-04-03T18:34:33.3081291Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T18:34:33.3118387Z http.https://github.com/.extraheader 2024-04-03T18:34:33.3158573Z Entering 'third_party/psimd' 2024-04-03T18:34:33.3195376Z http.https://github.com/.extraheader 2024-04-03T18:34:33.3233647Z Entering 'third_party/pthreadpool' 2024-04-03T18:34:33.3271832Z http.https://github.com/.extraheader 2024-04-03T18:34:33.3310287Z Entering 'third_party/pybind11' 2024-04-03T18:34:33.3347620Z http.https://github.com/.extraheader 2024-04-03T18:34:33.3386444Z Entering 'third_party/python-peachpy' 2024-04-03T18:34:33.3423525Z http.https://github.com/.extraheader 2024-04-03T18:34:33.3462528Z Entering 'third_party/sleef' 2024-04-03T18:34:33.3500439Z http.https://github.com/.extraheader 2024-04-03T18:34:33.3539249Z Entering 'third_party/tbb' 2024-04-03T18:34:33.3576416Z http.https://github.com/.extraheader 2024-04-03T18:34:33.3617062Z Entering 'third_party/tensorpipe' 2024-04-03T18:34:33.3654067Z http.https://github.com/.extraheader 2024-04-03T18:34:33.3692648Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T18:34:33.3728537Z http.https://github.com/.extraheader 2024-04-03T18:34:33.3767627Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T18:34:33.3803862Z http.https://github.com/.extraheader 2024-04-03T18:34:33.3842951Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T18:34:33.3878829Z http.https://github.com/.extraheader 2024-04-03T18:34:33.3918837Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T18:34:33.3954628Z http.https://github.com/.extraheader 2024-04-03T18:34:33.3991724Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T18:34:33.4027831Z http.https://github.com/.extraheader 2024-04-03T18:34:33.4069769Z Entering 'third_party/zstd' 2024-04-03T18:34:33.4106994Z http.https://github.com/.extraheader 2024-04-03T18:34:33.4504947Z Cleaning up orphan processes