2024-04-03T16:13:29.5137199Z Current runner version: '2.315.0' 2024-04-03T16:13:29.5146008Z Runner name: 'i-0ec6a6fea89e9eddd' 2024-04-03T16:13:29.5147133Z Runner group name: 'Default' 2024-04-03T16:13:29.5148217Z Machine name: 'ip-10-0-38-132' 2024-04-03T16:13:29.5154048Z ##[group]GITHUB_TOKEN Permissions 2024-04-03T16:13:29.5156820Z Actions: read 2024-04-03T16:13:29.5157485Z Checks: read 2024-04-03T16:13:29.5158132Z Contents: read 2024-04-03T16:13:29.5158853Z Deployments: read 2024-04-03T16:13:29.5159492Z Discussions: read 2024-04-03T16:13:29.5160166Z Issues: read 2024-04-03T16:13:29.5160911Z Metadata: read 2024-04-03T16:13:29.5161523Z Packages: read 2024-04-03T16:13:29.5162169Z Pages: read 2024-04-03T16:13:29.5162863Z PullRequests: read 2024-04-03T16:13:29.5163553Z RepositoryProjects: read 2024-04-03T16:13:29.5164340Z SecurityEvents: read 2024-04-03T16:13:29.5165130Z Statuses: read 2024-04-03T16:13:29.5165746Z ##[endgroup] 2024-04-03T16:13:29.5170517Z Secret source: None 2024-04-03T16:13:29.5171747Z Prepare workflow directory 2024-04-03T16:13:29.9984321Z Prepare all required actions 2024-04-03T16:13:30.0232609Z Getting action download info 2024-04-03T16:13:30.2537620Z Download action repository 'pytorch/test-infra@main' (SHA:f5543b9b2d7b7c7b8eccd6576b68fc0472b1fbde) 2024-04-03T16:13:30.6704743Z Download action repository 'pytorch/pytorch@main' (SHA:d895192e87f9a2c9ab3d6fab60f08855a08c3900) 2024-04-03T16:13:34.9518443Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-04-03T16:13:35.1056844Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-04-03T16:13:35.4208582Z Getting action download info 2024-04-03T16:13:35.5252372Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-04-03T16:13:35.7270402Z Getting action download info 2024-04-03T16:13:35.8407584Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-04-03T16:13:36.0155344Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/123154/merge (4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0) 2024-04-03T16:13:36.0158968Z ##[group] Inputs 2024-04-03T16:13:36.0159558Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-04-03T16:13:36.0161677Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-04-03T16:13:36.0164476Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:13:36.0165826Z sync-tag: 2024-04-03T16:13:36.0167006Z timeout-minutes: 360 2024-04-03T16:13:36.0167428Z use-gha: 2024-04-03T16:13:36.0167795Z dashboard-tag: 2024-04-03T16:13:36.0168205Z s3-bucket: gha-artifacts 2024-04-03T16:13:36.0168632Z aws-role-to-assume: 2024-04-03T16:13:36.0169050Z ##[endgroup] 2024-04-03T16:13:36.0170649Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:13:36.1424495Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-04-03T16:13:36.1425241Z with: 2024-04-03T16:13:36.1426022Z github-secret: *** 2024-04-03T16:13:36.1427222Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-04-03T16:13:36.1428512Z activate-with-label: false 2024-04-03T16:13:36.1429034Z label: with-ssh 2024-04-03T16:13:36.1429430Z remove-existing-keys: true 2024-04-03T16:13:36.1429906Z env: 2024-04-03T16:13:36.1430242Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:13:36.1430654Z ##[endgroup] 2024-04-03T16:13:36.2749200Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-04-03T16:13:36.6874199Z Grabbing public ssh keys from https://github.com/caic99.keys 2024-04-03T16:13:36.7585985Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-04-03T16:13:36.7609384Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-04-03T16:13:36.7654862Z Login using: ssh ec2-user@ec2-18-234-161-52.compute-1.amazonaws.com 2024-04-03T16:13:36.7655896Z All testing is done inside the container, to start an interactive session run: 2024-04-03T16:13:36.7657115Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-04-03T16:13:36.8045197Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-04-03T16:13:36.8046102Z with: 2024-04-03T16:13:36.8046535Z submodules: recursive 2024-04-03T16:13:36.8047042Z fetch-depth: 0 2024-04-03T16:13:36.8047463Z env: 2024-04-03T16:13:36.8047870Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:13:36.8048379Z ##[endgroup] 2024-04-03T16:13:36.8365335Z ##[group]Run retry () { 2024-04-03T16:13:36.8365938Z retry () { 2024-04-03T16:13:36.8366632Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-04-03T16:13:36.8367417Z } 2024-04-03T16:13:36.8367879Z echo "${GITHUB_WORKSPACE}" 2024-04-03T16:13:36.8368490Z if [ -z "${NO_SUDO}" ]; then 2024-04-03T16:13:36.8369435Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-04-03T16:13:36.8370117Z else 2024-04-03T16:13:36.8370629Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-04-03T16:13:36.8371231Z fi 2024-04-03T16:13:36.8371763Z mkdir "${GITHUB_WORKSPACE}" 2024-04-03T16:13:36.8402424Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:13:36.8403173Z env: 2024-04-03T16:13:36.8403581Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:13:36.8404094Z NO_SUDO: 2024-04-03T16:13:36.8404524Z ##[endgroup] 2024-04-03T16:13:36.8538176Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:13:40.3212024Z ##[group]Run malfet/checkout@silent-checkout 2024-04-03T16:13:40.3212641Z with: 2024-04-03T16:13:40.3213037Z ref: d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:13:40.3213596Z fetch-depth: 0 2024-04-03T16:13:40.3213981Z submodules: recursive 2024-04-03T16:13:40.3214384Z quiet-checkout: true 2024-04-03T16:13:40.3214817Z repository: pytorch/pytorch 2024-04-03T16:13:40.3215430Z token: *** 2024-04-03T16:13:40.3215769Z ssh-strict: true 2024-04-03T16:13:40.3216174Z persist-credentials: true 2024-04-03T16:13:40.3216616Z clean: true 2024-04-03T16:13:40.3217010Z sparse-checkout-cone-mode: true 2024-04-03T16:13:40.3217491Z lfs: false 2024-04-03T16:13:40.3217857Z set-safe-directory: true 2024-04-03T16:13:40.3218251Z env: 2024-04-03T16:13:40.3218581Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:13:40.3219003Z ##[endgroup] 2024-04-03T16:13:40.4539407Z Syncing repository: pytorch/pytorch 2024-04-03T16:13:40.4541606Z ##[group]Getting Git version info 2024-04-03T16:13:40.4542556Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-04-03T16:13:40.4543653Z [command]/usr/bin/git version 2024-04-03T16:13:40.4544119Z git version 2.40.1 2024-04-03T16:13:40.4547962Z ##[endgroup] 2024-04-03T16:13:40.4570419Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/26de8977-14e1-4f4e-9742-341bcf33b09c' before making global git config changes 2024-04-03T16:13:40.4572030Z Adding repository directory to the temporary git global config as a safe directory 2024-04-03T16:13:40.4577651Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:13:40.4621471Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-04-03T16:13:40.4627613Z ##[group]Initializing the repository 2024-04-03T16:13:40.4631876Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:13:40.4663082Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-04-03T16:13:40.4664546Z hint: is subject to change. To configure the initial branch name to use in all 2024-04-03T16:13:40.4665758Z hint: of your new repositories, which will suppress this warning, call: 2024-04-03T16:13:40.4666584Z hint: 2024-04-03T16:13:40.4667258Z hint: git config --global init.defaultBranch 2024-04-03T16:13:40.4667995Z hint: 2024-04-03T16:13:40.4668693Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-04-03T16:13:40.4669888Z hint: 'development'. The just-created branch can be renamed via this command: 2024-04-03T16:13:40.4670983Z hint: 2024-04-03T16:13:40.4671530Z hint: git branch -m 2024-04-03T16:13:40.4672530Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-04-03T16:13:40.4676416Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-04-03T16:13:40.4710186Z ##[endgroup] 2024-04-03T16:13:40.4711061Z ##[group]Disabling automatic garbage collection 2024-04-03T16:13:40.4714260Z [command]/usr/bin/git config --local gc.auto 0 2024-04-03T16:13:40.4747844Z ##[endgroup] 2024-04-03T16:13:40.4748632Z ##[group]Setting up auth 2024-04-03T16:13:40.4757264Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-04-03T16:13:40.4791732Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-04-03T16:13:40.5084690Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-04-03T16:13:40.5113799Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-04-03T16:13:40.5400627Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-04-03T16:13:40.5446797Z ##[endgroup] 2024-04-03T16:13:40.5447682Z ##[group]Fetching the repository 2024-04-03T16:13:40.5456022Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-04-03T16:13:45.7162137Z remote: Enumerating objects: 1123747 2024-04-03T16:13:45.7163171Z remote: Enumerating objects: 1124411, done. 2024-04-03T16:13:45.7164501Z remote: Counting objects: 0% (1/664) 2024-04-03T16:13:45.7165485Z remote: Counting objects: 1% (7/664) 2024-04-03T16:13:45.7166410Z remote: Counting objects: 2% (14/664) 2024-04-03T16:13:45.7167223Z remote: Counting objects: 3% (20/664) 2024-04-03T16:13:45.7167953Z remote: Counting objects: 4% (27/664) 2024-04-03T16:13:45.7168714Z remote: Counting objects: 5% (34/664) 2024-04-03T16:13:45.7169957Z remote: Counting objects: 6% (40/664) 2024-04-03T16:13:45.7170733Z remote: Counting objects: 7% (47/664) 2024-04-03T16:13:45.7171532Z remote: Counting objects: 8% (54/664) 2024-04-03T16:13:45.7172235Z remote: Counting objects: 9% (60/664) 2024-04-03T16:13:45.7172964Z remote: Counting objects: 10% (67/664) 2024-04-03T16:13:45.7173703Z remote: Counting objects: 11% (74/664) 2024-04-03T16:13:45.7174434Z remote: Counting objects: 12% (80/664) 2024-04-03T16:13:45.7175157Z remote: Counting objects: 13% (87/664) 2024-04-03T16:13:45.7175946Z remote: Counting objects: 14% (93/664) 2024-04-03T16:13:45.7176707Z remote: Counting objects: 15% (100/664) 2024-04-03T16:13:45.7177446Z remote: Counting objects: 16% (107/664) 2024-04-03T16:13:45.7178191Z remote: Counting objects: 17% (113/664) 2024-04-03T16:13:45.7178944Z remote: Counting objects: 18% (120/664) 2024-04-03T16:13:45.7179870Z remote: Counting objects: 19% (127/664) 2024-04-03T16:13:45.7181051Z remote: Counting objects: 20% (133/664) 2024-04-03T16:13:45.7181746Z remote: Counting objects: 21% (140/664) 2024-04-03T16:13:45.7182535Z remote: Counting objects: 22% (147/664) 2024-04-03T16:13:45.7183287Z remote: Counting objects: 23% (153/664) 2024-04-03T16:13:45.7183969Z remote: Counting objects: 24% (160/664) 2024-04-03T16:13:45.7184796Z remote: Counting objects: 25% (166/664) 2024-04-03T16:13:45.7185531Z remote: Counting objects: 26% (173/664) 2024-04-03T16:13:45.7229600Z remote: Counting objects: 27% (180/664) 2024-04-03T16:13:45.7230670Z remote: Counting objects: 28% (186/664) 2024-04-03T16:13:45.7231284Z remote: Counting objects: 29% (193/664) 2024-04-03T16:13:45.7231920Z remote: Counting objects: 30% (200/664) 2024-04-03T16:13:45.7232642Z remote: Counting objects: 31% (206/664) 2024-04-03T16:13:45.7233275Z remote: Counting objects: 32% (213/664) 2024-04-03T16:13:45.7233900Z remote: Counting objects: 33% (220/664) 2024-04-03T16:13:45.7234537Z remote: Counting objects: 34% (226/664) 2024-04-03T16:13:45.7235160Z remote: Counting objects: 35% (233/664) 2024-04-03T16:13:45.7235769Z remote: Counting objects: 36% (240/664) 2024-04-03T16:13:45.7236392Z remote: Counting objects: 37% (246/664) 2024-04-03T16:13:45.7237017Z remote: Counting objects: 38% (253/664) 2024-04-03T16:13:45.7237620Z remote: Counting objects: 39% (259/664) 2024-04-03T16:13:45.7238241Z remote: Counting objects: 40% (266/664) 2024-04-03T16:13:45.7238879Z remote: Counting objects: 41% (273/664) 2024-04-03T16:13:45.7239482Z remote: Counting objects: 42% (279/664) 2024-04-03T16:13:45.7240106Z remote: Counting objects: 43% (286/664) 2024-04-03T16:13:45.7240729Z remote: Counting objects: 44% (293/664) 2024-04-03T16:13:45.7241334Z remote: Counting objects: 45% (299/664) 2024-04-03T16:13:45.7241969Z remote: Counting objects: 46% (306/664) 2024-04-03T16:13:45.7242589Z remote: Counting objects: 47% (313/664) 2024-04-03T16:13:45.7243213Z remote: Counting objects: 48% (319/664) 2024-04-03T16:13:45.7243817Z remote: Counting objects: 49% (326/664) 2024-04-03T16:13:45.7244444Z remote: Counting objects: 50% (332/664) 2024-04-03T16:13:45.7245068Z remote: Counting objects: 51% (339/664) 2024-04-03T16:13:45.7245672Z remote: Counting objects: 52% (346/664) 2024-04-03T16:13:45.7246300Z remote: Counting objects: 53% (352/664) 2024-04-03T16:13:45.7246928Z remote: Counting objects: 54% (359/664) 2024-04-03T16:13:45.7247536Z remote: Counting objects: 55% (366/664) 2024-04-03T16:13:45.7248163Z remote: Counting objects: 56% (372/664) 2024-04-03T16:13:45.7248795Z remote: Counting objects: 57% (379/664) 2024-04-03T16:13:45.7249649Z remote: Counting objects: 58% (386/664) 2024-04-03T16:13:45.7250281Z remote: Counting objects: 59% (392/664) 2024-04-03T16:13:45.7250914Z remote: Counting objects: 60% (399/664) 2024-04-03T16:13:45.7251517Z remote: Counting objects: 61% (406/664) 2024-04-03T16:13:45.7252138Z remote: Counting objects: 62% (412/664) 2024-04-03T16:13:45.7252759Z remote: Counting objects: 63% (419/664) 2024-04-03T16:13:45.7253366Z remote: Counting objects: 64% (425/664) 2024-04-03T16:13:45.7253989Z remote: Counting objects: 65% (432/664) 2024-04-03T16:13:45.7254617Z remote: Counting objects: 66% (439/664) 2024-04-03T16:13:45.7255234Z remote: Counting objects: 67% (445/664) 2024-04-03T16:13:45.7255860Z remote: Counting objects: 68% (452/664) 2024-04-03T16:13:45.7256481Z remote: Counting objects: 69% (459/664) 2024-04-03T16:13:45.7257104Z remote: Counting objects: 70% (465/664) 2024-04-03T16:13:45.7257713Z remote: Counting objects: 71% (472/664) 2024-04-03T16:13:45.7258335Z remote: Counting objects: 72% (479/664) 2024-04-03T16:13:45.7259126Z remote: Counting objects: 73% (485/664) 2024-04-03T16:13:45.7259731Z remote: Counting objects: 74% (492/664) 2024-04-03T16:13:45.7260363Z remote: Counting objects: 75% (498/664) 2024-04-03T16:13:45.7260983Z remote: Counting objects: 76% (505/664) 2024-04-03T16:13:45.7261587Z remote: Counting objects: 77% (512/664) 2024-04-03T16:13:45.7262203Z remote: Counting objects: 78% (518/664) 2024-04-03T16:13:45.7262822Z remote: Counting objects: 79% (525/664) 2024-04-03T16:13:45.7263427Z remote: Counting objects: 80% (532/664) 2024-04-03T16:13:45.7264168Z remote: Counting objects: 81% (538/664) 2024-04-03T16:13:45.7265227Z remote: Counting objects: 82% (545/664) 2024-04-03T16:13:45.7266198Z remote: Counting objects: 83% (552/664) 2024-04-03T16:13:45.7266867Z remote: Counting objects: 84% (558/664) 2024-04-03T16:13:45.7267486Z remote: Counting objects: 85% (565/664) 2024-04-03T16:13:45.7268076Z remote: Counting objects: 86% (572/664) 2024-04-03T16:13:45.7268705Z remote: Counting objects: 87% (578/664) 2024-04-03T16:13:45.7269329Z remote: Counting objects: 88% (585/664) 2024-04-03T16:13:45.7269933Z remote: Counting objects: 89% (591/664) 2024-04-03T16:13:45.7270556Z remote: Counting objects: 90% (598/664) 2024-04-03T16:13:45.7271175Z remote: Counting objects: 91% (605/664) 2024-04-03T16:13:45.7271777Z remote: Counting objects: 92% (611/664) 2024-04-03T16:13:45.7272405Z remote: Counting objects: 93% (618/664) 2024-04-03T16:13:45.7273042Z remote: Counting objects: 94% (625/664) 2024-04-03T16:13:45.7273673Z remote: Counting objects: 95% (631/664) 2024-04-03T16:13:45.7274279Z remote: Counting objects: 96% (638/664) 2024-04-03T16:13:45.7274907Z remote: Counting objects: 97% (645/664) 2024-04-03T16:13:45.7275530Z remote: Counting objects: 98% (651/664) 2024-04-03T16:13:45.7276165Z remote: Counting objects: 99% (658/664) 2024-04-03T16:13:45.7276769Z remote: Counting objects: 100% (664/664) 2024-04-03T16:13:45.7277437Z remote: Counting objects: 100% (664/664), done. 2024-04-03T16:13:45.7278149Z remote: Compressing objects: 0% (1/369) 2024-04-03T16:13:45.7278784Z remote: Compressing objects: 1% (4/369) 2024-04-03T16:13:45.7334751Z remote: Compressing objects: 2% (8/369) 2024-04-03T16:13:45.7464708Z remote: Compressing objects: 3% (12/369) 2024-04-03T16:13:45.7528951Z remote: Compressing objects: 4% (15/369) 2024-04-03T16:13:45.7858603Z remote: Compressing objects: 5% (19/369) 2024-04-03T16:13:45.7936402Z remote: Compressing objects: 6% (23/369) 2024-04-03T16:13:45.8048950Z remote: Compressing objects: 7% (26/369) 2024-04-03T16:13:45.8171504Z remote: Compressing objects: 8% (30/369) 2024-04-03T16:13:45.8195966Z remote: Compressing objects: 9% (34/369) 2024-04-03T16:13:45.8244530Z remote: Compressing objects: 10% (37/369) 2024-04-03T16:13:45.8322131Z remote: Compressing objects: 11% (41/369) 2024-04-03T16:13:45.8333199Z remote: Compressing objects: 12% (45/369) 2024-04-03T16:13:45.8371646Z remote: Compressing objects: 13% (48/369) 2024-04-03T16:13:45.8381112Z remote: Compressing objects: 14% (52/369) 2024-04-03T16:13:45.8387665Z remote: Compressing objects: 15% (56/369) 2024-04-03T16:13:45.8419528Z remote: Compressing objects: 16% (60/369) 2024-04-03T16:13:45.8430397Z remote: Compressing objects: 17% (63/369) 2024-04-03T16:13:45.8441552Z remote: Compressing objects: 18% (67/369) 2024-04-03T16:13:45.8444459Z remote: Compressing objects: 19% (71/369) 2024-04-03T16:13:45.8448770Z remote: Compressing objects: 20% (74/369) 2024-04-03T16:13:45.8456016Z remote: Compressing objects: 21% (78/369) 2024-04-03T16:13:45.8456749Z remote: Compressing objects: 22% (82/369) 2024-04-03T16:13:45.8458514Z remote: Compressing objects: 23% (85/369) 2024-04-03T16:13:45.8461226Z remote: Compressing objects: 24% (89/369) 2024-04-03T16:13:45.8463853Z remote: Compressing objects: 25% (93/369) 2024-04-03T16:13:45.8471308Z remote: Compressing objects: 26% (96/369) 2024-04-03T16:13:45.8478308Z remote: Compressing objects: 27% (100/369) 2024-04-03T16:13:45.8479220Z remote: Compressing objects: 28% (104/369) 2024-04-03T16:13:45.8485992Z remote: Compressing objects: 29% (108/369) 2024-04-03T16:13:45.8487018Z remote: Compressing objects: 30% (111/369) 2024-04-03T16:13:45.8492334Z remote: Compressing objects: 31% (115/369) 2024-04-03T16:13:45.8493194Z remote: Compressing objects: 32% (119/369) 2024-04-03T16:13:45.8504073Z remote: Compressing objects: 33% (122/369) 2024-04-03T16:13:45.8507892Z remote: Compressing objects: 34% (126/369) 2024-04-03T16:13:45.8512946Z remote: Compressing objects: 35% (130/369) 2024-04-03T16:13:45.8521015Z remote: Compressing objects: 36% (133/369) 2024-04-03T16:13:45.8525344Z remote: Compressing objects: 37% (137/369) 2024-04-03T16:13:45.8533713Z remote: Compressing objects: 38% (141/369) 2024-04-03T16:13:45.8536347Z remote: Compressing objects: 39% (144/369) 2024-04-03T16:13:45.8539060Z remote: Compressing objects: 40% (148/369) 2024-04-03T16:13:45.8539939Z remote: Compressing objects: 41% (152/369) 2024-04-03T16:13:45.8540607Z remote: Compressing objects: 42% (155/369) 2024-04-03T16:13:45.8541483Z remote: Compressing objects: 43% (159/369) 2024-04-03T16:13:45.8544132Z remote: Compressing objects: 44% (163/369) 2024-04-03T16:13:45.8550025Z remote: Compressing objects: 45% (167/369) 2024-04-03T16:13:45.8550696Z remote: Compressing objects: 46% (170/369) 2024-04-03T16:13:45.8553102Z remote: Compressing objects: 47% (174/369) 2024-04-03T16:13:45.8553763Z remote: Compressing objects: 48% (178/369) 2024-04-03T16:13:45.8558980Z remote: Compressing objects: 49% (181/369) 2024-04-03T16:13:45.8561465Z remote: Compressing objects: 50% (185/369) 2024-04-03T16:13:45.8562384Z remote: Compressing objects: 51% (189/369) 2024-04-03T16:13:45.8563050Z remote: Compressing objects: 52% (192/369) 2024-04-03T16:13:45.8563721Z remote: Compressing objects: 53% (196/369) 2024-04-03T16:13:45.8564421Z remote: Compressing objects: 54% (200/369) 2024-04-03T16:13:45.8565286Z remote: Compressing objects: 55% (203/369) 2024-04-03T16:13:45.8565957Z remote: Compressing objects: 56% (207/369) 2024-04-03T16:13:45.8566779Z remote: Compressing objects: 57% (211/369) 2024-04-03T16:13:45.8568196Z remote: Compressing objects: 58% (215/369) 2024-04-03T16:13:45.8568872Z remote: Compressing objects: 59% (218/369) 2024-04-03T16:13:45.8569790Z remote: Compressing objects: 60% (222/369) 2024-04-03T16:13:45.8570445Z remote: Compressing objects: 61% (226/369) 2024-04-03T16:13:45.8571110Z remote: Compressing objects: 62% (229/369) 2024-04-03T16:13:45.8571769Z remote: Compressing objects: 63% (233/369) 2024-04-03T16:13:45.8572414Z remote: Compressing objects: 64% (237/369) 2024-04-03T16:13:45.8573074Z remote: Compressing objects: 65% (240/369) 2024-04-03T16:13:45.8573739Z remote: Compressing objects: 66% (244/369) 2024-04-03T16:13:45.8574384Z remote: Compressing objects: 67% (248/369) 2024-04-03T16:13:45.8575058Z remote: Compressing objects: 68% (251/369) 2024-04-03T16:13:45.8575727Z remote: Compressing objects: 69% (255/369) 2024-04-03T16:13:45.8576386Z remote: Compressing objects: 70% (259/369) 2024-04-03T16:13:45.8577031Z remote: Compressing objects: 71% (262/369) 2024-04-03T16:13:45.8577703Z remote: Compressing objects: 72% (266/369) 2024-04-03T16:13:45.8578562Z remote: Compressing objects: 73% (270/369) 2024-04-03T16:13:45.8579211Z remote: Compressing objects: 74% (274/369) 2024-04-03T16:13:45.8581496Z remote: Compressing objects: 75% (277/369) 2024-04-03T16:13:45.8584333Z remote: Compressing objects: 76% (281/369) 2024-04-03T16:13:45.8585775Z remote: Compressing objects: 77% (285/369) 2024-04-03T16:13:45.8586497Z remote: Compressing objects: 78% (288/369) 2024-04-03T16:13:45.8587298Z remote: Compressing objects: 79% (292/369) 2024-04-03T16:13:45.8588063Z remote: Compressing objects: 80% (296/369) 2024-04-03T16:13:45.8588868Z remote: Compressing objects: 81% (299/369) 2024-04-03T16:13:45.8589559Z remote: Compressing objects: 82% (303/369) 2024-04-03T16:13:45.8590219Z remote: Compressing objects: 83% (307/369) 2024-04-03T16:13:45.8590862Z remote: Compressing objects: 84% (310/369) 2024-04-03T16:13:45.8591519Z remote: Compressing objects: 85% (314/369) 2024-04-03T16:13:45.8592189Z remote: Compressing objects: 86% (318/369) 2024-04-03T16:13:45.8592827Z remote: Compressing objects: 87% (322/369) 2024-04-03T16:13:45.8594268Z remote: Compressing objects: 88% (325/369) 2024-04-03T16:13:45.8595175Z remote: Compressing objects: 89% (329/369) 2024-04-03T16:13:45.8595848Z remote: Compressing objects: 90% (333/369) 2024-04-03T16:13:45.8597015Z remote: Compressing objects: 91% (336/369) 2024-04-03T16:13:45.8597929Z remote: Compressing objects: 92% (340/369) 2024-04-03T16:13:45.8598608Z remote: Compressing objects: 93% (344/369) 2024-04-03T16:13:45.8599246Z remote: Compressing objects: 94% (347/369) 2024-04-03T16:13:45.8600098Z remote: Compressing objects: 95% (351/369) 2024-04-03T16:13:45.8600800Z remote: Compressing objects: 96% (355/369) 2024-04-03T16:13:45.8601438Z remote: Compressing objects: 97% (358/369) 2024-04-03T16:13:45.8602104Z remote: Compressing objects: 98% (362/369) 2024-04-03T16:13:45.8602785Z remote: Compressing objects: 99% (366/369) 2024-04-03T16:13:45.8603448Z remote: Compressing objects: 100% (369/369) 2024-04-03T16:13:45.8604143Z remote: Compressing objects: 100% (369/369), done. 2024-04-03T16:14:13.8068535Z remote: Total 1124411 (delta 364), reused 532 (delta 295), pack-reused 1123747 2024-04-03T16:14:39.9518934Z [command]/usr/bin/git rev-parse --verify --quiet d253bd7befc64137de913dbb8727b511f1052411^{object} 2024-04-03T16:14:39.9548233Z d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:14:39.9553973Z ##[endgroup] 2024-04-03T16:14:39.9554733Z ##[group]Determining the checkout info 2024-04-03T16:14:39.9555917Z ##[endgroup] 2024-04-03T16:14:39.9558464Z ##[group]Checking out the ref 2024-04-03T16:14:39.9560808Z [command]/usr/bin/git checkout --quiet --force d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:14:41.9117855Z ##[endgroup] 2024-04-03T16:14:41.9118651Z ##[group]Setting up auth for fetching submodules 2024-04-03T16:14:41.9125136Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-04-03T16:14:41.9180440Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-04-03T16:14:41.9213179Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-04-03T16:14:41.9245715Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-04-03T16:14:41.9275009Z ##[endgroup] 2024-04-03T16:14:41.9275665Z ##[group]Fetching submodules 2024-04-03T16:14:41.9280250Z [command]/usr/bin/git submodule sync --recursive 2024-04-03T16:14:41.9606461Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-04-03T16:14:41.9918016Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-04-03T16:14:41.9920290Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-04-03T16:14:41.9923910Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-04-03T16:14:41.9927099Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-04-03T16:14:41.9930578Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2024-04-03T16:14:41.9936513Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-04-03T16:14:41.9939174Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-04-03T16:14:41.9943491Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-04-03T16:14:41.9947355Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-04-03T16:14:41.9951354Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2024-04-03T16:14:41.9955760Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-04-03T16:14:41.9959967Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-04-03T16:14:41.9964388Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-04-03T16:14:41.9968966Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-04-03T16:14:41.9974592Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-04-03T16:14:41.9979235Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-04-03T16:14:41.9984287Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-04-03T16:14:41.9989834Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:14:41.9994948Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-04-03T16:14:42.0000488Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-04-03T16:14:42.0005957Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-04-03T16:14:42.0012343Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2024-04-03T16:14:42.0018038Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-04-03T16:14:42.0023981Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-04-03T16:14:42.0030184Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-04-03T16:14:42.0036347Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-04-03T16:14:42.0042643Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2024-04-03T16:14:42.0049320Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-04-03T16:14:42.0056135Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-04-03T16:14:42.0062900Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2024-04-03T16:14:42.0069693Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-04-03T16:14:42.0076716Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-04-03T16:14:42.0083785Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-04-03T16:14:42.0091635Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-04-03T16:14:42.0099168Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-04-03T16:14:42.0106683Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-04-03T16:14:42.0114015Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-04-03T16:14:42.0121673Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2024-04-03T16:14:42.0130239Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-04-03T16:14:42.0138502Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2024-04-03T16:14:42.0166167Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-04-03T16:14:42.3010085Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-04-03T16:14:42.4779442Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-04-03T16:14:42.6507224Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-04-03T16:14:42.9056065Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2024-04-03T16:14:43.1850815Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-04-03T16:14:45.3513918Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-04-03T16:14:56.1194120Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-04-03T16:14:56.6077549Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-04-03T16:14:57.3339040Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2024-04-03T16:14:58.9390241Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-04-03T16:15:00.4739145Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-04-03T16:15:03.6182817Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-04-03T16:15:12.4854590Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-04-03T16:15:13.8500559Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-04-03T16:15:15.6871533Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-04-03T16:15:17.0988898Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-04-03T16:15:17.2997438Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-04-03T16:15:17.7592273Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-04-03T16:15:18.0955204Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-04-03T16:15:19.3751903Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-04-03T16:15:19.7500933Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2024-04-03T16:15:20.1379299Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-04-03T16:15:20.3837327Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-04-03T16:15:22.2403309Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-04-03T16:15:22.9332516Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-04-03T16:15:23.5397316Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2024-04-03T16:15:23.8321221Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-04-03T16:15:31.3343541Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-04-03T16:15:33.8685788Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2024-04-03T16:15:34.3921137Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-04-03T16:15:34.6034644Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-04-03T16:15:44.0391106Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-04-03T16:15:44.2131086Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-04-03T16:15:44.4150187Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-04-03T16:15:45.4953616Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-04-03T16:15:45.8099248Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-04-03T16:15:46.4706914Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2024-04-03T16:15:48.9096718Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-04-03T16:15:49.3993020Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2024-04-03T16:15:52.0795383Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-04-03T16:15:52.0917950Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-04-03T16:15:52.1015309Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-04-03T16:15:52.1303429Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-04-03T16:15:52.1569632Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2024-04-03T16:15:52.2025747Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-04-03T16:15:53.3605086Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-04-03T16:15:53.3865632Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-04-03T16:15:53.5030459Z Submodule path 'third_party/cpuinfo': checked out 'd6860c477c99f1fce9e28eb206891af3c0e1a1d7' 2024-04-03T16:15:53.5438307Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2024-04-03T16:15:53.5761423Z Submodule path 'third_party/cudnn_frontend': checked out '150798fe976556078f443fdb059a1ff0361f58a2' 2024-04-03T16:15:54.2354546Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-04-03T16:15:54.5412544Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-04-03T16:15:54.6239625Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-04-03T16:15:54.6258231Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:15:54.6260904Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:15:54.6264386Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:15:54.6267646Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:15:54.6271514Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:15:54.6298559Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-04-03T16:15:55.7214758Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-04-03T16:15:56.4093439Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-04-03T16:15:59.2688128Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-04-03T16:16:00.5083680Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-04-03T16:16:00.8630168Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-04-03T16:16:00.9821320Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-04-03T16:16:01.5083780Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-04-03T16:16:01.5844334Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-04-03T16:16:01.5970788Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-04-03T16:16:01.7350466Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-04-03T16:16:01.7775851Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-04-03T16:16:01.7876825Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-04-03T16:16:01.8342075Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-04-03T16:16:01.8624766Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-04-03T16:16:01.9184422Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-04-03T16:16:01.9325236Z Submodule path 'third_party/ideep': checked out '8a6cc4e09dc509f04f83c085e38786b1fb44e14d' 2024-04-03T16:16:01.9342172Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-04-03T16:16:01.9368135Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-04-03T16:16:14.8739658Z Submodule path 'third_party/ideep/mkl-dnn': checked out '86e6af5974177e513fd3fee58425e1063e7f1361' 2024-04-03T16:16:14.8856252Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2024-04-03T16:16:14.9045590Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-04-03T16:16:15.0228857Z Submodule path 'third_party/kineto': checked out '8466a8b111b36dc725e6855d52a0b133d925a8e0' 2024-04-03T16:16:15.0247684Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:16:15.0250952Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:16:15.0256100Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:16:15.0283042Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-04-03T16:16:15.8429132Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-04-03T16:16:17.2419454Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-04-03T16:16:18.5803944Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-04-03T16:16:18.5821629Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:16:18.5825591Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:16:18.5829712Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:16:18.5833983Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:16:18.5838339Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:16:18.5842890Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:16:18.5847356Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:16:18.5852523Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:16:18.5880536Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-04-03T16:16:19.7309247Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-04-03T16:16:20.1171442Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-04-03T16:16:21.5074771Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-04-03T16:16:21.8107141Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-04-03T16:16:22.1814430Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-04-03T16:16:23.4036360Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-04-03T16:16:30.8312989Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-04-03T16:16:31.2565900Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-04-03T16:16:31.2773337Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-04-03T16:16:31.3218109Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-04-03T16:16:31.3362767Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-04-03T16:16:31.3380034Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:16:31.3406524Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-04-03T16:16:31.6989739Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-04-03T16:16:31.7193850Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-04-03T16:16:31.7670045Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-04-03T16:16:31.8957315Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-04-03T16:16:31.9141370Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-04-03T16:16:31.9555502Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-04-03T16:16:32.0233865Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-04-03T16:16:32.0647836Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-04-03T16:16:32.0917870Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-04-03T16:16:32.1070823Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2024-04-03T16:16:32.2403492Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-04-03T16:16:32.7415950Z Submodule path 'third_party/onnx': checked out 'ccde5da81388ffa770ca98b64e07f803ad089414' 2024-04-03T16:16:32.7620805Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-04-03T16:16:32.7627638Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-04-03T16:16:32.7707062Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-04-03T16:16:33.1719155Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-04-03T16:16:34.2142310Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-04-03T16:16:34.2538003Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-04-03T16:16:34.2750404Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2024-04-03T16:16:34.2768540Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:16:34.2794802Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2024-04-03T16:16:36.8321033Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2024-04-03T16:16:36.8343387Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:16:36.8346012Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:16:36.8373933Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2024-04-03T16:16:37.2630623Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2024-04-03T16:16:38.2754865Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2024-04-03T16:16:38.3607466Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2024-04-03T16:16:38.3626603Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:16:38.3652203Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2024-04-03T16:16:38.5693384Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-04-03T16:16:38.5797483Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-04-03T16:16:38.9041187Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-04-03T16:16:38.9066338Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:16:38.9068766Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-04-03T16:16:38.9096811Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-04-03T16:16:39.3206640Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-04-03T16:16:40.6752454Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-04-03T16:16:40.7614914Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-04-03T16:16:40.7709315Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-04-03T16:16:40.7841313Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-04-03T16:16:40.8229287Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-04-03T16:16:40.8553909Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-04-03T16:16:40.9040073Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-04-03T16:16:41.0406842Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2024-04-03T16:16:41.0720692Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-04-03T16:16:41.0738712Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:16:41.0741265Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:16:41.0744933Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:16:41.0748328Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:16:41.0774297Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-04-03T16:16:42.2901594Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-04-03T16:16:42.5334463Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-04-03T16:16:43.8909613Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-04-03T16:16:44.9414986Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-04-03T16:16:44.9584868Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-04-03T16:16:45.0403781Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-04-03T16:16:45.0724982Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-04-03T16:16:45.0742558Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:16:45.0768301Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-04-03T16:16:45.2804757Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-04-03T16:16:45.4544752Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2024-04-03T16:16:45.4579649Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-04-03T16:16:45.4910442Z Entering 'android/libs/fbjni' 2024-04-03T16:16:45.4953657Z Entering 'third_party/FP16' 2024-04-03T16:16:45.4997201Z Entering 'third_party/FXdiv' 2024-04-03T16:16:45.5040158Z Entering 'third_party/NNPACK' 2024-04-03T16:16:45.5081744Z Entering 'third_party/QNNPACK' 2024-04-03T16:16:45.5124458Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:16:45.5166377Z Entering 'third_party/XNNPACK' 2024-04-03T16:16:45.5227757Z Entering 'third_party/benchmark' 2024-04-03T16:16:45.5270510Z Entering 'third_party/cpuinfo' 2024-04-03T16:16:45.5314521Z Entering 'third_party/cub' 2024-04-03T16:16:45.5358155Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:16:45.5399446Z Entering 'third_party/cutlass' 2024-04-03T16:16:45.5451404Z Entering 'third_party/eigen' 2024-04-03T16:16:45.5495377Z Entering 'third_party/fbgemm' 2024-04-03T16:16:45.5538544Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:16:45.5579579Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:16:45.5620656Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:16:45.5668393Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:16:45.5709809Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:16:45.5750895Z Entering 'third_party/flatbuffers' 2024-04-03T16:16:45.5796824Z Entering 'third_party/fmt' 2024-04-03T16:16:45.5838879Z Entering 'third_party/foxi' 2024-04-03T16:16:45.5879472Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:16:45.5923078Z Entering 'third_party/gloo' 2024-04-03T16:16:45.5963975Z Entering 'third_party/googletest' 2024-04-03T16:16:45.6006407Z Entering 'third_party/ideep' 2024-04-03T16:16:45.6046324Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:16:45.6093933Z Entering 'third_party/ios-cmake' 2024-04-03T16:16:45.6135837Z Entering 'third_party/ittapi' 2024-04-03T16:16:45.6176012Z Entering 'third_party/kineto' 2024-04-03T16:16:45.6217247Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:16:45.6258835Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:16:45.6301615Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:16:45.6342217Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:16:45.6382304Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:16:45.6420808Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:16:45.6463226Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:16:45.6503199Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:16:45.6544013Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:16:45.6585547Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:16:45.6627023Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:16:45.6667113Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:16:45.6708895Z Entering 'third_party/mimalloc' 2024-04-03T16:16:45.6751532Z Entering 'third_party/nccl/nccl' 2024-04-03T16:16:45.6792708Z Entering 'third_party/neon2sse' 2024-04-03T16:16:45.6833196Z Entering 'third_party/nlohmann' 2024-04-03T16:16:45.6875523Z Entering 'third_party/onnx' 2024-04-03T16:16:45.6934362Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:16:45.6975978Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:16:45.7018802Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:16:45.7059287Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:16:45.7104363Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:16:45.7145223Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:16:45.7184871Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:16:45.7230554Z Entering 'third_party/pocketfft' 2024-04-03T16:16:45.7271713Z Entering 'third_party/protobuf' 2024-04-03T16:16:45.7315972Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:16:45.7356102Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:16:45.7398137Z Entering 'third_party/psimd' 2024-04-03T16:16:45.7440406Z Entering 'third_party/pthreadpool' 2024-04-03T16:16:45.7482635Z Entering 'third_party/pybind11' 2024-04-03T16:16:45.7523626Z Entering 'third_party/python-peachpy' 2024-04-03T16:16:45.7564590Z Entering 'third_party/sleef' 2024-04-03T16:16:45.7606541Z Entering 'third_party/tbb' 2024-04-03T16:16:45.7649229Z Entering 'third_party/tensorpipe' 2024-04-03T16:16:45.7690105Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:16:45.7730221Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:16:45.7770250Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:16:45.7811225Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:16:45.7849638Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:16:45.7894356Z Entering 'third_party/zstd' 2024-04-03T16:16:45.7949283Z ##[endgroup] 2024-04-03T16:16:45.7950262Z ##[group]Persisting credentials for submodules 2024-04-03T16:16:45.7956577Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-04-03T16:16:45.8271909Z Entering 'android/libs/fbjni' 2024-04-03T16:16:45.8322777Z Entering 'third_party/FP16' 2024-04-03T16:16:45.8376208Z Entering 'third_party/FXdiv' 2024-04-03T16:16:45.8428094Z Entering 'third_party/NNPACK' 2024-04-03T16:16:45.8479663Z Entering 'third_party/QNNPACK' 2024-04-03T16:16:45.8531698Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:16:45.8583893Z Entering 'third_party/XNNPACK' 2024-04-03T16:16:45.8653283Z Entering 'third_party/benchmark' 2024-04-03T16:16:45.8704753Z Entering 'third_party/cpuinfo' 2024-04-03T16:16:45.8757934Z Entering 'third_party/cub' 2024-04-03T16:16:45.8810113Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:16:45.8861153Z Entering 'third_party/cutlass' 2024-04-03T16:16:45.8920459Z Entering 'third_party/eigen' 2024-04-03T16:16:45.8974877Z Entering 'third_party/fbgemm' 2024-04-03T16:16:45.9027469Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:16:45.9078126Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:16:45.9128092Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:16:45.9187188Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:16:45.9238077Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:16:45.9290294Z Entering 'third_party/flatbuffers' 2024-04-03T16:16:45.9345057Z Entering 'third_party/fmt' 2024-04-03T16:16:45.9396317Z Entering 'third_party/foxi' 2024-04-03T16:16:45.9448356Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:16:45.9500005Z Entering 'third_party/gloo' 2024-04-03T16:16:45.9551287Z Entering 'third_party/googletest' 2024-04-03T16:16:45.9603359Z Entering 'third_party/ideep' 2024-04-03T16:16:45.9654101Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:16:45.9712034Z Entering 'third_party/ios-cmake' 2024-04-03T16:16:45.9762382Z Entering 'third_party/ittapi' 2024-04-03T16:16:45.9814726Z Entering 'third_party/kineto' 2024-04-03T16:16:45.9864754Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:16:45.9915490Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:16:45.9967484Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:16:46.0018467Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:16:46.0068468Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:16:46.0119408Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:16:46.0172075Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:16:46.0222055Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:16:46.0273230Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:16:46.0325354Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:16:46.0378812Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:16:46.0429128Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:16:46.0481604Z Entering 'third_party/mimalloc' 2024-04-03T16:16:46.0533069Z Entering 'third_party/nccl/nccl' 2024-04-03T16:16:46.0584926Z Entering 'third_party/neon2sse' 2024-04-03T16:16:46.0636313Z Entering 'third_party/nlohmann' 2024-04-03T16:16:46.0688419Z Entering 'third_party/onnx' 2024-04-03T16:16:46.0759193Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:16:46.0809770Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:16:46.0863884Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:16:46.0913896Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:16:46.0969887Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:16:46.1022408Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:16:46.1074906Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:16:46.1131197Z Entering 'third_party/pocketfft' 2024-04-03T16:16:46.1182150Z Entering 'third_party/protobuf' 2024-04-03T16:16:46.1236230Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:16:46.1286785Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:16:46.1339540Z Entering 'third_party/psimd' 2024-04-03T16:16:46.1391344Z Entering 'third_party/pthreadpool' 2024-04-03T16:16:46.1442432Z Entering 'third_party/pybind11' 2024-04-03T16:16:46.1494301Z Entering 'third_party/python-peachpy' 2024-04-03T16:16:46.1545378Z Entering 'third_party/sleef' 2024-04-03T16:16:46.1596128Z Entering 'third_party/tbb' 2024-04-03T16:16:46.1649730Z Entering 'third_party/tensorpipe' 2024-04-03T16:16:46.1699673Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:16:46.1749576Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:16:46.1799778Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:16:46.1850143Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:16:46.1898498Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:16:46.1951417Z Entering 'third_party/zstd' 2024-04-03T16:16:46.2019211Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-04-03T16:16:46.2335498Z Entering 'android/libs/fbjni' 2024-04-03T16:16:46.2382046Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-04-03T16:16:46.2400621Z Entering 'third_party/FP16' 2024-04-03T16:16:46.2447791Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-04-03T16:16:46.2467086Z Entering 'third_party/FXdiv' 2024-04-03T16:16:46.2515014Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-04-03T16:16:46.2534259Z Entering 'third_party/NNPACK' 2024-04-03T16:16:46.2581791Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-04-03T16:16:46.2600932Z Entering 'third_party/QNNPACK' 2024-04-03T16:16:46.2648280Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2024-04-03T16:16:46.2667592Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:16:46.2715332Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-04-03T16:16:46.2735521Z Entering 'third_party/XNNPACK' 2024-04-03T16:16:46.2782746Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-04-03T16:16:46.2820655Z Entering 'third_party/benchmark' 2024-04-03T16:16:46.2868890Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:16:46.2887651Z Entering 'third_party/cpuinfo' 2024-04-03T16:16:46.2936103Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-04-03T16:16:46.2956051Z Entering 'third_party/cub' 2024-04-03T16:16:46.3004129Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2024-04-03T16:16:46.3023914Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:16:46.3071187Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-04-03T16:16:46.3090034Z Entering 'third_party/cutlass' 2024-04-03T16:16:46.3137695Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-04-03T16:16:46.3164394Z Entering 'third_party/eigen' 2024-04-03T16:16:46.3212185Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-04-03T16:16:46.3233819Z Entering 'third_party/fbgemm' 2024-04-03T16:16:46.3281508Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-04-03T16:16:46.3300416Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:16:46.3347296Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-04-03T16:16:46.3365394Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:16:46.3412466Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-04-03T16:16:46.3430949Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:16:46.3477847Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-04-03T16:16:46.3502938Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:16:46.3549418Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:16:46.3567347Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:16:46.3615077Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-04-03T16:16:46.3633861Z Entering 'third_party/flatbuffers' 2024-04-03T16:16:46.3680965Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-04-03T16:16:46.3703199Z Entering 'third_party/fmt' 2024-04-03T16:16:46.3750478Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-04-03T16:16:46.3769130Z Entering 'third_party/foxi' 2024-04-03T16:16:46.3816384Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-04-03T16:16:46.3835718Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:16:46.3882888Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-04-03T16:16:46.3901835Z Entering 'third_party/gloo' 2024-04-03T16:16:46.3949474Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-04-03T16:16:46.3968151Z Entering 'third_party/googletest' 2024-04-03T16:16:46.4015921Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:16:46.4034516Z Entering 'third_party/ideep' 2024-04-03T16:16:46.4082421Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-04-03T16:16:46.4101168Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:16:46.4147716Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-04-03T16:16:46.4173470Z Entering 'third_party/ios-cmake' 2024-04-03T16:16:46.4220474Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2024-04-03T16:16:46.4238424Z Entering 'third_party/ittapi' 2024-04-03T16:16:46.4286862Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-04-03T16:16:46.4305234Z Entering 'third_party/kineto' 2024-04-03T16:16:46.4352912Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-04-03T16:16:46.4372340Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:16:46.4420574Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-04-03T16:16:46.4438157Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:16:46.4485820Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-04-03T16:16:46.4505825Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:16:46.4553775Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-04-03T16:16:46.4572367Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:16:46.4620408Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-04-03T16:16:46.4638520Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:16:46.4685587Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-04-03T16:16:46.4703981Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:16:46.4752118Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-04-03T16:16:46.4771698Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:16:46.4819845Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-04-03T16:16:46.4838402Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:16:46.4886177Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:16:46.4905107Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:16:46.4952828Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-04-03T16:16:46.4971961Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:16:46.5019017Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-04-03T16:16:46.5038386Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:16:46.5086000Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-04-03T16:16:46.5104344Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:16:46.5151488Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-04-03T16:16:46.5171303Z Entering 'third_party/mimalloc' 2024-04-03T16:16:46.5218304Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-04-03T16:16:46.5237679Z Entering 'third_party/nccl/nccl' 2024-04-03T16:16:46.5285257Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-04-03T16:16:46.5304594Z Entering 'third_party/neon2sse' 2024-04-03T16:16:46.5352134Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2024-04-03T16:16:46.5371896Z Entering 'third_party/nlohmann' 2024-04-03T16:16:46.5419279Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-04-03T16:16:46.5439498Z Entering 'third_party/onnx' 2024-04-03T16:16:46.5488642Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-04-03T16:16:46.5525188Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:16:46.5574760Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:16:46.5592830Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:16:46.5641502Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:16:46.5662977Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:16:46.5710614Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2024-04-03T16:16:46.5728453Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:16:46.5776105Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2024-04-03T16:16:46.5798988Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:16:46.5846999Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:16:46.5865317Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:16:46.5913300Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:16:46.5932037Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:16:46.5980008Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-04-03T16:16:46.6002943Z Entering 'third_party/pocketfft' 2024-04-03T16:16:46.6050554Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-04-03T16:16:46.6069691Z Entering 'third_party/protobuf' 2024-04-03T16:16:46.6118872Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-04-03T16:16:46.6141458Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:16:46.6188275Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:16:46.6206031Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:16:46.6253473Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:16:46.6273746Z Entering 'third_party/psimd' 2024-04-03T16:16:46.6322055Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-04-03T16:16:46.6340992Z Entering 'third_party/pthreadpool' 2024-04-03T16:16:46.6388384Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-04-03T16:16:46.6406964Z Entering 'third_party/pybind11' 2024-04-03T16:16:46.6456189Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:16:46.6475763Z Entering 'third_party/python-peachpy' 2024-04-03T16:16:46.6524784Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-04-03T16:16:46.6543967Z Entering 'third_party/sleef' 2024-04-03T16:16:46.6591323Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-04-03T16:16:46.6610258Z Entering 'third_party/tbb' 2024-04-03T16:16:46.6658270Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2024-04-03T16:16:46.6679442Z Entering 'third_party/tensorpipe' 2024-04-03T16:16:46.6728122Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-04-03T16:16:46.6747313Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:16:46.6795386Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:16:46.6814593Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:16:46.6861750Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-04-03T16:16:46.6879988Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:16:46.6927506Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-04-03T16:16:46.6946050Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:16:46.6993165Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:16:46.7010775Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:16:46.7058878Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-04-03T16:16:46.7080081Z Entering 'third_party/zstd' 2024-04-03T16:16:46.7130160Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2024-04-03T16:16:46.8142680Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-04-03T16:16:46.8462760Z Entering 'android/libs/fbjni' 2024-04-03T16:16:46.8504049Z Entering 'third_party/FP16' 2024-04-03T16:16:46.8545651Z Entering 'third_party/FXdiv' 2024-04-03T16:16:46.8587653Z Entering 'third_party/NNPACK' 2024-04-03T16:16:46.8629794Z Entering 'third_party/QNNPACK' 2024-04-03T16:16:46.8672577Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:16:46.8714173Z Entering 'third_party/XNNPACK' 2024-04-03T16:16:46.8775842Z Entering 'third_party/benchmark' 2024-04-03T16:16:46.8816980Z Entering 'third_party/cpuinfo' 2024-04-03T16:16:46.8860149Z Entering 'third_party/cub' 2024-04-03T16:16:46.8902484Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:16:46.8943875Z Entering 'third_party/cutlass' 2024-04-03T16:16:46.8993724Z Entering 'third_party/eigen' 2024-04-03T16:16:46.9037471Z Entering 'third_party/fbgemm' 2024-04-03T16:16:46.9080508Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:16:46.9120793Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:16:46.9162471Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:16:46.9209775Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:16:46.9249467Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:16:46.9290981Z Entering 'third_party/flatbuffers' 2024-04-03T16:16:46.9336854Z Entering 'third_party/fmt' 2024-04-03T16:16:46.9378490Z Entering 'third_party/foxi' 2024-04-03T16:16:46.9420180Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:16:46.9461825Z Entering 'third_party/gloo' 2024-04-03T16:16:46.9504337Z Entering 'third_party/googletest' 2024-04-03T16:16:46.9548784Z Entering 'third_party/ideep' 2024-04-03T16:16:46.9590567Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:16:46.9639731Z Entering 'third_party/ios-cmake' 2024-04-03T16:16:46.9682207Z Entering 'third_party/ittapi' 2024-04-03T16:16:46.9724169Z Entering 'third_party/kineto' 2024-04-03T16:16:46.9766253Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:16:46.9807043Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:16:46.9849038Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:16:46.9889365Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:16:46.9930291Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:16:46.9970201Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:16:47.0012499Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:16:47.0053486Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:16:47.0094698Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:16:47.0136773Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:16:47.0178540Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:16:47.0220570Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:16:47.0262592Z Entering 'third_party/mimalloc' 2024-04-03T16:16:47.0303979Z Entering 'third_party/nccl/nccl' 2024-04-03T16:16:47.0345953Z Entering 'third_party/neon2sse' 2024-04-03T16:16:47.0387276Z Entering 'third_party/nlohmann' 2024-04-03T16:16:47.0431856Z Entering 'third_party/onnx' 2024-04-03T16:16:47.0489740Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:16:47.0531297Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:16:47.0574868Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:16:47.0617568Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:16:47.0663351Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:16:47.0703652Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:16:47.0744054Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:16:47.0790555Z Entering 'third_party/pocketfft' 2024-04-03T16:16:47.0831480Z Entering 'third_party/protobuf' 2024-04-03T16:16:47.0876993Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:16:47.0917724Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:16:47.0959787Z Entering 'third_party/psimd' 2024-04-03T16:16:47.1001398Z Entering 'third_party/pthreadpool' 2024-04-03T16:16:47.1043978Z Entering 'third_party/pybind11' 2024-04-03T16:16:47.1087390Z Entering 'third_party/python-peachpy' 2024-04-03T16:16:47.1128674Z Entering 'third_party/sleef' 2024-04-03T16:16:47.1171574Z Entering 'third_party/tbb' 2024-04-03T16:16:47.1214871Z Entering 'third_party/tensorpipe' 2024-04-03T16:16:47.1256926Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:16:47.1298306Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:16:47.1339296Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:16:47.1380046Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:16:47.1419599Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:16:47.1463970Z Entering 'third_party/zstd' 2024-04-03T16:16:47.1521308Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-04-03T16:16:47.1834784Z Entering 'android/libs/fbjni' 2024-04-03T16:16:47.1876835Z Entering 'third_party/FP16' 2024-04-03T16:16:47.1918903Z Entering 'third_party/FXdiv' 2024-04-03T16:16:47.1960351Z Entering 'third_party/NNPACK' 2024-04-03T16:16:47.2002811Z Entering 'third_party/QNNPACK' 2024-04-03T16:16:47.2044740Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:16:47.2086695Z Entering 'third_party/XNNPACK' 2024-04-03T16:16:47.2147543Z Entering 'third_party/benchmark' 2024-04-03T16:16:47.2189299Z Entering 'third_party/cpuinfo' 2024-04-03T16:16:47.2232562Z Entering 'third_party/cub' 2024-04-03T16:16:47.2274664Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:16:47.2316589Z Entering 'third_party/cutlass' 2024-04-03T16:16:47.2366115Z Entering 'third_party/eigen' 2024-04-03T16:16:47.2410098Z Entering 'third_party/fbgemm' 2024-04-03T16:16:47.2452659Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:16:47.2493201Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:16:47.2536936Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:16:47.2583462Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:16:47.2623797Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:16:47.2666165Z Entering 'third_party/flatbuffers' 2024-04-03T16:16:47.2710498Z Entering 'third_party/fmt' 2024-04-03T16:16:47.2752041Z Entering 'third_party/foxi' 2024-04-03T16:16:47.2793424Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:16:47.2834908Z Entering 'third_party/gloo' 2024-04-03T16:16:47.2877788Z Entering 'third_party/googletest' 2024-04-03T16:16:47.2918780Z Entering 'third_party/ideep' 2024-04-03T16:16:47.2958745Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:16:47.3007294Z Entering 'third_party/ios-cmake' 2024-04-03T16:16:47.3048949Z Entering 'third_party/ittapi' 2024-04-03T16:16:47.3090021Z Entering 'third_party/kineto' 2024-04-03T16:16:47.3131883Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:16:47.3172213Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:16:47.3215814Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:16:47.3257806Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:16:47.3298672Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:16:47.3339432Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:16:47.3381323Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:16:47.3422257Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:16:47.3464919Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:16:47.3507408Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:16:47.3549597Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:16:47.3590371Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:16:47.3632617Z Entering 'third_party/mimalloc' 2024-04-03T16:16:47.3675446Z Entering 'third_party/nccl/nccl' 2024-04-03T16:16:47.3716753Z Entering 'third_party/neon2sse' 2024-04-03T16:16:47.3759223Z Entering 'third_party/nlohmann' 2024-04-03T16:16:47.3801956Z Entering 'third_party/onnx' 2024-04-03T16:16:47.3861965Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:16:47.3902787Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:16:47.3945834Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:16:47.3986615Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:16:47.4033055Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:16:47.4074642Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:16:47.4114726Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:16:47.4160083Z Entering 'third_party/pocketfft' 2024-04-03T16:16:47.4203723Z Entering 'third_party/protobuf' 2024-04-03T16:16:47.4248376Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:16:47.4288785Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:16:47.4332467Z Entering 'third_party/psimd' 2024-04-03T16:16:47.4373597Z Entering 'third_party/pthreadpool' 2024-04-03T16:16:47.4414763Z Entering 'third_party/pybind11' 2024-04-03T16:16:47.4456023Z Entering 'third_party/python-peachpy' 2024-04-03T16:16:47.4498609Z Entering 'third_party/sleef' 2024-04-03T16:16:47.4539855Z Entering 'third_party/tbb' 2024-04-03T16:16:47.4583125Z Entering 'third_party/tensorpipe' 2024-04-03T16:16:47.4624293Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:16:47.4664869Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:16:47.4705333Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:16:47.4746865Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:16:47.4785657Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:16:47.4829753Z Entering 'third_party/zstd' 2024-04-03T16:16:47.4884084Z ##[endgroup] 2024-04-03T16:16:47.4925628Z [command]/usr/bin/git log -1 --format='%H' 2024-04-03T16:16:47.4952426Z 'd253bd7befc64137de913dbb8727b511f1052411' 2024-04-03T16:16:47.5166204Z Prepare all required actions 2024-04-03T16:16:47.5166868Z Getting action download info 2024-04-03T16:16:47.6863688Z ##[group]Run ./.github/actions/setup-linux 2024-04-03T16:16:47.6864215Z env: 2024-04-03T16:16:47.6864540Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:47.6864976Z ##[endgroup] 2024-04-03T16:16:47.6946960Z ##[group]Run set -euo pipefail 2024-04-03T16:16:47.6947491Z set -euo pipefail 2024-04-03T16:16:47.6947984Z function get_ec2_metadata() { 2024-04-03T16:16:47.6948662Z  # Pulled from instance metadata endpoint for EC2 2024-04-03T16:16:47.6949787Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-04-03T16:16:47.6950780Z  category=$1 2024-04-03T16:16:47.6951450Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-04-03T16:16:47.6952238Z  runner_name_str=i-0ec6a6fea89e9eddd 2024-04-03T16:16:47.6952957Z  if [[ $runner_name_str != *"gcp"* ]]; then 2024-04-03T16:16:47.6953801Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-04-03T16:16:47.6954515Z  else 2024-04-03T16:16:47.6955166Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-04-03T16:16:47.6955925Z  fi 2024-04-03T16:16:47.6956262Z } 2024-04-03T16:16:47.6956706Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-04-03T16:16:47.6957439Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-04-03T16:16:47.6958247Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-04-03T16:16:47.6958962Z echo "system info $(uname -a)" 2024-04-03T16:16:47.6967852Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:47.6968479Z env: 2024-04-03T16:16:47.6968796Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:47.6969504Z ##[endgroup] 2024-04-03T16:16:47.7068584Z ami-id: ami-0ce0c36d7a00b20e2 2024-04-03T16:16:47.7133489Z instance-id: i-0ec6a6fea89e9eddd 2024-04-03T16:16:47.7196952Z instance-type: g3.8xlarge 2024-04-03T16:16:47.7205596Z system info Linux ip-10-0-38-132.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-04-03T16:16:47.7235227Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-04-03T16:16:47.7236470Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-04-03T16:16:47.7245152Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:47.7245850Z env: 2024-04-03T16:16:47.7246242Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:47.7246748Z ##[endgroup] 2024-04-03T16:16:47.7334208Z ##[group]Run if systemctl is-active --quiet docker; then 2024-04-03T16:16:47.7335016Z if systemctl is-active --quiet docker; then 2024-04-03T16:16:47.7335769Z  echo "Docker daemon is running..."; 2024-04-03T16:16:47.7336429Z else 2024-04-03T16:16:47.7337125Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-04-03T16:16:47.7337913Z fi 2024-04-03T16:16:47.7345853Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:47.7346552Z env: 2024-04-03T16:16:47.7346960Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:47.7347447Z ##[endgroup] 2024-04-03T16:16:47.7397595Z Docker daemon is running... 2024-04-03T16:16:47.7459318Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-04-03T16:16:47.7460103Z with: 2024-04-03T16:16:47.7460503Z shell: bash 2024-04-03T16:16:47.7460922Z timeout_minutes: 5 2024-04-03T16:16:47.7461395Z max_attempts: 3 2024-04-03T16:16:47.7461860Z retry_wait_seconds: 30 2024-04-03T16:16:47.7463850Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-04-03T16:16:47.7465932Z polling_interval_seconds: 1 2024-04-03T16:16:47.7466481Z warning_on_retry: true 2024-04-03T16:16:47.7466995Z continue_on_error: false 2024-04-03T16:16:47.7467468Z env: 2024-04-03T16:16:47.7467879Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:47.7468396Z AWS_RETRY_MODE: standard 2024-04-03T16:16:47.7468887Z AWS_MAX_ATTEMPTS: 5 2024-04-03T16:16:47.7469385Z AWS_DEFAULT_REGION: us-east-1 2024-04-03T16:16:47.7469927Z ##[endgroup] 2024-04-03T16:16:48.7743312Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-04-03T16:16:48.7745291Z Configure a credential helper to remove this warning. See 2024-04-03T16:16:48.7747141Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-04-03T16:16:48.7748265Z 2024-04-03T16:16:48.7748529Z Login Succeeded 2024-04-03T16:16:48.8205279Z Command completed after 1 attempt(s). 2024-04-03T16:16:48.8266388Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-04-03T16:16:48.8267343Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-04-03T16:16:48.8268219Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-04-03T16:16:48.8276884Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:48.8277522Z env: 2024-04-03T16:16:48.8277849Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:48.8278292Z ##[endgroup] 2024-04-03T16:16:48.8357696Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T16:16:48.8358630Z # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T16:16:48.8359384Z # shellcheck disable=SC2046 2024-04-03T16:16:48.8359968Z docker stop $(docker ps -q) || true 2024-04-03T16:16:48.8360559Z # Prune all of the docker images 2024-04-03T16:16:48.8361129Z docker system prune -af 2024-04-03T16:16:48.8369394Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:48.8370066Z env: 2024-04-03T16:16:48.8370400Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:48.8370840Z ##[endgroup] 2024-04-03T16:16:48.8798719Z "docker stop" requires at least 1 argument. 2024-04-03T16:16:48.8799452Z See 'docker stop --help'. 2024-04-03T16:16:48.8799760Z 2024-04-03T16:16:48.8800068Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-04-03T16:16:48.8800548Z 2024-04-03T16:16:48.8804341Z Stop one or more running containers 2024-04-03T16:16:48.9034333Z Total reclaimed space: 0B 2024-04-03T16:16:48.9073998Z ##[group]Run set +e 2024-04-03T16:16:48.9074448Z set +e 2024-04-03T16:16:48.9075015Z set -x 2024-04-03T16:16:48.9075693Z  2024-04-03T16:16:48.9076132Z PT_DOMAIN=download.pytorch.org 2024-04-03T16:16:48.9077116Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-04-03T16:16:48.9078513Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-04-03T16:16:48.9079487Z # one is returned at random 2024-04-03T16:16:48.9080159Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-04-03T16:16:48.9080810Z  2024-04-03T16:16:48.9081200Z if [ -z "${RESOLVED_IP}" ]; then 2024-04-03T16:16:48.9081970Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-04-03T16:16:48.9082934Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-04-03T16:16:48.9083644Z  2024-04-03T16:16:48.9084045Z  if [ -z "${RESOLVED_IP}" ]; then 2024-04-03T16:16:48.9084727Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-04-03T16:16:48.9085375Z  exit 1 2024-04-03T16:16:48.9085768Z  fi 2024-04-03T16:16:48.9086271Z fi 2024-04-03T16:16:48.9086880Z  2024-04-03T16:16:48.9087337Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-04-03T16:16:48.9088327Z  # Clean up any old records first 2024-04-03T16:16:48.9088987Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-04-03T16:16:48.9089833Z fi 2024-04-03T16:16:48.9090160Z  2024-04-03T16:16:48.9090698Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-04-03T16:16:48.9091391Z cat /etc/hosts 2024-04-03T16:16:48.9099484Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:48.9100095Z env: 2024-04-03T16:16:48.9100435Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:48.9100872Z ##[endgroup] 2024-04-03T16:16:48.9125881Z + PT_DOMAIN=download.pytorch.org 2024-04-03T16:16:48.9131544Z ++ dig -4 +short download.pytorch.org 2024-04-03T16:16:48.9132209Z ++ tail -n1 2024-04-03T16:16:48.9259947Z + RESOLVED_IP=108.138.64.61 2024-04-03T16:16:48.9260563Z + '[' -z 108.138.64.61 ']' 2024-04-03T16:16:48.9261056Z + grep -r download.pytorch.org /etc/hosts 2024-04-03T16:16:48.9269298Z 18.160.41.23 download.pytorch.org 2024-04-03T16:16:48.9270374Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-04-03T16:16:48.9414429Z + echo '108.138.64.61 download.pytorch.org' 2024-04-03T16:16:48.9415098Z + sudo tee -a /etc/hosts 2024-04-03T16:16:48.9705534Z 108.138.64.61 download.pytorch.org 2024-04-03T16:16:48.9721971Z + cat /etc/hosts 2024-04-03T16:16:48.9728013Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-04-03T16:16:48.9742719Z ::1 localhost6 localhost6.localdomain6 2024-04-03T16:16:48.9743358Z 108.138.64.61 download.pytorch.org 2024-04-03T16:16:48.9935568Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-04-03T16:16:48.9936375Z with: 2024-04-03T16:16:48.9937587Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:48.9939002Z docker-build-dir: .ci/docker 2024-04-03T16:16:48.9939479Z working-directory: . 2024-04-03T16:16:48.9940065Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:48.9940745Z force-push: false 2024-04-03T16:16:48.9941119Z env: 2024-04-03T16:16:48.9941432Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:48.9941865Z ##[endgroup] 2024-04-03T16:16:48.9966916Z ##[group]Run set -ex 2024-04-03T16:16:48.9967401Z set -ex 2024-04-03T16:16:48.9967771Z  2024-04-03T16:16:48.9968460Z # If the docker build directory or the build script doesn't exist, the action will 2024-04-03T16:16:48.9970125Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-04-03T16:16:48.9971142Z # job could then download the pre-built image as usual 2024-04-03T16:16:48.9972093Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-04-03T16:16:48.9972959Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-04-03T16:16:48.9973771Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:16:48.9974474Z  2024-04-03T16:16:48.9975125Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-04-03T16:16:48.9975920Z  exit 0 2024-04-03T16:16:48.9976279Z else 2024-04-03T16:16:48.9976736Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-04-03T16:16:48.9977306Z fi 2024-04-03T16:16:48.9977631Z  2024-04-03T16:16:48.9978218Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-04-03T16:16:48.9979323Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-04-03T16:16:48.9980305Z  # use it as it is, but first let's extract the tag 2024-04-03T16:16:48.9981197Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-04-03T16:16:48.9982115Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:16:48.9983006Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:16:48.9983938Z else 2024-04-03T16:16:48.9984483Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-04-03T16:16:48.9985321Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:16:48.9986440Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:16:48.9987399Z fi 2024-04-03T16:16:48.9995144Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:48.9995779Z env: 2024-04-03T16:16:48.9996104Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:48.9996551Z REPO_NAME: pytorch 2024-04-03T16:16:48.9997817Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:48.9999171Z DOCKER_BUILD_DIR: .ci/docker 2024-04-03T16:16:48.9999798Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:49.0000464Z ##[endgroup] 2024-04-03T16:16:49.0025157Z + [[ ! -d .ci/docker ]] 2024-04-03T16:16:49.0025667Z + [[ ! -f .ci/docker/build.sh ]] 2024-04-03T16:16:49.0026143Z + echo skip=false 2024-04-03T16:16:49.0028243Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-04-03T16:16:49.0033091Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:49.0034990Z ++ awk -F '[:,]' '{print $2}' 2024-04-03T16:16:49.0043512Z + DOCKER_TAG=16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:49.0044351Z + echo docker-tag=16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:49.0046273Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:49.0119610Z ##[group]Run set +e 2024-04-03T16:16:49.0120057Z set +e 2024-04-03T16:16:49.0120434Z set -x 2024-04-03T16:16:49.0120796Z  2024-04-03T16:16:49.0121098Z login() { 2024-04-03T16:16:49.0121902Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-04-03T16:16:49.0122779Z } 2024-04-03T16:16:49.0123078Z  2024-04-03T16:16:49.0123422Z retry () { 2024-04-03T16:16:49.0123972Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-04-03T16:16:49.0124515Z } 2024-04-03T16:16:49.0124852Z  2024-04-03T16:16:49.0125241Z retry login "${DOCKER_REGISTRY}" 2024-04-03T16:16:49.0125744Z  2024-04-03T16:16:49.0126334Z # Check if image already exists, if it does then skip building it 2024-04-03T16:16:49.0127226Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-04-03T16:16:49.0128290Z  exit 0 2024-04-03T16:16:49.0128689Z fi 2024-04-03T16:16:49.0129034Z  2024-04-03T16:16:49.0129909Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-04-03T16:16:49.0130964Z # be empty. The default action would be to continue rebuild the image 2024-04-03T16:16:49.0131958Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-04-03T16:16:49.0132816Z  # if we're on the base branch then use the parent commit 2024-04-03T16:16:49.0133544Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-04-03T16:16:49.0134085Z else 2024-04-03T16:16:49.0134670Z  # otherwise we're on a PR, so use the most recent base commit 2024-04-03T16:16:49.0135521Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-04-03T16:16:49.0136120Z fi 2024-04-03T16:16:49.0136449Z  2024-04-03T16:16:49.0137003Z if [[ -z "${MERGE_BASE}" ]]; then 2024-04-03T16:16:49.0137639Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-04-03T16:16:49.0138213Z  2024-04-03T16:16:49.0139527Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-04-03T16:16:49.0141100Z  exit 0 2024-04-03T16:16:49.0141477Z fi 2024-04-03T16:16:49.0141802Z  2024-04-03T16:16:49.0142356Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-04-03T16:16:49.0143603Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-04-03T16:16:49.0144621Z  exit 1 2024-04-03T16:16:49.0144974Z fi 2024-04-03T16:16:49.0145311Z  2024-04-03T16:16:49.0145918Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-04-03T16:16:49.0147109Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-04-03T16:16:49.0148225Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-04-03T16:16:49.0149456Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-04-03T16:16:49.0150855Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-04-03T16:16:49.0151665Z fi 2024-04-03T16:16:49.0152011Z  2024-04-03T16:16:49.0152603Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-04-03T16:16:49.0160462Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:49.0161068Z env: 2024-04-03T16:16:49.0161407Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:49.0161860Z DOCKER_BUILD_DIR: .ci/docker 2024-04-03T16:16:49.0162442Z BASE_REVISION: 1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:16:49.0163880Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:49.0165298Z DOCKER_TAG: 16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:49.0166029Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:49.0166683Z ##[endgroup] 2024-04-03T16:16:49.0191448Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:49.0192250Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:49.0194474Z + aws ecr get-login-password --region us-east-1 2024-04-03T16:16:49.0195418Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:49.5056827Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-04-03T16:16:49.5057899Z Configure a credential helper to remove this warning. See 2024-04-03T16:16:49.5058915Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-04-03T16:16:49.5059613Z 2024-04-03T16:16:49.5059761Z Login Succeeded 2024-04-03T16:16:49.5074764Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:49.8286275Z { 2024-04-03T16:16:49.8286924Z "schemaVersion": 2, 2024-04-03T16:16:49.8287664Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-04-03T16:16:49.8288404Z "config": { 2024-04-03T16:16:49.8288975Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-04-03T16:16:49.8290203Z "size": 44693, 2024-04-03T16:16:49.8290924Z "digest": "sha256:f0aa5113e491b8f8ef3b51fc066bce798422ecc2bdeb4be54a8a4efa6914052e" 2024-04-03T16:16:49.8291715Z }, 2024-04-03T16:16:49.8292002Z "layers": [ 2024-04-03T16:16:49.8292355Z { 2024-04-03T16:16:49.8292924Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8293600Z "size": 28580681, 2024-04-03T16:16:49.8294405Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-04-03T16:16:49.8295446Z }, 2024-04-03T16:16:49.8295755Z { 2024-04-03T16:16:49.8296306Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8296991Z "size": 7943353, 2024-04-03T16:16:49.8297693Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-04-03T16:16:49.8298488Z }, 2024-04-03T16:16:49.8298780Z { 2024-04-03T16:16:49.8299331Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8300086Z "size": 56226389, 2024-04-03T16:16:49.8300748Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-04-03T16:16:49.8301782Z }, 2024-04-03T16:16:49.8302337Z { 2024-04-03T16:16:49.8303340Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8304593Z "size": 187, 2024-04-03T16:16:49.8305270Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-04-03T16:16:49.8306092Z }, 2024-04-03T16:16:49.8306352Z { 2024-04-03T16:16:49.8306936Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8307704Z "size": 6885, 2024-04-03T16:16:49.8308813Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-04-03T16:16:49.8310470Z }, 2024-04-03T16:16:49.8310926Z { 2024-04-03T16:16:49.8311478Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8312400Z "size": 1383662475, 2024-04-03T16:16:49.8313842Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-04-03T16:16:49.8315383Z + exit 0 2024-04-03T16:16:49.8315879Z }, 2024-04-03T16:16:49.8316420Z { 2024-04-03T16:16:49.8317335Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8318541Z "size": 62282, 2024-04-03T16:16:49.8320062Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-04-03T16:16:49.8320923Z }, 2024-04-03T16:16:49.8321210Z { 2024-04-03T16:16:49.8321764Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8322475Z "size": 1684, 2024-04-03T16:16:49.8323157Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-04-03T16:16:49.8324013Z }, 2024-04-03T16:16:49.8324324Z { 2024-04-03T16:16:49.8324852Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8325558Z "size": 1522, 2024-04-03T16:16:49.8326246Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-04-03T16:16:49.8327010Z }, 2024-04-03T16:16:49.8327316Z { 2024-04-03T16:16:49.8327860Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8328549Z "size": 2445884909, 2024-04-03T16:16:49.8329684Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-04-03T16:16:49.8330688Z }, 2024-04-03T16:16:49.8330949Z { 2024-04-03T16:16:49.8331493Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8332200Z "size": 85882, 2024-04-03T16:16:49.8332868Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-04-03T16:16:49.8333647Z }, 2024-04-03T16:16:49.8333951Z { 2024-04-03T16:16:49.8334473Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8335181Z "size": 1827, 2024-04-03T16:16:49.8335886Z "digest": "sha256:020f47d0fa1be3a2df8dd2e863841d351ab72d00c7df4a4a16879ceea1239f4b" 2024-04-03T16:16:49.8336659Z }, 2024-04-03T16:16:49.8336964Z { 2024-04-03T16:16:49.8337509Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8338192Z "size": 654809190, 2024-04-03T16:16:49.8338906Z "digest": "sha256:8b29859251bec1f67480cc3935ceb13aeb54305ec53bcb4ef1d4977fcd62e2b0" 2024-04-03T16:16:49.8339897Z }, 2024-04-03T16:16:49.8340182Z { 2024-04-03T16:16:49.8340726Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8341430Z "size": 544, 2024-04-03T16:16:49.8342092Z "digest": "sha256:0e1508d7c9ab1501f0c741f4729a53eb3758856d8761c5258f732506d73c594e" 2024-04-03T16:16:49.8342859Z }, 2024-04-03T16:16:49.8343163Z { 2024-04-03T16:16:49.8343688Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8344387Z "size": 1282, 2024-04-03T16:16:49.8345079Z "digest": "sha256:89f208ed177f5a57c0375b5e3eabab8772559309b532c759b1ce8ea6cf1b3e4b" 2024-04-03T16:16:49.8345838Z }, 2024-04-03T16:16:49.8346142Z { 2024-04-03T16:16:49.8346688Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8347372Z "size": 483, 2024-04-03T16:16:49.8348072Z "digest": "sha256:ddecf600b7ac2f21cd805972b7cb9b1305ec1846970aca6b50db2c833e5b38cb" 2024-04-03T16:16:49.8348865Z }, 2024-04-03T16:16:49.8349149Z { 2024-04-03T16:16:49.8349699Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8350405Z "size": 91705074, 2024-04-03T16:16:49.8351101Z "digest": "sha256:7965ac4a7eb7caf381dde0f1bb10190588c124b84a8b1e9c77adf5143e9dc4f9" 2024-04-03T16:16:49.8351889Z }, 2024-04-03T16:16:49.8352192Z { 2024-04-03T16:16:49.8352714Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8353418Z "size": 3255, 2024-04-03T16:16:49.8354238Z "digest": "sha256:6b9458cb1b822d8bda5cc039459fec8b5d9e46187880eddb9e529319467f189b" 2024-04-03T16:16:49.8355024Z }, 2024-04-03T16:16:49.8355327Z { 2024-04-03T16:16:49.8355866Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8356543Z "size": 1904, 2024-04-03T16:16:49.8357237Z "digest": "sha256:a40a390f581ad70eca936ae2fe3aeadb7e19642cd6211232d77be76ba9fa7dd2" 2024-04-03T16:16:49.8358030Z }, 2024-04-03T16:16:49.8358315Z { 2024-04-03T16:16:49.8358859Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8359573Z "size": 699, 2024-04-03T16:16:49.8360262Z "digest": "sha256:01d4a84baddbd86b6da32efdf2e97a1088a27be6c25e024af3e1bc20f1db320e" 2024-04-03T16:16:49.8361066Z }, 2024-04-03T16:16:49.8361371Z { 2024-04-03T16:16:49.8361898Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8362598Z "size": 2803779640, 2024-04-03T16:16:49.8363320Z "digest": "sha256:db8dcb001b22779a63f46677dafc9340d330e77f96e6dcb21cd1b1210a88c0ae" 2024-04-03T16:16:49.8364100Z }, 2024-04-03T16:16:49.8364408Z { 2024-04-03T16:16:49.8364953Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8365630Z "size": 380, 2024-04-03T16:16:49.8366305Z "digest": "sha256:3eaa6199e98399217ecd8fe5727b666c8273744f0518bc3e5b1ac140ce3d123c" 2024-04-03T16:16:49.8367079Z }, 2024-04-03T16:16:49.8367363Z { 2024-04-03T16:16:49.8367905Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8368626Z "size": 11856, 2024-04-03T16:16:49.8369954Z "digest": "sha256:dd253632212f47f8e65518f3f9c4a80baeddd5f04f7bf0eb3840a976c93b5276" 2024-04-03T16:16:49.8370792Z }, 2024-04-03T16:16:49.8371080Z { 2024-04-03T16:16:49.8371620Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8372322Z "size": 802, 2024-04-03T16:16:49.8373033Z "digest": "sha256:5e6597056347c056681d9213c22a37f96446cb7b0d9b6838b5e6101ac98c7e8a" 2024-04-03T16:16:49.8373793Z }, 2024-04-03T16:16:49.8374094Z { 2024-04-03T16:16:49.8374628Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8375330Z "size": 106, 2024-04-03T16:16:49.8376015Z "digest": "sha256:858ba61782a0452f8aa8a1cef9b4fd1f301e540347dff3ed187bf69086efb8cc" 2024-04-03T16:16:49.8376773Z }, 2024-04-03T16:16:49.8377077Z { 2024-04-03T16:16:49.8377619Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8378295Z "size": 503, 2024-04-03T16:16:49.8379135Z "digest": "sha256:9cb381d49846ff5212226715691aa2c6abc9987a95b92335e81a6d03d9c4f944" 2024-04-03T16:16:49.8379907Z }, 2024-04-03T16:16:49.8380193Z { 2024-04-03T16:16:49.8380733Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8381436Z "size": 121477305, 2024-04-03T16:16:49.8382116Z "digest": "sha256:610c26b5e9a03476b891eae042a708b3462a40bc118e1e5e472f936448896531" 2024-04-03T16:16:49.8382877Z }, 2024-04-03T16:16:49.8383181Z { 2024-04-03T16:16:49.8383712Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8384415Z "size": 109, 2024-04-03T16:16:49.8385088Z "digest": "sha256:03a7067427cf481c50c8195f331a42f0ec54f72e472ea1fdf170ad664c44530a" 2024-04-03T16:16:49.8385841Z }, 2024-04-03T16:16:49.8386142Z { 2024-04-03T16:16:49.8386683Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8387359Z "size": 557, 2024-04-03T16:16:49.8388046Z "digest": "sha256:b4f17b88d23ef498cfc2839ad0358588aaca3004f277c84f3ec7d68336482c1e" 2024-04-03T16:16:49.8388922Z }, 2024-04-03T16:16:49.8389226Z { 2024-04-03T16:16:49.8389767Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8390447Z "size": 1175558, 2024-04-03T16:16:49.8391147Z "digest": "sha256:bc6bfcbe5530b217c0792229e143c80718d420f57f5e5db203ecf7be083b4d29" 2024-04-03T16:16:49.8391923Z }, 2024-04-03T16:16:49.8392207Z { 2024-04-03T16:16:49.8392877Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8393602Z "size": 103, 2024-04-03T16:16:49.8394262Z "digest": "sha256:df877d3106b553016cfe184088c5aa43dc0e564cb21a18e430d08bbdf8277f3d" 2024-04-03T16:16:49.8395047Z }, 2024-04-03T16:16:49.8395350Z { 2024-04-03T16:16:49.8395872Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8396574Z "size": 1489, 2024-04-03T16:16:49.8397264Z "digest": "sha256:0fbc7e84a570fa00c6b12df3ffc46a2b4ff2786d9e5564886359fe90b556556d" 2024-04-03T16:16:49.8398049Z }, 2024-04-03T16:16:49.8398350Z { 2024-04-03T16:16:49.8398890Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8399578Z "size": 442747152, 2024-04-03T16:16:49.8400267Z "digest": "sha256:1570829375e6e129fa627aa95c6619f8006d73f425c85d72f4595e55a0f74e0f" 2024-04-03T16:16:49.8401021Z }, 2024-04-03T16:16:49.8401306Z { 2024-04-03T16:16:49.8401846Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8402550Z "size": 161, 2024-04-03T16:16:49.8403203Z "digest": "sha256:6281fee72d3f38f69e3d04981823f93b775ef06fd399fc2308e9d5c951ec7bc1" 2024-04-03T16:16:49.8403974Z }, 2024-04-03T16:16:49.8404272Z { 2024-04-03T16:16:49.8404795Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8405499Z "size": 564, 2024-04-03T16:16:49.8406168Z "digest": "sha256:d01dc0055b877335000d5b68ac1c5e71147255853e3de4edfa032d27feff6452" 2024-04-03T16:16:49.8406927Z }, 2024-04-03T16:16:49.8407227Z { 2024-04-03T16:16:49.8407764Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8408450Z "size": 33199039, 2024-04-03T16:16:49.8409506Z "digest": "sha256:2abb072c55408b8bf96c5a0a24c729cb31029e0d42ef0192a2eba0cab079933c" 2024-04-03T16:16:49.8410372Z }, 2024-04-03T16:16:49.8410656Z { 2024-04-03T16:16:49.8411198Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8411894Z "size": 104, 2024-04-03T16:16:49.8412552Z "digest": "sha256:1323b806817860cf618c3a24ee430b0ccdd26464c11bcc46f717b6cbf241820c" 2024-04-03T16:16:49.8413322Z }, 2024-04-03T16:16:49.8413623Z { 2024-04-03T16:16:49.8414143Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8414839Z "size": 424, 2024-04-03T16:16:49.8415536Z "digest": "sha256:473ae4beebc0bc7146babdb210cabec932238e00b4a3f74f9f0bf5d7bf577593" 2024-04-03T16:16:49.8416321Z }, 2024-04-03T16:16:49.8416805Z { 2024-04-03T16:16:49.8417350Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8418035Z "size": 20262067, 2024-04-03T16:16:49.8418739Z "digest": "sha256:289a48eb62b5823f2c7d86b1875b0edd766b78c9e0edd7e1d5f54e94f004f0dc" 2024-04-03T16:16:49.8419517Z }, 2024-04-03T16:16:49.8419804Z { 2024-04-03T16:16:49.8420348Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8421045Z "size": 439, 2024-04-03T16:16:49.8421724Z "digest": "sha256:6ac1d762661f0e7800aadeafce3ba5f8d3337ad5c27ff47253ac76bbb7895841" 2024-04-03T16:16:49.8422517Z }, 2024-04-03T16:16:49.8422818Z { 2024-04-03T16:16:49.8423339Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8424095Z "size": 699, 2024-04-03T16:16:49.8424794Z "digest": "sha256:01d4a84baddbd86b6da32efdf2e97a1088a27be6c25e024af3e1bc20f1db320e" 2024-04-03T16:16:49.8425575Z }, 2024-04-03T16:16:49.8425875Z { 2024-04-03T16:16:49.8426429Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8427115Z "size": 142, 2024-04-03T16:16:49.8427783Z "digest": "sha256:796d028fd584b098f569e5427fcf37e2248f308b02af6185104c4faeeef48b05" 2024-04-03T16:16:49.8428556Z }, 2024-04-03T16:16:49.8428839Z { 2024-04-03T16:16:49.8429382Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8430081Z "size": 136, 2024-04-03T16:16:49.8430877Z "digest": "sha256:a3890e7012d35f6e60a4f6bdd11bf6bba9b20824a9bc7e4795e3f620f8eecdd5" 2024-04-03T16:16:49.8431684Z }, 2024-04-03T16:16:49.8431987Z { 2024-04-03T16:16:49.8432508Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8433207Z "size": 32, 2024-04-03T16:16:49.8433888Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:16:49.8434657Z }, 2024-04-03T16:16:49.8434959Z { 2024-04-03T16:16:49.8435500Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8436192Z "size": 188, 2024-04-03T16:16:49.8436884Z "digest": "sha256:b3d241c09291e0fe13d64493f03dbee53b27579eaa163e4d569d67ed310775f0" 2024-04-03T16:16:49.8437667Z }, 2024-04-03T16:16:49.8437950Z { 2024-04-03T16:16:49.8438496Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8439251Z "size": 564, 2024-04-03T16:16:49.8439928Z "digest": "sha256:c584774f42519400f1d08734f82c70984be83c66222c2bcf7b6d6703895f5c79" 2024-04-03T16:16:49.8440707Z }, 2024-04-03T16:16:49.8441002Z { 2024-04-03T16:16:49.8441539Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8442244Z "size": 43165822, 2024-04-03T16:16:49.8442938Z "digest": "sha256:e4b3bc5cda0494786fe54c1838bc55bce19728edf9a927c05bd5884dcbd6e205" 2024-04-03T16:16:49.8443728Z }, 2024-04-03T16:16:49.8444030Z { 2024-04-03T16:16:49.8444559Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8445269Z "size": 106, 2024-04-03T16:16:49.8445943Z "digest": "sha256:8b7274bb07ecf3280dfc69b15646503e9f67d0cf05be86b04469183960f72e64" 2024-04-03T16:16:49.8446692Z }, 2024-04-03T16:16:49.8446994Z { 2024-04-03T16:16:49.8447530Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8448212Z "size": 1192, 2024-04-03T16:16:49.8448875Z "digest": "sha256:f2934ba32628883717478385639d4c5f0403a09363b8cf2bb73b328e3b8cc042" 2024-04-03T16:16:49.8450093Z }, 2024-04-03T16:16:49.8450380Z { 2024-04-03T16:16:49.8450936Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8451637Z "size": 699, 2024-04-03T16:16:49.8452321Z "digest": "sha256:01d4a84baddbd86b6da32efdf2e97a1088a27be6c25e024af3e1bc20f1db320e" 2024-04-03T16:16:49.8453121Z }, 2024-04-03T16:16:49.8453426Z { 2024-04-03T16:16:49.8453950Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8454655Z "size": 137, 2024-04-03T16:16:49.8455516Z "digest": "sha256:d1c6f5fd2ce358717d4f731eb8c70e2757ed206be14c1f9999718a81f29d2874" 2024-04-03T16:16:49.8456280Z }, 2024-04-03T16:16:49.8456581Z { 2024-04-03T16:16:49.8457122Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8457798Z "size": 119, 2024-04-03T16:16:49.8458480Z "digest": "sha256:9c3c29d675a6f3566456d0b187f18d1a5deac38949fea55e068dbb7bc69d8bb4" 2024-04-03T16:16:49.8459263Z }, 2024-04-03T16:16:49.8459548Z { 2024-04-03T16:16:49.8460093Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8460800Z "size": 1704547419, 2024-04-03T16:16:49.8461476Z "digest": "sha256:e400461a29326092b6477a61196ef84989ebf7c7aee931714fd37259cc862d8b" 2024-04-03T16:16:49.8462234Z }, 2024-04-03T16:16:49.8462536Z { 2024-04-03T16:16:49.8463056Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8463755Z "size": 172, 2024-04-03T16:16:49.8464445Z "digest": "sha256:f3b4b7bb5d910fb4f87f578cc5c2180774bdff4c30831c40bc5425b29658682c" 2024-04-03T16:16:49.8465222Z }, 2024-04-03T16:16:49.8465521Z { 2024-04-03T16:16:49.8466060Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8466741Z "size": 1840, 2024-04-03T16:16:49.8467439Z "digest": "sha256:d39766ef1e6b8b4b6e2ae3d7d9f23f104b3c0309eefbd8bb4e5d3e362937299a" 2024-04-03T16:16:49.8468230Z }, 2024-04-03T16:16:49.8468509Z { 2024-04-03T16:16:49.8469176Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8469906Z "size": 7529780, 2024-04-03T16:16:49.8470590Z "digest": "sha256:8ce169be4b1d2efb4c8dbe9188699e646f24d62988505b7bc263e4605d20ff32" 2024-04-03T16:16:49.8471379Z }, 2024-04-03T16:16:49.8471682Z { 2024-04-03T16:16:49.8472207Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8473086Z "size": 164, 2024-04-03T16:16:49.8473774Z "digest": "sha256:fe2cd77be091e130f7e397bb03a127255f90a88edb85e9b91957619adac1d7b7" 2024-04-03T16:16:49.8474541Z }, 2024-04-03T16:16:49.8474844Z { 2024-04-03T16:16:49.8475385Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8476065Z "size": 7938, 2024-04-03T16:16:49.8476772Z "digest": "sha256:2a5bd85dd325ef3ddf1e8d9bd46e473bae5adf46aa7df529b7d898b7c4a42add" 2024-04-03T16:16:49.8477576Z }, 2024-04-03T16:16:49.8477861Z { 2024-04-03T16:16:49.8478396Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8479100Z "size": 8070, 2024-04-03T16:16:49.8479768Z "digest": "sha256:d1190269f41e7ab64b2eadaeba8697af4731d9667e9a95ac49d95c717878351f" 2024-04-03T16:16:49.8480552Z }, 2024-04-03T16:16:49.8480856Z { 2024-04-03T16:16:49.8481380Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8482086Z "size": 302, 2024-04-03T16:16:49.8482774Z "digest": "sha256:e8752c122b392a996acd66fc47ae9c18fe3f4960cbeee8bb1eeea5ab3dfbd9c2" 2024-04-03T16:16:49.8483554Z }, 2024-04-03T16:16:49.8483856Z { 2024-04-03T16:16:49.8484396Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8485079Z "size": 7620337, 2024-04-03T16:16:49.8485782Z "digest": "sha256:1eaea607f89d48ad9275ec68ec91f10e1cb3dda60bb0189f2793cef6780ad02f" 2024-04-03T16:16:49.8486568Z }, 2024-04-03T16:16:49.8486847Z { 2024-04-03T16:16:49.8487388Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8488087Z "size": 108, 2024-04-03T16:16:49.8488741Z "digest": "sha256:9ef6078991a0e9655c1656e4b3f24cdf5377a1c1ad41e4418b4a5b912856183f" 2024-04-03T16:16:49.8489923Z }, 2024-04-03T16:16:49.8490232Z { 2024-04-03T16:16:49.8490754Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8491456Z "size": 54145458, 2024-04-03T16:16:49.8492171Z "digest": "sha256:c0dab7a64f9be557be292ba18570e24ade19f386e38b36b5dfe3a5a844ef38eb" 2024-04-03T16:16:49.8492941Z }, 2024-04-03T16:16:49.8493397Z { 2024-04-03T16:16:49.8493940Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8494668Z "size": 498, 2024-04-03T16:16:49.8495349Z "digest": "sha256:20e7673e255ed89c9eca5e8f08025a96bf06263edb006590d7ff6137cf90730f" 2024-04-03T16:16:49.8496123Z }, 2024-04-03T16:16:49.8496406Z { 2024-04-03T16:16:49.8496944Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8497648Z "size": 1498859701, 2024-04-03T16:16:49.8498336Z "digest": "sha256:865db895296b5ec9e81d86e205660c33454f259b5dd7ed88f9402cbd87699077" 2024-04-03T16:16:49.8499103Z }, 2024-04-03T16:16:49.8499405Z { 2024-04-03T16:16:49.8499925Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8500622Z "size": 106, 2024-04-03T16:16:49.8501309Z "digest": "sha256:effd5b6e0122c9c96f918d64d4e1b49efb8b08c08324eea3054750d8d2b37652" 2024-04-03T16:16:49.8502079Z }, 2024-04-03T16:16:49.8502381Z { 2024-04-03T16:16:49.8502928Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8503607Z "size": 444, 2024-04-03T16:16:49.8504275Z "digest": "sha256:6150e99963233ce451f0f0e5a16b96087b4924cdbb89bb7701503081c6ca5027" 2024-04-03T16:16:49.8505033Z }, 2024-04-03T16:16:49.8505311Z { 2024-04-03T16:16:49.8505856Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8506561Z "size": 36532201, 2024-04-03T16:16:49.8507362Z "digest": "sha256:7ac5ac4b15768fbc183814207bf88e8b69cf1655e4651c06267591dad5d842d5" 2024-04-03T16:16:49.8508161Z }, 2024-04-03T16:16:49.8508462Z { 2024-04-03T16:16:49.8508983Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8509682Z "size": 111, 2024-04-03T16:16:49.8510355Z "digest": "sha256:f31a22b80d01bb61f572065e872c8276ef92389cb05b1f1b6853fda52ce462a2" 2024-04-03T16:16:49.8511110Z }, 2024-04-03T16:16:49.8511409Z { 2024-04-03T16:16:49.8511949Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8512634Z "size": 32, 2024-04-03T16:16:49.8513314Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:16:49.8514101Z }, 2024-04-03T16:16:49.8514383Z { 2024-04-03T16:16:49.8514921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8515616Z "size": 32, 2024-04-03T16:16:49.8516278Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:16:49.8517064Z }, 2024-04-03T16:16:49.8517370Z { 2024-04-03T16:16:49.8517890Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:16:49.8518590Z "size": 32, 2024-04-03T16:16:49.8519272Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:16:49.8520035Z } 2024-04-03T16:16:49.8520342Z ] 2024-04-03T16:16:49.8520643Z } 2024-04-03T16:16:49.8747017Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-04-03T16:16:49.8747779Z tag=${ECR_DOCKER_IMAGE##*/} 2024-04-03T16:16:49.8748480Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-04-03T16:16:49.8757442Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:49.8758054Z env: 2024-04-03T16:16:49.8758393Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:49.8759692Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:49.8761018Z ##[endgroup] 2024-04-03T16:16:49.8789603Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9-16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:49.8855001Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-04-03T16:16:49.8855738Z with: 2024-04-03T16:16:49.8856914Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:49.8858580Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:49.8859203Z env: 2024-04-03T16:16:49.8859536Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:49.8859965Z ##[endgroup] 2024-04-03T16:16:49.8882700Z ##[group]Run set -x 2024-04-03T16:16:49.8883117Z set -x 2024-04-03T16:16:49.8883481Z set +e 2024-04-03T16:16:49.8883835Z  2024-04-03T16:16:49.8884152Z login() { 2024-04-03T16:16:49.8884982Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-04-03T16:16:49.8885888Z } 2024-04-03T16:16:49.8886203Z  2024-04-03T16:16:49.8886597Z retry () { 2024-04-03T16:16:49.8887069Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-04-03T16:16:49.8887628Z } 2024-04-03T16:16:49.8887961Z  2024-04-03T16:16:49.8888325Z retry login "${DOCKER_REGISTRY}" 2024-04-03T16:16:49.8888839Z  2024-04-03T16:16:49.8889414Z set -e 2024-04-03T16:16:49.8889998Z # ignore output since only exit code is used for conditional 2024-04-03T16:16:49.8890882Z # only pull docker image if it's not available locally 2024-04-03T16:16:49.8891845Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-04-03T16:16:49.8892715Z  retry docker pull "${DOCKER_IMAGE}" 2024-04-03T16:16:49.8893241Z fi 2024-04-03T16:16:49.8900745Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:16:49.8901368Z env: 2024-04-03T16:16:49.8901705Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:16:49.8902960Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:49.8904442Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:49.8905090Z ##[endgroup] 2024-04-03T16:16:49.8928289Z + set +e 2024-04-03T16:16:49.8928920Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:49.8930081Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:49.8932091Z + aws ecr get-login-password --region us-east-1 2024-04-03T16:16:49.8933418Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:16:50.3755698Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-04-03T16:16:50.3756860Z Configure a credential helper to remove this warning. See 2024-04-03T16:16:50.3757969Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-04-03T16:16:50.3758687Z 2024-04-03T16:16:50.3758897Z Login Succeeded 2024-04-03T16:16:50.3772895Z + set -e 2024-04-03T16:16:50.3774449Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:50.3997511Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:50.3999921Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:16:50.6429375Z 16884143090f148aa56809ceba26eeb699b5dc49: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9 2024-04-03T16:16:50.6431309Z 7a2c55901189: Pulling fs layer 2024-04-03T16:16:50.6431928Z 09d415c238d7: Pulling fs layer 2024-04-03T16:16:50.6432717Z 9fe6e2e61518: Pulling fs layer 2024-04-03T16:16:50.6433635Z 41f16248e682: Pulling fs layer 2024-04-03T16:16:50.6434552Z 95d7b7817039: Pulling fs layer 2024-04-03T16:16:50.6435331Z 8f6c90485347: Pulling fs layer 2024-04-03T16:16:50.6435848Z ab17245097e4: Pulling fs layer 2024-04-03T16:16:50.6436404Z dfecd7e9912b: Pulling fs layer 2024-04-03T16:16:50.6437104Z 464a8f745445: Pulling fs layer 2024-04-03T16:16:50.6437780Z d67c111fa588: Pulling fs layer 2024-04-03T16:16:50.6438231Z 0723abde19c0: Pulling fs layer 2024-04-03T16:16:50.6438705Z 020f47d0fa1b: Pulling fs layer 2024-04-03T16:16:50.6439166Z 8b29859251be: Pulling fs layer 2024-04-03T16:16:50.6439612Z 0e1508d7c9ab: Pulling fs layer 2024-04-03T16:16:50.6440083Z 89f208ed177f: Pulling fs layer 2024-04-03T16:16:50.6440558Z ddecf600b7ac: Pulling fs layer 2024-04-03T16:16:50.6441014Z 7965ac4a7eb7: Pulling fs layer 2024-04-03T16:16:50.6441475Z 6b9458cb1b82: Pulling fs layer 2024-04-03T16:16:50.6441926Z 8f6c90485347: Waiting 2024-04-03T16:16:50.6442321Z a40a390f581a: Pulling fs layer 2024-04-03T16:16:50.6442857Z 01d4a84baddb: Pulling fs layer 2024-04-03T16:16:50.6443326Z ab17245097e4: Waiting 2024-04-03T16:16:50.6443740Z db8dcb001b22: Pulling fs layer 2024-04-03T16:16:50.6444168Z d67c111fa588: Waiting 2024-04-03T16:16:50.6444581Z 3eaa6199e983: Pulling fs layer 2024-04-03T16:16:50.6445037Z dfecd7e9912b: Waiting 2024-04-03T16:16:50.6445454Z dd253632212f: Pulling fs layer 2024-04-03T16:16:50.6445906Z 464a8f745445: Waiting 2024-04-03T16:16:50.6446297Z 020f47d0fa1b: Waiting 2024-04-03T16:16:50.6446686Z 5e6597056347: Pulling fs layer 2024-04-03T16:16:50.6447148Z 858ba61782a0: Pulling fs layer 2024-04-03T16:16:50.6447925Z 8b29859251be: Waiting 2024-04-03T16:16:50.6448313Z 0723abde19c0: Waiting 2024-04-03T16:16:50.6448708Z 0e1508d7c9ab: Waiting 2024-04-03T16:16:50.6449589Z 9cb381d49846: Pulling fs layer 2024-04-03T16:16:50.6450049Z 610c26b5e9a0: Pulling fs layer 2024-04-03T16:16:50.6450503Z 7965ac4a7eb7: Waiting 2024-04-03T16:16:50.6450911Z 03a7067427cf: Pulling fs layer 2024-04-03T16:16:50.6451363Z b4f17b88d23e: Pulling fs layer 2024-04-03T16:16:50.6451818Z 89f208ed177f: Waiting 2024-04-03T16:16:50.6452215Z 41f16248e682: Waiting 2024-04-03T16:16:50.6452596Z ddecf600b7ac: Waiting 2024-04-03T16:16:50.6453014Z bc6bfcbe5530: Pulling fs layer 2024-04-03T16:16:50.6453487Z df877d3106b5: Pulling fs layer 2024-04-03T16:16:50.6453918Z 95d7b7817039: Waiting 2024-04-03T16:16:50.6454306Z 6b9458cb1b82: Waiting 2024-04-03T16:16:50.6454689Z 610c26b5e9a0: Waiting 2024-04-03T16:16:50.6455100Z 0fbc7e84a570: Pulling fs layer 2024-04-03T16:16:50.6455550Z 3eaa6199e983: Waiting 2024-04-03T16:16:50.6455936Z 1570829375e6: Pulling fs layer 2024-04-03T16:16:50.6456400Z 6281fee72d3f: Pulling fs layer 2024-04-03T16:16:50.6456867Z d01dc0055b87: Pulling fs layer 2024-04-03T16:16:50.6457295Z 858ba61782a0: Waiting 2024-04-03T16:16:50.6457683Z 5e6597056347: Waiting 2024-04-03T16:16:50.6458090Z 2abb072c5540: Pulling fs layer 2024-04-03T16:16:50.6458522Z dd253632212f: Waiting 2024-04-03T16:16:50.6458950Z 9cb381d49846: Waiting 2024-04-03T16:16:50.6459355Z 1323b8068178: Pulling fs layer 2024-04-03T16:16:50.6459809Z 473ae4beebc0: Pulling fs layer 2024-04-03T16:16:50.6460277Z 289a48eb62b5: Pulling fs layer 2024-04-03T16:16:50.6460739Z 6ac1d762661f: Pulling fs layer 2024-04-03T16:16:50.6461170Z a40a390f581a: Waiting 2024-04-03T16:16:50.6461555Z 1570829375e6: Waiting 2024-04-03T16:16:50.6461948Z d01dc0055b87: Waiting 2024-04-03T16:16:50.6462332Z 2abb072c5540: Waiting 2024-04-03T16:16:50.6462722Z 473ae4beebc0: Waiting 2024-04-03T16:16:50.6463136Z 796d028fd584: Pulling fs layer 2024-04-03T16:16:50.6463566Z 1323b8068178: Waiting 2024-04-03T16:16:50.6463954Z bc6bfcbe5530: Waiting 2024-04-03T16:16:50.6464341Z 6ac1d762661f: Waiting 2024-04-03T16:16:50.6464711Z 0fbc7e84a570: Waiting 2024-04-03T16:16:50.6465106Z 01d4a84baddb: Waiting 2024-04-03T16:16:50.6465497Z 03a7067427cf: Waiting 2024-04-03T16:16:50.6465888Z a3890e7012d3: Pulling fs layer 2024-04-03T16:16:50.6466341Z b4f17b88d23e: Waiting 2024-04-03T16:16:50.6466753Z 4f4fb700ef54: Pulling fs layer 2024-04-03T16:16:50.6467241Z b3d241c09291: Pulling fs layer 2024-04-03T16:16:50.6467888Z c584774f4251: Pulling fs layer 2024-04-03T16:16:50.6468390Z e4b3bc5cda04: Pulling fs layer 2024-04-03T16:16:50.6468843Z 8b7274bb07ec: Pulling fs layer 2024-04-03T16:16:50.6469311Z f2934ba32628: Pulling fs layer 2024-04-03T16:16:50.6469777Z d1c6f5fd2ce3: Pulling fs layer 2024-04-03T16:16:50.6470229Z 9c3c29d675a6: Pulling fs layer 2024-04-03T16:16:50.6470831Z e400461a2932: Pulling fs layer 2024-04-03T16:16:50.6471273Z c584774f4251: Waiting 2024-04-03T16:16:50.6471645Z 8b7274bb07ec: Waiting 2024-04-03T16:16:50.6472060Z f3b4b7bb5d91: Pulling fs layer 2024-04-03T16:16:50.6472526Z d39766ef1e6b: Pulling fs layer 2024-04-03T16:16:50.6472953Z f2934ba32628: Waiting 2024-04-03T16:16:50.6473362Z 8ce169be4b1d: Pulling fs layer 2024-04-03T16:16:50.6473812Z a3890e7012d3: Waiting 2024-04-03T16:16:50.6474210Z fe2cd77be091: Pulling fs layer 2024-04-03T16:16:50.6474665Z e4b3bc5cda04: Waiting 2024-04-03T16:16:50.6475057Z 4f4fb700ef54: Waiting 2024-04-03T16:16:50.6475437Z d1c6f5fd2ce3: Waiting 2024-04-03T16:16:50.6475859Z 2a5bd85dd325: Pulling fs layer 2024-04-03T16:16:50.6476305Z e400461a2932: Waiting 2024-04-03T16:16:50.6476698Z d1190269f41e: Pulling fs layer 2024-04-03T16:16:50.6477139Z b3d241c09291: Waiting 2024-04-03T16:16:50.6477530Z 8ce169be4b1d: Waiting 2024-04-03T16:16:50.6477905Z 796d028fd584: Waiting 2024-04-03T16:16:50.6478322Z e8752c122b39: Pulling fs layer 2024-04-03T16:16:50.6478766Z d1190269f41e: Waiting 2024-04-03T16:16:50.6479160Z 1eaea607f89d: Pulling fs layer 2024-04-03T16:16:50.6479611Z 2a5bd85dd325: Waiting 2024-04-03T16:16:50.6480021Z 9ef6078991a0: Pulling fs layer 2024-04-03T16:16:50.6480453Z e8752c122b39: Waiting 2024-04-03T16:16:50.6480873Z c0dab7a64f9b: Pulling fs layer 2024-04-03T16:16:50.6481348Z 20e7673e255e: Pulling fs layer 2024-04-03T16:16:50.6481798Z 865db895296b: Pulling fs layer 2024-04-03T16:16:50.6482246Z 9c3c29d675a6: Waiting 2024-04-03T16:16:50.6482643Z effd5b6e0122: Pulling fs layer 2024-04-03T16:16:50.6483108Z 6150e9996323: Pulling fs layer 2024-04-03T16:16:50.6483553Z 865db895296b: Waiting 2024-04-03T16:16:50.6483955Z 7ac5ac4b1576: Pulling fs layer 2024-04-03T16:16:50.6484412Z effd5b6e0122: Waiting 2024-04-03T16:16:50.6484830Z f31a22b80d01: Pulling fs layer 2024-04-03T16:16:50.6485263Z c0dab7a64f9b: Waiting 2024-04-03T16:16:50.6485663Z 7ac5ac4b1576: Waiting 2024-04-03T16:16:50.6486059Z 9ef6078991a0: Waiting 2024-04-03T16:16:50.6486472Z 20e7673e255e: Waiting 2024-04-03T16:16:50.6486898Z f3b4b7bb5d91: Waiting 2024-04-03T16:16:50.9512423Z 09d415c238d7: Verifying Checksum 2024-04-03T16:16:50.9512975Z 09d415c238d7: Download complete 2024-04-03T16:16:51.0350499Z 41f16248e682: Verifying Checksum 2024-04-03T16:16:51.0351020Z 41f16248e682: Download complete 2024-04-03T16:16:51.1052149Z 7a2c55901189: Download complete 2024-04-03T16:16:51.1233158Z 95d7b7817039: Verifying Checksum 2024-04-03T16:16:51.1233686Z 95d7b7817039: Download complete 2024-04-03T16:16:51.2333447Z ab17245097e4: Download complete 2024-04-03T16:16:51.3488727Z dfecd7e9912b: Download complete 2024-04-03T16:16:51.4625958Z 464a8f745445: Verifying Checksum 2024-04-03T16:16:51.4626498Z 464a8f745445: Download complete 2024-04-03T16:16:51.8868752Z 7a2c55901189: Pull complete 2024-04-03T16:16:52.1752145Z 09d415c238d7: Pull complete 2024-04-03T16:16:52.5245011Z 9fe6e2e61518: Verifying Checksum 2024-04-03T16:16:52.5245579Z 9fe6e2e61518: Download complete 2024-04-03T16:16:52.6610506Z 0723abde19c0: Verifying Checksum 2024-04-03T16:16:52.6611560Z 0723abde19c0: Download complete 2024-04-03T16:16:52.7556846Z 020f47d0fa1b: Verifying Checksum 2024-04-03T16:16:52.7558146Z 020f47d0fa1b: Download complete 2024-04-03T16:16:53.5901014Z 9fe6e2e61518: Pull complete 2024-04-03T16:16:53.6979400Z 41f16248e682: Pull complete 2024-04-03T16:16:53.8313984Z 95d7b7817039: Pull complete 2024-04-03T16:17:09.0595601Z 8b29859251be: Verifying Checksum 2024-04-03T16:17:09.0596375Z 8b29859251be: Download complete 2024-04-03T16:17:09.1516758Z 0e1508d7c9ab: Verifying Checksum 2024-04-03T16:17:09.1517586Z 0e1508d7c9ab: Download complete 2024-04-03T16:17:09.2418600Z 89f208ed177f: Verifying Checksum 2024-04-03T16:17:09.2419197Z 89f208ed177f: Download complete 2024-04-03T16:17:09.3324062Z ddecf600b7ac: Download complete 2024-04-03T16:17:12.5235704Z 7965ac4a7eb7: Verifying Checksum 2024-04-03T16:17:12.5236795Z 7965ac4a7eb7: Download complete 2024-04-03T16:17:12.6093106Z 6b9458cb1b82: Download complete 2024-04-03T16:17:12.6934044Z a40a390f581a: Verifying Checksum 2024-04-03T16:17:12.6934630Z a40a390f581a: Download complete 2024-04-03T16:17:12.7890167Z 01d4a84baddb: Download complete 2024-04-03T16:17:15.7663872Z 8f6c90485347: Verifying Checksum 2024-04-03T16:17:15.7664495Z 8f6c90485347: Download complete 2024-04-03T16:17:15.8409681Z 3eaa6199e983: Verifying Checksum 2024-04-03T16:17:15.8410821Z 3eaa6199e983: Download complete 2024-04-03T16:17:15.9370018Z dd253632212f: Download complete 2024-04-03T16:17:16.0214412Z 5e6597056347: Verifying Checksum 2024-04-03T16:17:16.0215063Z 5e6597056347: Download complete 2024-04-03T16:17:16.1071443Z 858ba61782a0: Verifying Checksum 2024-04-03T16:17:16.1072066Z 858ba61782a0: Download complete 2024-04-03T16:17:16.1967896Z 9cb381d49846: Verifying Checksum 2024-04-03T16:17:16.1968637Z 9cb381d49846: Download complete 2024-04-03T16:17:19.2064011Z 610c26b5e9a0: Verifying Checksum 2024-04-03T16:17:19.2064664Z 610c26b5e9a0: Download complete 2024-04-03T16:17:19.3003952Z 03a7067427cf: Verifying Checksum 2024-04-03T16:17:19.3004701Z 03a7067427cf: Download complete 2024-04-03T16:17:19.3886591Z b4f17b88d23e: Download complete 2024-04-03T16:17:19.6063611Z bc6bfcbe5530: Download complete 2024-04-03T16:17:19.6796685Z df877d3106b5: Verifying Checksum 2024-04-03T16:17:19.6797264Z df877d3106b5: Download complete 2024-04-03T16:17:19.7585540Z 0fbc7e84a570: Verifying Checksum 2024-04-03T16:17:19.7586033Z 0fbc7e84a570: Download complete 2024-04-03T16:17:30.3229391Z 1570829375e6: Verifying Checksum 2024-04-03T16:17:30.4244155Z 6281fee72d3f: Verifying Checksum 2024-04-03T16:17:30.4244890Z 6281fee72d3f: Download complete 2024-04-03T16:17:30.5147187Z d01dc0055b87: Verifying Checksum 2024-04-03T16:17:31.2991710Z 2abb072c5540: Verifying Checksum 2024-04-03T16:17:31.2992313Z 2abb072c5540: Download complete 2024-04-03T16:17:31.3983336Z 1323b8068178: Download complete 2024-04-03T16:17:31.4970519Z 473ae4beebc0: Download complete 2024-04-03T16:17:32.0011340Z d67c111fa588: Verifying Checksum 2024-04-03T16:17:32.0011935Z d67c111fa588: Download complete 2024-04-03T16:17:32.0409744Z 289a48eb62b5: Verifying Checksum 2024-04-03T16:17:32.0410298Z 289a48eb62b5: Download complete 2024-04-03T16:17:32.1162230Z 6ac1d762661f: Verifying Checksum 2024-04-03T16:17:32.1162770Z 6ac1d762661f: Download complete 2024-04-03T16:17:32.1250443Z 796d028fd584: Verifying Checksum 2024-04-03T16:17:32.1251220Z 796d028fd584: Download complete 2024-04-03T16:17:32.1358507Z 4f4fb700ef54: Verifying Checksum 2024-04-03T16:17:32.1359031Z 4f4fb700ef54: Download complete 2024-04-03T16:17:32.2134531Z a3890e7012d3: Verifying Checksum 2024-04-03T16:17:32.2135083Z a3890e7012d3: Download complete 2024-04-03T16:17:32.2335155Z b3d241c09291: Download complete 2024-04-03T16:17:32.3003003Z c584774f4251: Download complete 2024-04-03T16:17:32.4173626Z 8b7274bb07ec: Download complete 2024-04-03T16:17:32.5275941Z f2934ba32628: Verifying Checksum 2024-04-03T16:17:32.5276558Z f2934ba32628: Download complete 2024-04-03T16:17:32.6513046Z d1c6f5fd2ce3: Download complete 2024-04-03T16:17:32.7385941Z 9c3c29d675a6: Verifying Checksum 2024-04-03T16:17:32.7386535Z 9c3c29d675a6: Download complete 2024-04-03T16:17:33.4818413Z e4b3bc5cda04: Verifying Checksum 2024-04-03T16:17:33.4819121Z e4b3bc5cda04: Download complete 2024-04-03T16:17:33.5650223Z f3b4b7bb5d91: Verifying Checksum 2024-04-03T16:17:33.5650786Z f3b4b7bb5d91: Download complete 2024-04-03T16:17:33.6693244Z d39766ef1e6b: Download complete 2024-04-03T16:17:34.0216846Z 8ce169be4b1d: Verifying Checksum 2024-04-03T16:17:34.0217406Z 8ce169be4b1d: Download complete 2024-04-03T16:17:34.1802945Z fe2cd77be091: Verifying Checksum 2024-04-03T16:17:34.1803596Z fe2cd77be091: Download complete 2024-04-03T16:17:34.2624521Z 2a5bd85dd325: Verifying Checksum 2024-04-03T16:17:34.2625176Z 2a5bd85dd325: Download complete 2024-04-03T16:17:34.3616809Z d1190269f41e: Verifying Checksum 2024-04-03T16:17:34.3617366Z d1190269f41e: Download complete 2024-04-03T16:17:34.4872575Z e8752c122b39: Verifying Checksum 2024-04-03T16:17:34.4873063Z e8752c122b39: Download complete 2024-04-03T16:17:34.8378476Z 1eaea607f89d: Verifying Checksum 2024-04-03T16:17:34.8379098Z 1eaea607f89d: Download complete 2024-04-03T16:17:34.9770193Z 9ef6078991a0: Verifying Checksum 2024-04-03T16:17:34.9770701Z 9ef6078991a0: Download complete 2024-04-03T16:17:35.3820820Z 8f6c90485347: Pull complete 2024-04-03T16:17:36.6918179Z c0dab7a64f9b: Verifying Checksum 2024-04-03T16:17:36.6918786Z c0dab7a64f9b: Download complete 2024-04-03T16:17:36.9080284Z 20e7673e255e: Verifying Checksum 2024-04-03T16:17:36.9080863Z 20e7673e255e: Download complete 2024-04-03T16:17:37.2296570Z ab17245097e4: Pull complete 2024-04-03T16:17:39.0778895Z dfecd7e9912b: Pull complete 2024-04-03T16:17:40.9558680Z 464a8f745445: Pull complete 2024-04-03T16:18:12.7761639Z 865db895296b: Verifying Checksum 2024-04-03T16:18:12.7762365Z 865db895296b: Download complete 2024-04-03T16:18:12.8792182Z effd5b6e0122: Verifying Checksum 2024-04-03T16:18:12.8792798Z effd5b6e0122: Download complete 2024-04-03T16:18:12.9755149Z 6150e9996323: Verifying Checksum 2024-04-03T16:18:12.9755717Z 6150e9996323: Download complete 2024-04-03T16:18:14.0861499Z 7ac5ac4b1576: Verifying Checksum 2024-04-03T16:18:14.0862613Z 7ac5ac4b1576: Download complete 2024-04-03T16:18:14.1807860Z f31a22b80d01: Verifying Checksum 2024-04-03T16:18:14.1808428Z f31a22b80d01: Download complete 2024-04-03T16:18:15.1818117Z d67c111fa588: Pull complete 2024-04-03T16:18:16.3375554Z e400461a2932: Verifying Checksum 2024-04-03T16:18:17.0653907Z 0723abde19c0: Pull complete 2024-04-03T16:18:19.1541073Z 020f47d0fa1b: Pull complete 2024-04-03T16:18:20.2766686Z db8dcb001b22: Verifying Checksum 2024-04-03T16:18:20.2767320Z db8dcb001b22: Download complete 2024-04-03T16:18:32.3010629Z 8b29859251be: Pull complete 2024-04-03T16:18:34.6170040Z 0e1508d7c9ab: Pull complete 2024-04-03T16:18:36.9134187Z 89f208ed177f: Pull complete 2024-04-03T16:18:38.9625645Z ddecf600b7ac: Pull complete 2024-04-03T16:18:43.4482414Z 7965ac4a7eb7: Pull complete 2024-04-03T16:18:45.3258439Z 6b9458cb1b82: Pull complete 2024-04-03T16:18:47.1182909Z a40a390f581a: Pull complete 2024-04-03T16:18:49.0912267Z 01d4a84baddb: Pull complete 2024-04-03T16:19:36.9364701Z db8dcb001b22: Pull complete 2024-04-03T16:19:38.7389089Z 3eaa6199e983: Pull complete 2024-04-03T16:19:40.7112282Z dd253632212f: Pull complete 2024-04-03T16:19:42.5602718Z 5e6597056347: Pull complete 2024-04-03T16:19:44.4299374Z 858ba61782a0: Pull complete 2024-04-03T16:19:46.3080105Z 9cb381d49846: Pull complete 2024-04-03T16:19:52.0044575Z 610c26b5e9a0: Pull complete 2024-04-03T16:19:52.1114980Z 03a7067427cf: Pull complete 2024-04-03T16:19:52.2160250Z b4f17b88d23e: Pull complete 2024-04-03T16:19:52.3612470Z bc6bfcbe5530: Pull complete 2024-04-03T16:19:52.4676114Z df877d3106b5: Pull complete 2024-04-03T16:19:52.6099486Z 0fbc7e84a570: Pull complete 2024-04-03T16:20:01.2605648Z 1570829375e6: Pull complete 2024-04-03T16:20:03.2143354Z 6281fee72d3f: Pull complete 2024-04-03T16:20:06.3299494Z d01dc0055b87: Pull complete 2024-04-03T16:20:10.0744983Z 2abb072c5540: Pull complete 2024-04-03T16:20:12.4130463Z 1323b8068178: Pull complete 2024-04-03T16:20:15.3083691Z 473ae4beebc0: Pull complete 2024-04-03T16:20:17.7050683Z 289a48eb62b5: Pull complete 2024-04-03T16:20:19.6442319Z 6ac1d762661f: Pull complete 2024-04-03T16:20:23.8718347Z 796d028fd584: Pull complete 2024-04-03T16:20:25.8295716Z a3890e7012d3: Pull complete 2024-04-03T16:20:27.8063829Z 4f4fb700ef54: Pull complete 2024-04-03T16:20:29.8468861Z b3d241c09291: Pull complete 2024-04-03T16:20:32.3066907Z c584774f4251: Pull complete 2024-04-03T16:20:37.4296377Z e4b3bc5cda04: Pull complete 2024-04-03T16:20:39.3714736Z 8b7274bb07ec: Pull complete 2024-04-03T16:20:41.4255862Z f2934ba32628: Pull complete 2024-04-03T16:20:43.9350904Z d1c6f5fd2ce3: Pull complete 2024-04-03T16:20:44.0541827Z 9c3c29d675a6: Pull complete 2024-04-03T16:21:17.2172631Z e400461a2932: Pull complete 2024-04-03T16:21:19.3946310Z f3b4b7bb5d91: Pull complete 2024-04-03T16:21:21.2736746Z d39766ef1e6b: Pull complete 2024-04-03T16:21:23.2015226Z 8ce169be4b1d: Pull complete 2024-04-03T16:21:25.0780305Z fe2cd77be091: Pull complete 2024-04-03T16:21:26.9479642Z 2a5bd85dd325: Pull complete 2024-04-03T16:21:28.8232978Z d1190269f41e: Pull complete 2024-04-03T16:21:30.6684017Z e8752c122b39: Pull complete 2024-04-03T16:21:33.5557336Z 1eaea607f89d: Pull complete 2024-04-03T16:21:35.3952368Z 9ef6078991a0: Pull complete 2024-04-03T16:21:39.4376925Z c0dab7a64f9b: Pull complete 2024-04-03T16:21:41.2857762Z 20e7673e255e: Pull complete 2024-04-03T16:22:00.6988423Z 865db895296b: Pull complete 2024-04-03T16:22:00.8150423Z effd5b6e0122: Pull complete 2024-04-03T16:22:00.9142040Z 6150e9996323: Pull complete 2024-04-03T16:22:01.5833607Z 7ac5ac4b1576: Pull complete 2024-04-03T16:22:01.6790032Z f31a22b80d01: Pull complete 2024-04-03T16:22:01.9946257Z Digest: sha256:68e0ca2e38253fb3bd10c84d380560df0602da1b16bccdfd7d4dd7ac22529759 2024-04-03T16:22:01.9990315Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:22:02.0030055Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:22:02.0219490Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-04-03T16:22:02.0220212Z with: 2024-04-03T16:22:02.0220539Z driver-version: 535.54.03 2024-04-03T16:22:02.0220967Z env: 2024-04-03T16:22:02.0221300Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:02.0221714Z ##[endgroup] 2024-04-03T16:22:02.0275919Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-04-03T16:22:02.0276626Z with: 2024-04-03T16:22:02.0276950Z timeout_minutes: 10 2024-04-03T16:22:02.0277358Z max_attempts: 3 2024-04-03T16:22:02.0318042Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-04-03T16:22:02.0358494Z retry_wait_seconds: 10 2024-04-03T16:22:02.0358955Z polling_interval_seconds: 1 2024-04-03T16:22:02.0359406Z warning_on_retry: true 2024-04-03T16:22:02.0359846Z continue_on_error: false 2024-04-03T16:22:02.0360262Z env: 2024-04-03T16:22:02.0360574Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:02.0361034Z DRIVER_VERSION: 535.54.03 2024-04-03T16:22:02.0361475Z ##[endgroup] 2024-04-03T16:22:02.1129876Z == Installing nvidia driver NVIDIA-Linux-x86_64-535.54.03.run == 2024-04-03T16:22:02.1131579Z + pre_install_nvidia_driver_amzn2 2024-04-03T16:22:02.1132352Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-04-03T16:22:02.6331238Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-04-03T16:22:02.6915969Z No Match for argument: nvidia-driver-latest-dkms 2024-04-03T16:22:02.7293723Z No Packages marked for removal 2024-04-03T16:22:02.7458577Z + install_nvidia_driver_common 2024-04-03T16:22:02.7462604Z + echo 'Before installing NVIDIA driver' 2024-04-03T16:22:02.7463599Z + lspci 2024-04-03T16:22:02.7466655Z Before installing NVIDIA driver 2024-04-03T16:22:02.7641083Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-04-03T16:22:02.7643099Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-04-03T16:22:02.7644352Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-04-03T16:22:02.7645581Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-04-03T16:22:02.7646466Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-04-03T16:22:02.7647354Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-04-03T16:22:02.7648422Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-04-03T16:22:02.7650162Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-04-03T16:22:02.7651391Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-04-03T16:22:02.7652139Z + lsmod 2024-04-03T16:22:02.7665413Z Module Size Used by 2024-04-03T16:22:02.7666431Z ib_core 266240 0 2024-04-03T16:22:02.7666905Z nvidia_modeset 1306624 0 2024-04-03T16:22:02.7667316Z veth 16384 0 2024-04-03T16:22:02.7667802Z nvidia_uvm 1466368 0 2024-04-03T16:22:02.7668411Z nvidia 56422400 2 nvidia_uvm,nvidia_modeset 2024-04-03T16:22:02.7669034Z drm 421888 1 nvidia 2024-04-03T16:22:02.7669781Z i2c_core 77824 2 nvidia,drm 2024-04-03T16:22:02.7670426Z backlight 16384 1 nvidia_modeset 2024-04-03T16:22:02.7670961Z xt_conntrack 16384 1 2024-04-03T16:22:02.7671430Z ipt_MASQUERADE 16384 1 2024-04-03T16:22:02.7671990Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-04-03T16:22:02.7672626Z nf_conntrack_netlink 49152 0 2024-04-03T16:22:02.7673142Z nfnetlink 16384 2 nf_conntrack_netlink 2024-04-03T16:22:02.7673709Z xfrm_user 45056 1 2024-04-03T16:22:02.7674257Z xfrm_algo 16384 1 xfrm_user 2024-04-03T16:22:02.7674789Z iptable_nat 16384 1 2024-04-03T16:22:02.7675536Z nf_conntrack_ipv4 16384 3 2024-04-03T16:22:02.7676582Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-04-03T16:22:02.7677787Z nf_nat_ipv4 16384 1 iptable_nat 2024-04-03T16:22:02.7678516Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-04-03T16:22:02.7679652Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-04-03T16:22:02.7681522Z xt_addrtype 16384 2 2024-04-03T16:22:02.7682390Z iptable_filter 16384 1 2024-04-03T16:22:02.7683310Z br_netfilter 24576 0 2024-04-03T16:22:02.7684308Z bridge 172032 1 br_netfilter 2024-04-03T16:22:02.7685433Z stp 16384 1 bridge 2024-04-03T16:22:02.7686443Z llc 16384 2 bridge,stp 2024-04-03T16:22:02.7687461Z overlay 86016 0 2024-04-03T16:22:02.7688200Z sunrpc 393216 1 2024-04-03T16:22:02.7688681Z dm_mirror 28672 0 2024-04-03T16:22:02.7689743Z dm_region_hash 20480 1 dm_mirror 2024-04-03T16:22:02.7690556Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-04-03T16:22:02.7691218Z dm_mod 143360 2 dm_log,dm_mirror 2024-04-03T16:22:02.7691778Z dax 69632 1 dm_mod 2024-04-03T16:22:02.7692268Z sb_edac 24576 0 2024-04-03T16:22:02.7692737Z crc32_pclmul 16384 0 2024-04-03T16:22:02.7693210Z ghash_clmulni_intel 16384 0 2024-04-03T16:22:02.7693667Z pcbc 16384 0 2024-04-03T16:22:02.7694137Z aesni_intel 188416 0 2024-04-03T16:22:02.7694640Z aes_x86_64 20480 1 aesni_intel 2024-04-03T16:22:02.7695186Z crypto_simd 16384 1 aesni_intel 2024-04-03T16:22:02.7695748Z glue_helper 16384 1 aesni_intel 2024-04-03T16:22:02.7696282Z mousedev 24576 0 2024-04-03T16:22:02.7696899Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-04-03T16:22:02.7697759Z pcc_cpufreq 16384 0 2024-04-03T16:22:02.7698336Z psmouse 32768 0 2024-04-03T16:22:02.7698680Z evdev 20480 3 2024-04-03T16:22:02.7699484Z button 16384 0 2024-04-03T16:22:02.7700024Z ena 139264 0 2024-04-03T16:22:02.7700386Z ptp 20480 1 ena 2024-04-03T16:22:02.7700888Z pps_core 20480 1 ptp 2024-04-03T16:22:02.7701386Z ata_piix 36864 0 2024-04-03T16:22:02.7701835Z xen_blkfront 49152 2 2024-04-03T16:22:02.7702323Z libata 266240 1 ata_piix 2024-04-03T16:22:02.7702840Z crc32c_intel 24576 0 2024-04-03T16:22:02.7703297Z scsi_mod 245760 1 libata 2024-04-03T16:22:02.7703795Z autofs4 49152 2 2024-04-03T16:22:02.7704243Z + modinfo nvidia 2024-04-03T16:22:02.7705070Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-04-03T16:22:02.7705952Z firmware: nvidia/535.54.03/gsp_tu10x.bin 2024-04-03T16:22:02.7706558Z firmware: nvidia/535.54.03/gsp_ga10x.bin 2024-04-03T16:22:02.7707161Z alias: char-major-195-* 2024-04-03T16:22:02.7707653Z version: 535.54.03 2024-04-03T16:22:02.7708085Z supported: external 2024-04-03T16:22:02.7708490Z license: NVIDIA 2024-04-03T16:22:02.7709084Z srcversion: EA9C7EF32617E104C8240C4 2024-04-03T16:22:02.7709703Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-04-03T16:22:02.7710316Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-04-03T16:22:02.7710937Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-04-03T16:22:02.7711556Z depends: i2c-core,drm 2024-04-03T16:22:02.7711987Z retpoline: Y 2024-04-03T16:22:02.7712367Z name: nvidia 2024-04-03T16:22:02.7713062Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-04-03T16:22:02.7713874Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-04-03T16:22:02.7714696Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-04-03T16:22:02.7715448Z parm: NVreg_ResmanDebugLevel:int 2024-04-03T16:22:02.7716001Z parm: NVreg_RmLogonRC:int 2024-04-03T16:22:02.7716588Z parm: NVreg_ModifyDeviceFiles:int 2024-04-03T16:22:02.7717163Z parm: NVreg_DeviceFileUID:int 2024-04-03T16:22:02.7717713Z parm: NVreg_DeviceFileGID:int 2024-04-03T16:22:02.7718254Z parm: NVreg_DeviceFileMode:int 2024-04-03T16:22:02.7718919Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-04-03T16:22:02.7719630Z parm: NVreg_UsePageAttributeTable:int 2024-04-03T16:22:02.7720222Z parm: NVreg_EnablePCIeGen3:int 2024-04-03T16:22:02.7720763Z parm: NVreg_EnableMSI:int 2024-04-03T16:22:02.7721286Z parm: NVreg_TCEBypassMode:int 2024-04-03T16:22:02.7721841Z parm: NVreg_EnableStreamMemOPs:int 2024-04-03T16:22:02.7722510Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-04-03T16:22:02.7723249Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-04-03T16:22:02.7723941Z parm: NVreg_EnableS0ixPowerManagement:int 2024-04-03T16:22:02.7724725Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-04-03T16:22:02.7725486Z parm: NVreg_DynamicPowerManagement:int 2024-04-03T16:22:02.7726250Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-04-03T16:22:02.7727004Z parm: NVreg_EnableGpuFirmware:int 2024-04-03T16:22:02.7727623Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-04-03T16:22:02.7728279Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-04-03T16:22:02.7728967Z parm: NVreg_EnableUserNUMAManagement:int 2024-04-03T16:22:02.7730343Z parm: NVreg_MemoryPoolSize:int 2024-04-03T16:22:02.7730918Z parm: NVreg_KMallocHeapMaxSize:int 2024-04-03T16:22:02.7731527Z parm: NVreg_VMallocHeapMaxSize:int 2024-04-03T16:22:02.7732125Z parm: NVreg_IgnoreMMIOCheck:int 2024-04-03T16:22:02.7732669Z parm: NVreg_NvLinkDisable:int 2024-04-03T16:22:02.7733303Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-04-03T16:22:02.7734153Z parm: NVreg_RegisterPCIDriver:int 2024-04-03T16:22:02.7734735Z parm: NVreg_EnableResizableBar:int 2024-04-03T16:22:02.7735346Z parm: NVreg_EnableDbgBreakpoint:int 2024-04-03T16:22:02.7735957Z parm: NVreg_RegistryDwords:charp 2024-04-03T16:22:02.7736590Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-04-03T16:22:02.7737178Z parm: NVreg_RmMsg:charp 2024-04-03T16:22:02.7737696Z parm: NVreg_GpuBlacklist:charp 2024-04-03T16:22:02.7738288Z parm: NVreg_TemporaryFilePath:charp 2024-04-03T16:22:02.7738862Z parm: NVreg_ExcludedGpus:charp 2024-04-03T16:22:02.7739432Z parm: NVreg_DmaRemapPeerMmio:int 2024-04-03T16:22:02.7740031Z parm: NVreg_RmNvlinkBandwidth:charp 2024-04-03T16:22:02.7740599Z parm: rm_firmware_active:charp 2024-04-03T16:22:02.7741118Z + HAS_NVIDIA_DRIVER=0 2024-04-03T16:22:02.7741600Z ++ command -v nvidia-smi 2024-04-03T16:22:02.7742075Z + '[' -x /usr/bin/nvidia-smi ']' 2024-04-03T16:22:02.7742546Z + set +e 2024-04-03T16:22:02.7743197Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-04-03T16:22:05.1004752Z + INSTALLED_DRIVER_VERSION=535.54.03 2024-04-03T16:22:05.1006578Z + NVIDIA_SMI_STATUS=0 2024-04-03T16:22:05.1007431Z + '[' 0 -ne 0 ']' 2024-04-03T16:22:05.1007886Z + '[' 535.54.03 '!=' 535.54.03 ']' 2024-04-03T16:22:05.1008361Z + HAS_NVIDIA_DRIVER=1 2024-04-03T16:22:05.1009997Z + echo 'NVIDIA driver (535.54.03) has already been installed. Skipping NVIDIA driver installation' 2024-04-03T16:22:05.1011294Z + set -e 2024-04-03T16:22:05.1011670Z + '[' 1 -eq 0 ']' 2024-04-03T16:22:05.1012377Z NVIDIA driver (535.54.03) has already been installed. Skipping NVIDIA driver installation 2024-04-03T16:22:05.1013265Z + post_install_nvidia_driver_common 2024-04-03T16:22:05.1013772Z + sudo modprobe nvidia 2024-04-03T16:22:05.1158173Z + echo 'After installing NVIDIA driver' 2024-04-03T16:22:05.1159216Z + lspci 2024-04-03T16:22:05.1160427Z After installing NVIDIA driver 2024-04-03T16:22:05.1358518Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-04-03T16:22:05.1360275Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-04-03T16:22:05.1361276Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-04-03T16:22:05.1362228Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-04-03T16:22:05.1363030Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-04-03T16:22:05.1363899Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-04-03T16:22:05.1364936Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-04-03T16:22:05.1365993Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-04-03T16:22:05.1367048Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-04-03T16:22:05.1367790Z + lsmod 2024-04-03T16:22:05.1382275Z Module Size Used by 2024-04-03T16:22:05.1383302Z ib_core 266240 0 2024-04-03T16:22:05.1384317Z nvidia_modeset 1306624 0 2024-04-03T16:22:05.1385013Z veth 16384 0 2024-04-03T16:22:05.1385500Z nvidia_uvm 1466368 0 2024-04-03T16:22:05.1385993Z nvidia 56422400 2 nvidia_uvm,nvidia_modeset 2024-04-03T16:22:05.1386643Z drm 421888 1 nvidia 2024-04-03T16:22:05.1387173Z i2c_core 77824 2 nvidia,drm 2024-04-03T16:22:05.1387767Z backlight 16384 1 nvidia_modeset 2024-04-03T16:22:05.1388322Z xt_conntrack 16384 1 2024-04-03T16:22:05.1388795Z ipt_MASQUERADE 16384 1 2024-04-03T16:22:05.1389273Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-04-03T16:22:05.1390247Z nf_conntrack_netlink 49152 0 2024-04-03T16:22:05.1391148Z nfnetlink 16384 2 nf_conntrack_netlink 2024-04-03T16:22:05.1391994Z xfrm_user 45056 1 2024-04-03T16:22:05.1393051Z xfrm_algo 16384 1 xfrm_user 2024-04-03T16:22:05.1394408Z iptable_nat 16384 1 2024-04-03T16:22:05.1395354Z nf_conntrack_ipv4 16384 3 2024-04-03T16:22:05.1396511Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-04-03T16:22:05.1397771Z nf_nat_ipv4 16384 1 iptable_nat 2024-04-03T16:22:05.1399168Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-04-03T16:22:05.1400824Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-04-03T16:22:05.1401897Z xt_addrtype 16384 2 2024-04-03T16:22:05.1402359Z iptable_filter 16384 1 2024-04-03T16:22:05.1402833Z br_netfilter 24576 0 2024-04-03T16:22:05.1403350Z bridge 172032 1 br_netfilter 2024-04-03T16:22:05.1403884Z stp 16384 1 bridge 2024-04-03T16:22:05.1404410Z llc 16384 2 bridge,stp 2024-04-03T16:22:05.1404933Z overlay 86016 0 2024-04-03T16:22:05.1405434Z sunrpc 393216 1 2024-04-03T16:22:05.1405938Z dm_mirror 28672 0 2024-04-03T16:22:05.1406474Z dm_region_hash 20480 1 dm_mirror 2024-04-03T16:22:05.1407304Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-04-03T16:22:05.1408097Z dm_mod 143360 2 dm_log,dm_mirror 2024-04-03T16:22:05.1408673Z dax 69632 1 dm_mod 2024-04-03T16:22:05.1409537Z sb_edac 24576 0 2024-04-03T16:22:05.1410016Z crc32_pclmul 16384 0 2024-04-03T16:22:05.1410493Z ghash_clmulni_intel 16384 0 2024-04-03T16:22:05.1410947Z pcbc 16384 0 2024-04-03T16:22:05.1411408Z aesni_intel 188416 0 2024-04-03T16:22:05.1411902Z aes_x86_64 20480 1 aesni_intel 2024-04-03T16:22:05.1412446Z crypto_simd 16384 1 aesni_intel 2024-04-03T16:22:05.1413011Z glue_helper 16384 1 aesni_intel 2024-04-03T16:22:05.1413546Z mousedev 24576 0 2024-04-03T16:22:05.1414175Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-04-03T16:22:05.1414864Z pcc_cpufreq 16384 0 2024-04-03T16:22:05.1415336Z psmouse 32768 0 2024-04-03T16:22:05.1415784Z evdev 20480 3 2024-04-03T16:22:05.1416244Z button 16384 0 2024-04-03T16:22:05.1416747Z ena 139264 0 2024-04-03T16:22:05.1417190Z ptp 20480 1 ena 2024-04-03T16:22:05.1417685Z pps_core 20480 1 ptp 2024-04-03T16:22:05.1418177Z ata_piix 36864 0 2024-04-03T16:22:05.1418620Z xen_blkfront 49152 2 2024-04-03T16:22:05.1419101Z libata 266240 1 ata_piix 2024-04-03T16:22:05.1419613Z crc32c_intel 24576 0 2024-04-03T16:22:05.1420071Z scsi_mod 245760 1 libata 2024-04-03T16:22:05.1420569Z autofs4 49152 2 2024-04-03T16:22:05.1421022Z + modinfo nvidia 2024-04-03T16:22:05.1421837Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-04-03T16:22:05.1422727Z firmware: nvidia/535.54.03/gsp_tu10x.bin 2024-04-03T16:22:05.1423336Z firmware: nvidia/535.54.03/gsp_ga10x.bin 2024-04-03T16:22:05.1423942Z alias: char-major-195-* 2024-04-03T16:22:05.1424422Z version: 535.54.03 2024-04-03T16:22:05.1424852Z supported: external 2024-04-03T16:22:05.1425260Z license: NVIDIA 2024-04-03T16:22:05.1425709Z srcversion: EA9C7EF32617E104C8240C4 2024-04-03T16:22:05.1426305Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-04-03T16:22:05.1426919Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-04-03T16:22:05.1427538Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-04-03T16:22:05.1428152Z depends: i2c-core,drm 2024-04-03T16:22:05.1428601Z retpoline: Y 2024-04-03T16:22:05.1428969Z name: nvidia 2024-04-03T16:22:05.1429668Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-04-03T16:22:05.1430493Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-04-03T16:22:05.1431467Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-04-03T16:22:05.1432226Z parm: NVreg_ResmanDebugLevel:int 2024-04-03T16:22:05.1432783Z parm: NVreg_RmLogonRC:int 2024-04-03T16:22:05.1433313Z parm: NVreg_ModifyDeviceFiles:int 2024-04-03T16:22:05.1433885Z parm: NVreg_DeviceFileUID:int 2024-04-03T16:22:05.1434426Z parm: NVreg_DeviceFileGID:int 2024-04-03T16:22:05.1434954Z parm: NVreg_DeviceFileMode:int 2024-04-03T16:22:05.1435620Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-04-03T16:22:05.1436333Z parm: NVreg_UsePageAttributeTable:int 2024-04-03T16:22:05.1436923Z parm: NVreg_EnablePCIeGen3:int 2024-04-03T16:22:05.1437463Z parm: NVreg_EnableMSI:int 2024-04-03T16:22:05.1437986Z parm: NVreg_TCEBypassMode:int 2024-04-03T16:22:05.1438568Z parm: NVreg_EnableStreamMemOPs:int 2024-04-03T16:22:05.1439248Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-04-03T16:22:05.1439984Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-04-03T16:22:05.1440796Z parm: NVreg_EnableS0ixPowerManagement:int 2024-04-03T16:22:05.1441582Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-04-03T16:22:05.1442338Z parm: NVreg_DynamicPowerManagement:int 2024-04-03T16:22:05.1443090Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-04-03T16:22:05.1443844Z parm: NVreg_EnableGpuFirmware:int 2024-04-03T16:22:05.1444465Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-04-03T16:22:05.1445137Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-04-03T16:22:05.1445811Z parm: NVreg_EnableUserNUMAManagement:int 2024-04-03T16:22:05.1446442Z parm: NVreg_MemoryPoolSize:int 2024-04-03T16:22:05.1447021Z parm: NVreg_KMallocHeapMaxSize:int 2024-04-03T16:22:05.1447607Z parm: NVreg_VMallocHeapMaxSize:int 2024-04-03T16:22:05.1448210Z parm: NVreg_IgnoreMMIOCheck:int 2024-04-03T16:22:05.1448765Z parm: NVreg_NvLinkDisable:int 2024-04-03T16:22:05.1449597Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-04-03T16:22:05.1450273Z parm: NVreg_RegisterPCIDriver:int 2024-04-03T16:22:05.1450878Z parm: NVreg_EnableResizableBar:int 2024-04-03T16:22:05.1451472Z parm: NVreg_EnableDbgBreakpoint:int 2024-04-03T16:22:05.1452082Z parm: NVreg_RegistryDwords:charp 2024-04-03T16:22:05.1452707Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-04-03T16:22:05.1453296Z parm: NVreg_RmMsg:charp 2024-04-03T16:22:05.1453811Z parm: NVreg_GpuBlacklist:charp 2024-04-03T16:22:05.1454395Z parm: NVreg_TemporaryFilePath:charp 2024-04-03T16:22:05.1454972Z parm: NVreg_ExcludedGpus:charp 2024-04-03T16:22:05.1455550Z parm: NVreg_DmaRemapPeerMmio:int 2024-04-03T16:22:05.1456149Z parm: NVreg_RmNvlinkBandwidth:charp 2024-04-03T16:22:05.1456730Z parm: rm_firmware_active:charp 2024-04-03T16:22:05.1457235Z + set +e 2024-04-03T16:22:05.1457612Z + nvidia-smi 2024-04-03T16:22:06.5986669Z Wed Apr 3 16:22:06 2024 2024-04-03T16:22:06.5988335Z +---------------------------------------------------------------------------------------+ 2024-04-03T16:22:06.5989420Z | NVIDIA-SMI 535.54.03 Driver Version: 535.54.03 CUDA Version: 12.2 | 2024-04-03T16:22:06.5990382Z |-----------------------------------------+----------------------+----------------------+ 2024-04-03T16:22:06.5991388Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-04-03T16:22:06.5992535Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-04-03T16:22:06.5993394Z | | | MIG M. | 2024-04-03T16:22:06.5994085Z |=========================================+======================+======================| 2024-04-03T16:22:06.6130197Z | 0 Tesla M60 Off | 00000000:00:1D.0 Off | 0 | 2024-04-03T16:22:06.6131309Z | N/A 24C P0 37W / 150W | 0MiB / 7680MiB | 93% Default | 2024-04-03T16:22:06.6132076Z | | | N/A | 2024-04-03T16:22:06.6132928Z +-----------------------------------------+----------------------+----------------------+ 2024-04-03T16:22:06.6188423Z | 1 Tesla M60 Off | 00000000:00:1E.0 Off | 0 | 2024-04-03T16:22:06.6189647Z | N/A 24C P0 36W / 150W | 0MiB / 7680MiB | 6% Default | 2024-04-03T16:22:06.6190361Z | | | N/A | 2024-04-03T16:22:06.6191201Z +-----------------------------------------+----------------------+----------------------+ 2024-04-03T16:22:06.6191942Z 2024-04-03T16:22:06.6192736Z +---------------------------------------------------------------------------------------+ 2024-04-03T16:22:06.6193737Z | Processes: | 2024-04-03T16:22:06.6194660Z | GPU GI CI PID Type Process name GPU Memory | 2024-04-03T16:22:06.6195529Z | ID ID Usage | 2024-04-03T16:22:06.6196225Z |=======================================================================================| 2024-04-03T16:22:06.6349351Z | No running processes found | 2024-04-03T16:22:06.6350336Z +---------------------------------------------------------------------------------------+ 2024-04-03T16:22:07.5565326Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-04-03T16:22:09.0128696Z Tesla M60 2024-04-03T16:22:09.8885736Z + NVIDIA_SMI_STATUS=0 2024-04-03T16:22:09.8886938Z + '[' 0 -eq 0 ']' 2024-04-03T16:22:09.8887494Z + echo 'INFO: Ignoring allowed status 0' 2024-04-03T16:22:09.8888046Z + set -e 2024-04-03T16:22:09.8888412Z INFO: Ignoring allowed status 0 2024-04-03T16:22:09.8892788Z == Installing nvidia container toolkit for amzn2 == 2024-04-03T16:22:09.8896442Z + sudo yum install -y yum-utils 2024-04-03T16:22:10.4070089Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-04-03T16:22:12.0813725Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-04-03T16:22:12.0814644Z Nothing to do 2024-04-03T16:22:12.2386181Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-04-03T16:22:12.7725760Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-04-03T16:22:12.8219521Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-04-03T16:22:12.8221308Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-04-03T16:22:12.8222559Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-04-03T16:22:12.8369969Z + sudo yum install -y nvidia-docker2 2024-04-03T16:22:13.3619280Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-04-03T16:22:15.0920316Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-04-03T16:22:15.0921162Z Nothing to do 2024-04-03T16:22:15.2436422Z + sudo systemctl restart docker 2024-04-03T16:22:33.1436632Z Command completed after 1 attempt(s). 2024-04-03T16:22:33.1547483Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-04-03T16:22:33.1548740Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-04-03T16:22:33.1549668Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-04-03T16:22:33.1550586Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:22:33.1559685Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:22:33.1560340Z env: 2024-04-03T16:22:33.1560697Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:33.1561287Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:33.1561898Z ##[endgroup] 2024-04-03T16:22:33.4600739Z Defaulting to user installation because normal site-packages is not writeable 2024-04-03T16:22:33.4817071Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-04-03T16:22:33.4943003Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-04-03T16:22:33.6122681Z Prepare all required actions 2024-04-03T16:22:33.6123363Z Getting action download info 2024-04-03T16:22:33.7120239Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-04-03T16:22:34.0068197Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-04-03T16:22:34.1318637Z ##[group]Run ./.github/actions/download-build-artifacts 2024-04-03T16:22:34.1319264Z with: 2024-04-03T16:22:34.1319641Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-04-03T16:22:34.1320197Z s3-bucket: gha-artifacts 2024-04-03T16:22:34.1320624Z env: 2024-04-03T16:22:34.1320948Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:34.1321526Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:34.1322145Z ##[endgroup] 2024-04-03T16:22:34.1367416Z ##[group]Run seemethere/download-artifact-s3@v4 2024-04-03T16:22:34.1367962Z with: 2024-04-03T16:22:34.1368416Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-04-03T16:22:34.1368954Z s3-bucket: gha-artifacts 2024-04-03T16:22:34.1369897Z region: us-east-1 2024-04-03T16:22:34.1370273Z env: 2024-04-03T16:22:34.1370588Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:34.1371156Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:34.1371792Z ##[endgroup] 2024-04-03T16:22:34.8187131Z (node:102598) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-04-03T16:22:34.8188065Z 2024-04-03T16:22:34.8188402Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-04-03T16:22:34.8189333Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-04-03T16:22:34.8190410Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-04-03T16:22:35.0368639Z Found 1 objects with prefix pytorch/pytorch/8536950919/linux-focal-cuda11.8-py3.10-gcc9/ 2024-04-03T16:22:35.0370395Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-04-03T16:22:43.3103233Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-04-03T16:22:43.3114623Z Artifact download has finished successfully 2024-04-03T16:22:43.3355292Z ##[group]Run unzip -o artifacts.zip 2024-04-03T16:22:43.3355875Z unzip -o artifacts.zip 2024-04-03T16:22:43.3364517Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:22:43.3365150Z env: 2024-04-03T16:22:43.3365494Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:43.3366047Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:43.3366658Z ##[endgroup] 2024-04-03T16:22:43.3407041Z Archive: artifacts.zip 2024-04-03T16:22:43.3409027Z creating: dist/ 2024-04-03T16:22:45.7473081Z inflating: dist/torch-2.4.0a0+gitd253bd7-cp310-cp310-linux_x86_64.whl 2024-04-03T16:22:45.7473903Z creating: build/custom_test_artifacts/ 2024-04-03T16:22:45.7474633Z creating: build/custom_test_artifacts/custom-op-build/ 2024-04-03T16:22:45.7475572Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-04-03T16:22:45.7476645Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-04-03T16:22:45.7482694Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-04-03T16:22:45.7484281Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-04-03T16:22:45.7485450Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-04-03T16:22:45.7486759Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-04-03T16:22:45.7487976Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-04-03T16:22:45.7489783Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-04-03T16:22:45.7491377Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-04-03T16:22:45.7492753Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-04-03T16:22:45.7493983Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-04-03T16:22:45.7495498Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-04-03T16:22:45.7497062Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-04-03T16:22:45.7498479Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-04-03T16:22:45.7500017Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-04-03T16:22:45.7501530Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-04-03T16:22:45.7503005Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-04-03T16:22:45.7504338Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-04-03T16:22:45.7505648Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-04-03T16:22:45.7561320Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-04-03T16:22:45.7563162Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-04-03T16:22:45.7564992Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-04-03T16:22:45.7566905Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-04-03T16:22:45.7568816Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-04-03T16:22:45.7570924Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-04-03T16:22:45.7572724Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-04-03T16:22:45.7574564Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-04-03T16:22:45.7576369Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-04-03T16:22:45.7616990Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-04-03T16:22:45.7666385Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-04-03T16:22:45.7668221Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-04-03T16:22:45.7669875Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-04-03T16:22:45.7671554Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-04-03T16:22:45.7673331Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-04-03T16:22:45.7674875Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-04-03T16:22:45.7676443Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-04-03T16:22:45.7678012Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-04-03T16:22:45.7749959Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-04-03T16:22:45.7825697Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-04-03T16:22:45.7827258Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-04-03T16:22:45.7828536Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-04-03T16:22:45.7829639Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-04-03T16:22:45.7830843Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-04-03T16:22:45.7832037Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-04-03T16:22:45.7833354Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-04-03T16:22:45.7834819Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-04-03T16:22:45.7836271Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-04-03T16:22:45.7837605Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-04-03T16:22:45.7839001Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-04-03T16:22:45.7840418Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-04-03T16:22:45.7841825Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-04-03T16:22:45.7843196Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-04-03T16:22:45.7844587Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-04-03T16:22:45.7859917Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-04-03T16:22:45.8026615Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-04-03T16:22:45.8027910Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-04-03T16:22:45.8029322Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-04-03T16:22:45.8030900Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-04-03T16:22:45.8032423Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-04-03T16:22:45.8033782Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-04-03T16:22:45.8035255Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-04-03T16:22:45.8036742Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-04-03T16:22:45.8038212Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-04-03T16:22:45.8039695Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-04-03T16:22:45.8041388Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-04-03T16:22:45.8059410Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-04-03T16:22:45.8162283Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-04-03T16:22:45.8163851Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-03T16:22:45.8165255Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-04-03T16:22:45.8166680Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-04-03T16:22:45.8167840Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-04-03T16:22:45.8168996Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-04-03T16:22:45.8170532Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-04-03T16:22:45.8171626Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-04-03T16:22:45.8172595Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-04-03T16:22:45.8173641Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-04-03T16:22:45.8311280Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-04-03T16:22:45.8388766Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-04-03T16:22:45.8389663Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-04-03T16:22:45.8390593Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-04-03T16:22:45.8391626Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-04-03T16:22:45.8398759Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-04-03T16:22:45.8399982Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-04-03T16:22:45.8401138Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-04-03T16:22:45.8402414Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-04-03T16:22:45.8403628Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-04-03T16:22:45.8405077Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-04-03T16:22:45.8406467Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-04-03T16:22:45.8407772Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-04-03T16:22:45.8409018Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-04-03T16:22:45.8410834Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-04-03T16:22:45.8412331Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-04-03T16:22:45.8413749Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-04-03T16:22:45.8415256Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-04-03T16:22:45.8416805Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-04-03T16:22:45.8418252Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-04-03T16:22:45.8419581Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-04-03T16:22:45.8420869Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-04-03T16:22:45.8475609Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-04-03T16:22:45.8477654Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-04-03T16:22:45.8479484Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-04-03T16:22:45.8481416Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-04-03T16:22:45.8483273Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-04-03T16:22:45.8485171Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-04-03T16:22:45.8487000Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-04-03T16:22:45.8488836Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-04-03T16:22:45.8490947Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-04-03T16:22:45.8531997Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-04-03T16:22:45.8581886Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-04-03T16:22:45.8583694Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-04-03T16:22:45.8585365Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-04-03T16:22:45.8586959Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-04-03T16:22:45.8588548Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-04-03T16:22:45.8590101Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-04-03T16:22:45.8591648Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-04-03T16:22:45.8593204Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-04-03T16:22:45.8665086Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-04-03T16:22:45.8741143Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-04-03T16:22:45.8742679Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-04-03T16:22:45.8743932Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-04-03T16:22:45.8745034Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-04-03T16:22:45.8746174Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-04-03T16:22:45.8747392Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-04-03T16:22:45.8748796Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-04-03T16:22:45.8750291Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-04-03T16:22:45.8751774Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-04-03T16:22:45.8753152Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-04-03T16:22:45.8754598Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-04-03T16:22:45.8756312Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-04-03T16:22:45.8757742Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-04-03T16:22:45.8759182Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-04-03T16:22:45.8760606Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-04-03T16:22:45.8775921Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-04-03T16:22:45.8855489Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-04-03T16:22:45.8857003Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-03T16:22:45.8858419Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-04-03T16:22:45.8859688Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-04-03T16:22:45.8860796Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-04-03T16:22:45.8861942Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-04-03T16:22:45.8863106Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-04-03T16:22:45.8864194Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-04-03T16:22:45.8865240Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-04-03T16:22:45.8866234Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-04-03T16:22:45.8926813Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-04-03T16:22:45.8927733Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-04-03T16:22:45.8928700Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-04-03T16:22:45.8930134Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-04-03T16:22:45.8936975Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-04-03T16:22:45.8938252Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-04-03T16:22:45.8939551Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-04-03T16:22:45.8940920Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-04-03T16:22:45.8942247Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-04-03T16:22:45.8943797Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-04-03T16:22:45.8945287Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-04-03T16:22:45.8946709Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-04-03T16:22:45.8948021Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-04-03T16:22:45.8949604Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-04-03T16:22:45.8951219Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-04-03T16:22:45.8952740Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-04-03T16:22:45.8954317Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-04-03T16:22:45.8955897Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-04-03T16:22:45.8957670Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-04-03T16:22:45.8959083Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-04-03T16:22:45.8960470Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-04-03T16:22:45.9014311Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-04-03T16:22:45.9016310Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-04-03T16:22:45.9018373Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-04-03T16:22:45.9020442Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-04-03T16:22:45.9022422Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-04-03T16:22:45.9024309Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-04-03T16:22:45.9026189Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-04-03T16:22:45.9028094Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-04-03T16:22:45.9029998Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-04-03T16:22:45.9093749Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-04-03T16:22:45.9143807Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-04-03T16:22:45.9145710Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-04-03T16:22:45.9147465Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-04-03T16:22:45.9149166Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-04-03T16:22:45.9150817Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-04-03T16:22:45.9152472Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-04-03T16:22:45.9154123Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-04-03T16:22:45.9155797Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-04-03T16:22:45.9227209Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-04-03T16:22:45.9303953Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-04-03T16:22:45.9305603Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-04-03T16:22:45.9306956Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-04-03T16:22:45.9308156Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-04-03T16:22:45.9309411Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-04-03T16:22:45.9310702Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-04-03T16:22:45.9312189Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-04-03T16:22:45.9313969Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-04-03T16:22:45.9315595Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-04-03T16:22:45.9317104Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-04-03T16:22:45.9318604Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-04-03T16:22:45.9320293Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-04-03T16:22:45.9321861Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-04-03T16:22:45.9323399Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-04-03T16:22:45.9324920Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-04-03T16:22:45.9326541Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-04-03T16:22:45.9474516Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-04-03T16:22:45.9476040Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-04-03T16:22:45.9477563Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-04-03T16:22:45.9479284Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-04-03T16:22:45.9480935Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-04-03T16:22:45.9482493Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-04-03T16:22:45.9484091Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-04-03T16:22:45.9485705Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-04-03T16:22:45.9487317Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-04-03T16:22:45.9488912Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-04-03T16:22:45.9490813Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-04-03T16:22:45.9507451Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-04-03T16:22:45.9576226Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-04-03T16:22:45.9577904Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-03T16:22:45.9579417Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-04-03T16:22:45.9580767Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-04-03T16:22:45.9582015Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-04-03T16:22:45.9583236Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-04-03T16:22:45.9584518Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-04-03T16:22:45.9585707Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-04-03T16:22:45.9586729Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-04-03T16:22:45.9588061Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-04-03T16:22:45.9713491Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-04-03T16:22:45.9766372Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-04-03T16:22:45.9767135Z creating: build/lib/ 2024-04-03T16:22:45.9778597Z inflating: build/lib/libpthreadpool.a 2024-04-03T16:22:45.9779455Z inflating: build/lib/libclog.a 2024-04-03T16:22:45.9782524Z inflating: build/lib/libnnpack_reference_layers.a 2024-04-03T16:22:45.9791982Z inflating: build/lib/libcpuinfo_internals.a 2024-04-03T16:22:45.9802093Z inflating: build/lib/libcpuinfo.a 2024-04-03T16:22:45.9912729Z inflating: build/lib/libprotobuf-lite.a 2024-04-03T16:22:46.0479936Z inflating: build/lib/libprotobuf.a 2024-04-03T16:22:46.0561040Z inflating: build/lib/libgtest.a 2024-04-03T16:22:46.0654271Z inflating: build/lib/libbenchmark.a 2024-04-03T16:22:46.0663682Z inflating: build/lib/libittnotify.a 2024-04-03T16:22:46.0743177Z inflating: build/lib/libasmjit.a 2024-04-03T16:22:46.0777500Z inflating: build/lib/libtensorpipe_uv.a 2024-04-03T16:22:46.0935891Z inflating: build/lib/libgloo.a 2024-04-03T16:22:46.0962601Z inflating: build/lib/libfmt.a 2024-04-03T16:22:46.0965553Z inflating: build/lib/libcaffe2_nvrtc.so 2024-04-03T16:22:46.1077068Z inflating: build/lib/libc10.so 2024-04-03T16:22:46.1077763Z inflating: build/lib/libfoxi_loader.a 2024-04-03T16:22:46.1079838Z inflating: build/lib/libtorch_global_deps.so 2024-04-03T16:22:46.1096832Z inflating: build/lib/libqnnpack.a 2024-04-03T16:22:46.1120906Z inflating: build/lib/libpytorch_qnnpack.a 2024-04-03T16:22:46.1748457Z inflating: build/lib/libprotoc.a 2024-04-03T16:22:46.1771214Z inflating: build/lib/libgmock.a 2024-04-03T16:22:46.1771879Z inflating: build/lib/libgtest_main.a 2024-04-03T16:22:46.1772916Z inflating: build/lib/libbenchmark_main.a 2024-04-03T16:22:46.1795915Z inflating: build/lib/libnnpack.a 2024-04-03T16:22:47.3623014Z inflating: build/lib/libdnnl.a 2024-04-03T16:22:47.4337399Z inflating: build/lib/libtensorpipe.a 2024-04-03T16:22:47.4404431Z inflating: build/lib/libc10_cuda.so 2024-04-03T16:22:47.5966968Z inflating: build/lib/libfbgemm.a 2024-04-03T16:22:47.5967551Z inflating: build/lib/libgmock_main.a 2024-04-03T16:22:47.6286586Z inflating: build/lib/libtensorpipe_cuda.a 2024-04-03T16:22:47.6924132Z inflating: build/lib/libkineto.a 2024-04-03T16:22:47.7410765Z inflating: build/lib/libgloo_cuda.a 2024-04-03T16:22:47.7459153Z inflating: build/lib/libcaffe2_protos.a 2024-04-03T16:22:47.7510960Z inflating: build/lib/libonnx_proto.a 2024-04-03T16:22:47.7740844Z inflating: build/lib/libXNNPACK.a 2024-04-03T16:22:47.8583826Z inflating: build/lib/libonnx.a 2024-04-03T16:22:50.7792087Z inflating: build/lib/libtorch_cpu.so 2024-04-03T16:22:50.7796817Z inflating: build/lib/libshm.so 2024-04-03T16:22:50.7802538Z inflating: build/lib/libunbox_lib.a 2024-04-03T16:22:53.2072223Z inflating: build/lib/libtorch_cuda.so 2024-04-03T16:22:53.2073142Z inflating: build/lib/libtorch.so 2024-04-03T16:22:53.2076436Z inflating: build/lib/libc10d_cuda_test.so 2024-04-03T16:22:54.3458787Z inflating: build/lib/libtorch_cuda_linalg.so 2024-04-03T16:22:54.3484054Z inflating: build/lib/libjitbackend_test.so 2024-04-03T16:22:54.3555390Z inflating: build/lib/libtorchbind_test.so 2024-04-03T16:22:54.3586841Z inflating: build/lib/libbackend_with_compiler.so 2024-04-03T16:22:54.5927935Z inflating: build/lib/libtorch_python.so 2024-04-03T16:22:54.5970634Z inflating: build/lib/libnnapi_backend.so 2024-04-03T16:22:54.5971218Z creating: build/bin/ 2024-04-03T16:22:54.6033418Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-04-03T16:22:54.6097884Z inflating: build/bin/c10_DeviceGuard_test 2024-04-03T16:22:54.6160682Z inflating: build/bin/c10_Device_test 2024-04-03T16:22:54.6234663Z inflating: build/bin/c10_DispatchKeySet_test 2024-04-03T16:22:54.6295860Z inflating: build/bin/c10_StreamGuard_test 2024-04-03T16:22:54.6362669Z inflating: build/bin/c10_Scalar_test 2024-04-03T16:22:54.6426668Z inflating: build/bin/c10_SymInt_test 2024-04-03T16:22:54.6494317Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-04-03T16:22:54.6563975Z inflating: build/bin/c10_InlineStreamGuard_test 2024-04-03T16:22:54.6634361Z inflating: build/bin/c10_SizesAndStrides_test 2024-04-03T16:22:54.6723436Z inflating: build/bin/c10_cow_test 2024-04-03T16:22:54.6790168Z inflating: build/bin/c10_Bitset_test 2024-04-03T16:22:54.6851828Z inflating: build/bin/c10_ConstexprCrc_test 2024-04-03T16:22:54.6913583Z inflating: build/bin/c10_DeadlockDetection_test 2024-04-03T16:22:54.6977254Z inflating: build/bin/c10_Half_test 2024-04-03T16:22:54.7047519Z inflating: build/bin/c10_LeftRight_test 2024-04-03T16:22:54.7116412Z inflating: build/bin/c10_Metaprogramming_test 2024-04-03T16:22:54.7179006Z inflating: build/bin/c10_Synchronized_test 2024-04-03T16:22:54.7248323Z inflating: build/bin/c10_ThreadLocal_test 2024-04-03T16:22:54.7313362Z inflating: build/bin/c10_TypeIndex_test 2024-04-03T16:22:54.7377910Z inflating: build/bin/c10_TypeList_test 2024-04-03T16:22:54.7438907Z inflating: build/bin/c10_TypeTraits_test 2024-04-03T16:22:54.7503717Z inflating: build/bin/c10_accumulate_test 2024-04-03T16:22:54.7573485Z inflating: build/bin/c10_bfloat16_test 2024-04-03T16:22:54.7636550Z inflating: build/bin/c10_bit_cast_test 2024-04-03T16:22:54.7707130Z inflating: build/bin/c10_complex_math_test 2024-04-03T16:22:54.7777232Z inflating: build/bin/c10_complex_test 2024-04-03T16:22:54.7842606Z inflating: build/bin/c10_exception_test 2024-04-03T16:22:54.7905926Z inflating: build/bin/c10_flags_test 2024-04-03T16:22:54.7968293Z inflating: build/bin/c10_generic_math_test 2024-04-03T16:22:54.8032764Z inflating: build/bin/c10_irange_test 2024-04-03T16:22:54.8239001Z inflating: build/bin/c10_intrusive_ptr_test 2024-04-03T16:22:54.8308833Z inflating: build/bin/c10_logging_test 2024-04-03T16:22:54.8402410Z inflating: build/bin/c10_optional_test 2024-04-03T16:22:54.8480395Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-04-03T16:22:54.8547996Z inflating: build/bin/c10_registry_test 2024-04-03T16:22:54.8737102Z inflating: build/bin/c10_small_vector_test 2024-04-03T16:22:54.8801542Z inflating: build/bin/c10_ssize_test 2024-04-03T16:22:54.8866702Z inflating: build/bin/c10_string_util_test 2024-04-03T16:22:54.8929592Z inflating: build/bin/c10_tempfile_test 2024-04-03T16:22:54.9002947Z inflating: build/bin/c10_string_view_test 2024-04-03T16:22:54.9063802Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-04-03T16:22:54.9133870Z inflating: build/bin/c10_typeid_test 2024-04-03T16:22:54.9691508Z inflating: build/bin/protoc-3.13.0.0 2024-04-03T16:22:55.0248244Z inflating: build/bin/protoc 2024-04-03T16:22:55.0314513Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-04-03T16:22:55.0380814Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-04-03T16:22:55.0446568Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-04-03T16:22:55.0511845Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-04-03T16:22:55.0578171Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-04-03T16:22:55.0644187Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-04-03T16:22:55.0710287Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-04-03T16:22:55.0771441Z inflating: build/bin/c10_cuda_CUDATest 2024-04-03T16:22:55.1182697Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-04-03T16:22:55.1629294Z inflating: build/bin/vec_test_all_types_AVX2 2024-04-03T16:22:55.2058876Z inflating: build/bin/vec_test_all_types_AVX512 2024-04-03T16:22:55.2124928Z inflating: build/bin/FileStoreTest 2024-04-03T16:22:55.2194517Z inflating: build/bin/TCPStoreTest 2024-04-03T16:22:55.2261294Z inflating: build/bin/HashStoreTest 2024-04-03T16:22:55.2278103Z inflating: build/bin/ProcessGroupMPITest 2024-04-03T16:22:55.2283377Z inflating: build/bin/torch_shm_manager 2024-04-03T16:22:55.2350634Z inflating: build/bin/test_edge_op_registration 2024-04-03T16:22:55.2354267Z inflating: build/bin/example_allreduce 2024-04-03T16:22:55.2419546Z inflating: build/bin/Dimname_test 2024-04-03T16:22:55.2511791Z inflating: build/bin/Dict_test 2024-04-03T16:22:55.2592502Z inflating: build/bin/MaybeOwned_test 2024-04-03T16:22:55.2666528Z inflating: build/bin/apply_utils_test 2024-04-03T16:22:55.2738271Z inflating: build/bin/NamedTensor_test 2024-04-03T16:22:55.2811894Z inflating: build/bin/atest 2024-04-03T16:22:55.2890644Z inflating: build/bin/basic 2024-04-03T16:22:55.2959279Z inflating: build/bin/broadcast_test 2024-04-03T16:22:55.3022886Z inflating: build/bin/cpu_allocator_test 2024-04-03T16:22:55.3095423Z inflating: build/bin/cpu_generator_test 2024-04-03T16:22:55.3161915Z inflating: build/bin/cpu_profiling_allocator_test 2024-04-03T16:22:55.3278427Z inflating: build/bin/cpu_rng_test 2024-04-03T16:22:55.3341366Z inflating: build/bin/dispatch_key_set_test 2024-04-03T16:22:55.3404284Z inflating: build/bin/dlconvertor_test 2024-04-03T16:22:55.3476492Z inflating: build/bin/extension_backend_test 2024-04-03T16:22:55.3545759Z inflating: build/bin/half_test 2024-04-03T16:22:55.3665506Z inflating: build/bin/ivalue_test 2024-04-03T16:22:55.3727509Z inflating: build/bin/lazy_tensor_test 2024-04-03T16:22:55.3795581Z inflating: build/bin/math_kernel_test 2024-04-03T16:22:55.3863273Z inflating: build/bin/memory_format_test 2024-04-03T16:22:55.3929640Z inflating: build/bin/memory_overlapping_test 2024-04-03T16:22:55.3995928Z inflating: build/bin/mobile_memory_cleanup 2024-04-03T16:22:55.4059170Z inflating: build/bin/operator_name_test 2024-04-03T16:22:55.4129003Z inflating: build/bin/native_test 2024-04-03T16:22:55.4193673Z inflating: build/bin/operators_test 2024-04-03T16:22:55.4258629Z inflating: build/bin/packedtensoraccessor_test 2024-04-03T16:22:55.4342658Z inflating: build/bin/pow_test 2024-04-03T16:22:55.4414669Z inflating: build/bin/quantized_test 2024-04-03T16:22:55.4476788Z inflating: build/bin/reduce_ops_test 2024-04-03T16:22:55.4540752Z inflating: build/bin/reportMemoryUsage_test 2024-04-03T16:22:55.4611395Z inflating: build/bin/scalar_tensor_test 2024-04-03T16:22:55.4685061Z inflating: build/bin/scalar_test 2024-04-03T16:22:55.4749948Z inflating: build/bin/StorageUtils_test 2024-04-03T16:22:55.4815726Z inflating: build/bin/stride_properties_test 2024-04-03T16:22:55.4914180Z inflating: build/bin/tensor_iterator_test 2024-04-03T16:22:55.4983921Z inflating: build/bin/type_ptr_test 2024-04-03T16:22:55.4987558Z inflating: build/bin/thread_init_test 2024-04-03T16:22:55.5055346Z inflating: build/bin/test_parallel 2024-04-03T16:22:55.5120253Z inflating: build/bin/undefined_tensor_test 2024-04-03T16:22:55.5195172Z inflating: build/bin/type_test 2024-04-03T16:22:55.5197146Z inflating: build/bin/verify_api_visibility 2024-04-03T16:22:55.5284139Z inflating: build/bin/legacy_vmap_test 2024-04-03T16:22:55.5349148Z inflating: build/bin/weakref_test 2024-04-03T16:22:55.5413779Z inflating: build/bin/wrapdim_test 2024-04-03T16:22:55.5488796Z inflating: build/bin/IListRef_test 2024-04-03T16:22:55.5553430Z inflating: build/bin/xla_tensor_test 2024-04-03T16:22:55.5686022Z inflating: build/bin/List_test 2024-04-03T16:22:55.5838525Z inflating: build/bin/kernel_function_legacy_test 2024-04-03T16:22:55.5921778Z inflating: build/bin/KernelFunction_test 2024-04-03T16:22:55.6042403Z inflating: build/bin/kernel_function_test 2024-04-03T16:22:55.6203271Z inflating: build/bin/kernel_lambda_legacy_test 2024-04-03T16:22:55.6333275Z inflating: build/bin/kernel_lambda_test 2024-04-03T16:22:55.6409282Z inflating: build/bin/kernel_stackbased_test 2024-04-03T16:22:55.6529743Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-04-03T16:22:55.6593955Z inflating: build/bin/CppSignature_test 2024-04-03T16:22:55.6655618Z inflating: build/bin/op_allowlist_test 2024-04-03T16:22:55.7032230Z inflating: build/bin/op_registration_test 2024-04-03T16:22:55.7102383Z inflating: build/bin/backend_fallback_test 2024-04-03T16:22:55.7182061Z inflating: build/bin/inline_container_test 2024-04-03T16:22:55.7247993Z inflating: build/bin/cuda_apply_test 2024-04-03T16:22:55.7311934Z inflating: build/bin/cuda_allocator_test 2024-04-03T16:22:55.7386637Z inflating: build/bin/cuda_atomic_ops_test 2024-04-03T16:22:55.7454836Z inflating: build/bin/cuda_caching_host_allocator_test 2024-04-03T16:22:55.7542747Z inflating: build/bin/cuda_complex_math_test 2024-04-03T16:22:55.7604642Z inflating: build/bin/cuda_device_test 2024-04-03T16:22:55.7678503Z inflating: build/bin/cuda_complex_test 2024-04-03T16:22:55.7750797Z inflating: build/bin/cuda_cub_test 2024-04-03T16:22:55.7814559Z inflating: build/bin/cuda_dlconvertor_test 2024-04-03T16:22:55.7878288Z inflating: build/bin/cuda_integer_divider_test 2024-04-03T16:22:55.7960094Z inflating: build/bin/cuda_distributions_test 2024-04-03T16:22:55.8022427Z inflating: build/bin/cuda_half_test 2024-04-03T16:22:55.8093364Z inflating: build/bin/cuda_generator_test 2024-04-03T16:22:55.8154913Z inflating: build/bin/cuda_optional_test 2024-04-03T16:22:55.8217479Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-04-03T16:22:55.8283099Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-04-03T16:22:55.8358800Z inflating: build/bin/cuda_stream_test 2024-04-03T16:22:55.8423272Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-04-03T16:22:55.8484934Z inflating: build/bin/cuda_cudnn_test 2024-04-03T16:22:55.8550101Z inflating: build/bin/cuda_vectorized_test 2024-04-03T16:22:55.8568819Z inflating: build/bin/tutorial_tensorexpr 2024-04-03T16:22:55.8650978Z inflating: build/bin/ProcessGroupGlooTest 2024-04-03T16:22:55.8722323Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-04-03T16:22:55.8800645Z inflating: build/bin/ProcessGroupNCCLTest 2024-04-03T16:22:55.8866418Z inflating: build/bin/ProcessGroupUCCTest 2024-04-03T16:22:55.8943498Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-04-03T16:22:55.9012259Z inflating: build/bin/test_dist_autograd 2024-04-03T16:22:55.9015471Z inflating: build/bin/parallel_benchmark 2024-04-03T16:22:55.9101928Z inflating: build/bin/test_cpp_rpc 2024-04-03T16:22:55.9186112Z inflating: build/bin/test_mobile_nnc 2024-04-03T16:22:55.9197104Z inflating: build/bin/aot_model_compiler_test 2024-04-03T16:22:56.0235833Z inflating: build/bin/test_tensorexpr 2024-04-03T16:22:56.0668183Z inflating: build/bin/test_lazy 2024-04-03T16:22:56.1375286Z inflating: build/bin/test_jit 2024-04-03T16:22:56.2850585Z inflating: build/bin/test_api 2024-04-03T16:22:56.2851134Z creating: .additional_ci_files/ 2024-04-03T16:22:56.2896008Z inflating: .additional_ci_files/test-times.json 2024-04-03T16:22:56.3075750Z inflating: .additional_ci_files/test-class-times.json 2024-04-03T16:22:56.3193473Z ##[group]Run rm artifacts.zip 2024-04-03T16:22:56.3194064Z rm artifacts.zip 2024-04-03T16:22:56.3203244Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:22:56.3203884Z env: 2024-04-03T16:22:56.3204236Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:56.3204796Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:56.3205411Z ##[endgroup] 2024-04-03T16:22:56.4391638Z ##[group]Run df -H 2024-04-03T16:22:56.4392021Z df -H 2024-04-03T16:22:56.4400347Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:22:56.4400980Z env: 2024-04-03T16:22:56.4401328Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:56.4401882Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:56.4402493Z ##[endgroup] 2024-04-03T16:22:56.4437173Z Filesystem Size Used Avail Use% Mounted on 2024-04-03T16:22:56.4438668Z devtmpfs 129G 0 129G 0% /dev 2024-04-03T16:22:56.4439443Z tmpfs 129G 0 129G 0% /dev/shm 2024-04-03T16:22:56.4440052Z tmpfs 129G 533k 129G 1% /run 2024-04-03T16:22:56.4441365Z tmpfs 129G 0 129G 0% /sys/fs/cgroup 2024-04-03T16:22:56.4441996Z /dev/xvda1 162G 37G 125G 23% / 2024-04-03T16:22:56.4491915Z Prepare all required actions 2024-04-03T16:22:56.4492500Z Getting action download info 2024-04-03T16:22:56.5816612Z ##[group]Run ./.github/actions/download-td-artifacts 2024-04-03T16:22:56.5817214Z with: 2024-04-03T16:22:56.5817528Z env: 2024-04-03T16:22:56.5817870Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:56.5818430Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:56.5819045Z ##[endgroup] 2024-04-03T16:22:56.5879784Z ##[group]Run seemethere/download-artifact-s3@v4 2024-04-03T16:22:56.5880382Z with: 2024-04-03T16:22:56.5880710Z name: td_results 2024-04-03T16:22:56.5881135Z s3-bucket: gha-artifacts 2024-04-03T16:22:56.5881607Z region: us-east-1 2024-04-03T16:22:56.5881978Z env: 2024-04-03T16:22:56.5882335Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:56.5882940Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:56.5883561Z ##[endgroup] 2024-04-03T16:22:57.1966345Z (node:102625) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-04-03T16:22:57.1967510Z 2024-04-03T16:22:57.1967879Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-04-03T16:22:57.1968799Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-04-03T16:22:57.1970155Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-04-03T16:22:57.2957838Z Found 1 objects with prefix pytorch/pytorch/8536950919/td_results/ 2024-04-03T16:22:57.2960237Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-04-03T16:22:57.3742222Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-04-03T16:22:57.3752831Z Artifact download has finished successfully 2024-04-03T16:22:57.3956998Z ##[group]Run mkdir -p .additional_ci_files 2024-04-03T16:22:57.3957612Z mkdir -p .additional_ci_files 2024-04-03T16:22:57.3958340Z mv td_results.json .additional_ci_files/td_results.json 2024-04-03T16:22:57.3967140Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:22:57.3967771Z env: 2024-04-03T16:22:57.3968116Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:57.3968683Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:57.3969569Z ##[endgroup] 2024-04-03T16:22:57.4071187Z ##[group]Run .github/scripts/parse_ref.py 2024-04-03T16:22:57.4071928Z .github/scripts/parse_ref.py 2024-04-03T16:22:57.4080038Z shell: /usr/bin/bash -e {0} 2024-04-03T16:22:57.4080491Z env: 2024-04-03T16:22:57.4080836Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:57.4081395Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:57.4082010Z ##[endgroup] 2024-04-03T16:22:57.4392394Z Prepare all required actions 2024-04-03T16:22:57.4443187Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-04-03T16:22:57.4443765Z with: 2024-04-03T16:22:57.4444462Z github-token: *** 2024-04-03T16:22:57.4444827Z env: 2024-04-03T16:22:57.4445164Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:57.4445729Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:57.4446317Z ##[endgroup] 2024-04-03T16:22:57.4469080Z ##[group]Run set -eux 2024-04-03T16:22:57.4469506Z set -eux 2024-04-03T16:22:57.4470244Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-04-03T16:22:57.4478311Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:22:57.4478925Z env: 2024-04-03T16:22:57.4479267Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:22:57.4479839Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:22:57.4480663Z GITHUB_TOKEN: *** 2024-04-03T16:22:57.4481037Z ##[endgroup] 2024-04-03T16:22:57.4506177Z + python3 .github/scripts/get_workflow_job_id.py 8536950919 i-0ec6a6fea89e9eddd 2024-04-03T16:23:05.7082598Z setting job-id=23402705639 2024-04-03T16:23:05.7083802Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:23:05.7356236Z Prepare all required actions 2024-04-03T16:23:05.7356957Z Getting action download info 2024-04-03T16:23:05.8458075Z ##[group]Run ./.github/actions/filter-test-configs 2024-04-03T16:23:05.8458667Z with: 2024-04-03T16:23:05.8459243Z github-token: *** 2024-04-03T16:23:05.8461082Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-04-03T16:23:05.8463452Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:23:05.8464374Z env: 2024-04-03T16:23:05.8464698Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:05.8465269Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:05.8465874Z ##[endgroup] 2024-04-03T16:23:05.8521323Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-04-03T16:23:05.8522015Z with: 2024-04-03T16:23:05.8522350Z shell: bash 2024-04-03T16:23:05.8522721Z timeout_minutes: 10 2024-04-03T16:23:05.8523119Z max_attempts: 5 2024-04-03T16:23:05.8523507Z retry_wait_seconds: 30 2024-04-03T16:23:05.8524482Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-04-03T16:23:05.8525543Z polling_interval_seconds: 1 2024-04-03T16:23:05.8525993Z warning_on_retry: true 2024-04-03T16:23:05.8526425Z continue_on_error: false 2024-04-03T16:23:05.8526826Z env: 2024-04-03T16:23:05.8527163Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:05.8527725Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:05.8528543Z GITHUB_TOKEN: *** 2024-04-03T16:23:05.8528916Z ##[endgroup] 2024-04-03T16:23:05.9266927Z + python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-04-03T16:23:06.2154163Z Defaulting to user installation because normal site-packages is not writeable 2024-04-03T16:23:06.2362224Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0) 2024-04-03T16:23:06.2545039Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-04-03T16:23:06.2556581Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.6) 2024-04-03T16:23:06.2572884Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12) 2024-04-03T16:23:06.2598241Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.18) 2024-04-03T16:23:06.2839308Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2024.2.2) 2024-04-03T16:23:06.9258695Z Command completed after 1 attempt(s). 2024-04-03T16:23:06.9326332Z ##[group]Run set -x 2024-04-03T16:23:06.9326807Z set -x 2024-04-03T16:23:06.9327575Z  2024-04-03T16:23:06.9328313Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-04-03T16:23:06.9330064Z # in runner workspace 2024-04-03T16:23:06.9331107Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-04-03T16:23:06.9341036Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:23:06.9341724Z env: 2024-04-03T16:23:06.9342118Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:06.9342714Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:06.9343375Z ##[endgroup] 2024-04-03T16:23:06.9375952Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-04-03T16:23:06.9685432Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-04-03T16:23:06.9686351Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-04-03T16:23:06.9686975Z echo "Job name: ${JOB_NAME}" 2024-04-03T16:23:06.9687507Z  2024-04-03T16:23:06.9688238Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-04-03T16:23:06.9689397Z # in runner workspace 2024-04-03T16:23:06.9690182Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-04-03T16:23:06.9691020Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-04-03T16:23:06.9691615Z  --job-name "${JOB_NAME}" \ 2024-04-03T16:23:06.9693651Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" \ 2024-04-03T16:23:06.9695735Z  --pr-number "${PR_NUMBER}" \ 2024-04-03T16:23:06.9696302Z  --tag "${TAG}" \ 2024-04-03T16:23:06.9696828Z  --event-name "${EVENT_NAME}" \ 2024-04-03T16:23:06.9697403Z  --schedule "${SCHEDULE}" \ 2024-04-03T16:23:06.9697968Z  --branch "${HEAD_BRANCH}" 2024-04-03T16:23:06.9706471Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:23:06.9707128Z env: 2024-04-03T16:23:06.9707471Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:06.9708069Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:06.9708963Z GITHUB_TOKEN: *** 2024-04-03T16:23:06.9709792Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:23:06.9710739Z PR_NUMBER: 123154 2024-04-03T16:23:06.9711147Z TAG: 2024-04-03T16:23:06.9711494Z EVENT_NAME: pull_request 2024-04-03T16:23:06.9711952Z SCHEDULE: 2024-04-03T16:23:06.9712335Z HEAD_BRANCH: 2024-04-03T16:23:06.9712700Z ##[endgroup] 2024-04-03T16:23:06.9738868Z Workflow: pull 2024-04-03T16:23:06.9739861Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:23:07.2156026Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-04-03T16:23:07.3168785Z fatal: unknown commit main 2024-04-03T16:23:07.3195851Z /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/filter_test_configs.py:458: UserWarning: failed to get commit messages: Command '['git', 'cherry', '-v', 'main']' returned non-zero exit status 128. 2024-04-03T16:23:07.3197873Z warnings.warn(f"failed to get commit messages: {e}") 2024-04-03T16:23:07.3350201Z ##[group]Run echo "Filtered matrix:" 2024-04-03T16:23:07.3350771Z echo "Filtered matrix:" 2024-04-03T16:23:07.3352723Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" 2024-04-03T16:23:07.3354648Z  2024-04-03T16:23:07.3355195Z echo 2024-04-03T16:23:07.3355704Z echo "Is the current job unstable? False" 2024-04-03T16:23:07.3356319Z  2024-04-03T16:23:07.3356657Z echo 2024-04-03T16:23:07.3357140Z echo "Is keep-going label set? False" 2024-04-03T16:23:07.3357731Z  2024-04-03T16:23:07.3358066Z echo 2024-04-03T16:23:07.3358484Z echo "Renabled issues? " 2024-04-03T16:23:07.3366766Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:23:07.3367433Z env: 2024-04-03T16:23:07.3367779Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:07.3368394Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:07.3369040Z ##[endgroup] 2024-04-03T16:23:07.3394230Z Filtered matrix: 2024-04-03T16:23:07.3396724Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}]} 2024-04-03T16:23:07.3398448Z 2024-04-03T16:23:07.3398637Z Is the current job unstable? False 2024-04-03T16:23:07.3398994Z 2024-04-03T16:23:07.3399267Z Is keep-going label set? False 2024-04-03T16:23:07.3399596Z 2024-04-03T16:23:07.3399747Z Renabled issues? 2024-04-03T16:23:07.3453370Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-04-03T16:23:07.3454312Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-04-03T16:23:07.3462175Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:23:07.3462809Z env: 2024-04-03T16:23:07.3463136Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:07.3463710Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:07.3464330Z JOB_TIMEOUT: 360 2024-04-03T16:23:07.3464694Z ##[endgroup] 2024-04-03T16:23:07.3559139Z ##[group]Run set -x 2024-04-03T16:23:07.3559674Z set -x 2024-04-03T16:23:07.3560045Z  2024-04-03T16:23:07.3560499Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-04-03T16:23:07.3561182Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-04-03T16:23:07.3561894Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-04-03T16:23:07.3562535Z  TEST_COMMAND=.ci/onnx/test.sh 2024-04-03T16:23:07.3563035Z else 2024-04-03T16:23:07.3563465Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-04-03T16:23:07.3563999Z fi 2024-04-03T16:23:07.3564324Z  2024-04-03T16:23:07.3564909Z # detached container should get cleaned up by teardown_ec2_linux 2024-04-03T16:23:07.3565884Z # TODO: Stop building test binaries as part of the build phase 2024-04-03T16:23:07.3566747Z # Used for GPU_FLAG since that doesn't play nice 2024-04-03T16:23:07.3567490Z # shellcheck disable=SC2086,SC2090 2024-04-03T16:23:07.3568065Z container_name=$(docker run \ 2024-04-03T16:23:07.3568597Z  ${GPU_FLAG:-} \ 2024-04-03T16:23:07.3569371Z  -e BUILD_ENVIRONMENT \ 2024-04-03T16:23:07.3569992Z  -e PR_NUMBER \ 2024-04-03T16:23:07.3570455Z  -e GITHUB_ACTIONS \ 2024-04-03T16:23:07.3570943Z  -e GITHUB_REPOSITORY \ 2024-04-03T16:23:07.3571433Z  -e GITHUB_WORKFLOW \ 2024-04-03T16:23:07.3571914Z  -e GITHUB_JOB \ 2024-04-03T16:23:07.3572368Z  -e GITHUB_RUN_ID \ 2024-04-03T16:23:07.3572832Z  -e GITHUB_RUN_NUMBER \ 2024-04-03T16:23:07.3573344Z  -e GITHUB_RUN_ATTEMPT \ 2024-04-03T16:23:07.3573841Z  -e JOB_ID \ 2024-04-03T16:23:07.3574247Z  -e JOB_NAME \ 2024-04-03T16:23:07.3574682Z  -e BASE_SHA \ 2024-04-03T16:23:07.3575111Z  -e BRANCH \ 2024-04-03T16:23:07.3575512Z  -e SHA1 \ 2024-04-03T16:23:07.3575946Z  -e AWS_DEFAULT_REGION \ 2024-04-03T16:23:07.3576455Z  -e IN_WHEEL_TEST \ 2024-04-03T16:23:07.3576911Z  -e SHARD_NUMBER \ 2024-04-03T16:23:07.3577585Z  -e TEST_CONFIG \ 2024-04-03T16:23:07.3578048Z  -e NUM_TEST_SHARDS \ 2024-04-03T16:23:07.3578522Z  -e REENABLED_ISSUES \ 2024-04-03T16:23:07.3579041Z  -e CONTINUE_THROUGH_ERROR \ 2024-04-03T16:23:07.3579579Z  -e VERBOSE_TEST_LOGS \ 2024-04-03T16:23:07.3580063Z  -e NO_TEST_TIMEOUT \ 2024-04-03T16:23:07.3580535Z  -e NO_TD \ 2024-04-03T16:23:07.3580955Z  -e PR_LABELS \ 2024-04-03T16:23:07.3581451Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-04-03T16:23:07.3582027Z  -e SCCACHE_BUCKET \ 2024-04-03T16:23:07.3582526Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-04-03T16:23:07.3583026Z  -e XLA_CUDA \ 2024-04-03T16:23:07.3583530Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-04-03T16:23:07.3584181Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-04-03T16:23:07.3584827Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-04-03T16:23:07.3585482Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-04-03T16:23:07.3586082Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-04-03T16:23:07.3586611Z  -e DASHBOARD_TAG \ 2024-04-03T16:23:07.3587198Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-04-03T16:23:07.3587877Z  --ulimit stack=10485760:83886080 \ 2024-04-03T16:23:07.3588490Z  --security-opt seccomp=unconfined \ 2024-04-03T16:23:07.3589058Z  --cap-add=SYS_PTRACE \ 2024-04-03T16:23:07.3589555Z  --ipc=host \ 2024-04-03T16:23:07.3590005Z  --shm-size="${SHM_SIZE}" \ 2024-04-03T16:23:07.3590486Z  --tty \ 2024-04-03T16:23:07.3590877Z  --detach \ 2024-04-03T16:23:07.3591322Z  --name="${container_name}" \ 2024-04-03T16:23:07.3591831Z  --user jenkins \ 2024-04-03T16:23:07.3592580Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-04-03T16:23:07.3593310Z  -w /var/lib/jenkins/workspace \ 2024-04-03T16:23:07.3593840Z  "${DOCKER_IMAGE}" 2024-04-03T16:23:07.3594290Z ) 2024-04-03T16:23:07.3594802Z # Propagate download.pytorch.org IP to container 2024-04-03T16:23:07.3595972Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-04-03T16:23:07.3597211Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-04-03T16:23:07.3598380Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-04-03T16:23:07.3606709Z shell: /usr/bin/bash -e {0} 2024-04-03T16:23:07.3607156Z env: 2024-04-03T16:23:07.3607484Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:23:07.3608060Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:07.3608812Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-04-03T16:23:07.3609807Z PR_NUMBER: 123154 2024-04-03T16:23:07.3610244Z GITHUB_REPOSITORY: pytorch/pytorch 2024-04-03T16:23:07.3610770Z GITHUB_WORKFLOW: pull 2024-04-03T16:23:07.3611187Z GITHUB_JOB: test 2024-04-03T16:23:07.3611580Z GITHUB_RUN_ID: 8536950919 2024-04-03T16:23:07.3612030Z GITHUB_RUN_NUMBER: 192590 2024-04-03T16:23:07.3612463Z GITHUB_RUN_ATTEMPT: 2 2024-04-03T16:23:07.3612881Z JOB_ID: 23402705639 2024-04-03T16:23:07.3613704Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:23:07.3614616Z BRANCH: pull/123154 2024-04-03T16:23:07.3615151Z SHA1: d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:23:07.3615814Z BASE_SHA: 1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:23:07.3616394Z TEST_CONFIG: distributed 2024-04-03T16:23:07.3616837Z SHARD_NUMBER: 1 2024-04-03T16:23:07.3617223Z NUM_TEST_SHARDS: 3 2024-04-03T16:23:07.3617612Z REENABLED_ISSUES: 2024-04-03T16:23:07.3618045Z CONTINUE_THROUGH_ERROR: False 2024-04-03T16:23:07.3618533Z VERBOSE_TEST_LOGS: False 2024-04-03T16:23:07.3618961Z NO_TEST_TIMEOUT: False 2024-04-03T16:23:07.3619563Z NO_TD: False 2024-04-03T16:23:07.3620044Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-04-03T16:23:07.3620649Z SCCACHE_S3_KEY_PREFIX: pull 2024-04-03T16:23:07.3621107Z SHM_SIZE: 2g 2024-04-03T16:23:07.3622319Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:23:07.3623627Z XLA_CUDA: 2024-04-03T16:23:07.3624244Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-04-03T16:23:07.3625020Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-04-03T16:23:07.3625567Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-04-03T16:23:07.3626065Z DASHBOARD_TAG: 2024-04-03T16:23:07.3626463Z HUGGING_FACE_HUB_TOKEN: 2024-04-03T16:23:07.3626893Z ##[endgroup] 2024-04-03T16:23:07.3652037Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-04-03T16:23:07.3652797Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-04-03T16:23:07.3653422Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-04-03T16:23:07.3662304Z +++ nproc --ignore=2 2024-04-03T16:23:07.3680059Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_8536950919 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:23:18.4360460Z + container_name=b758c44c85badb76c4c382a8895cd5e1bfdf37df294e5f1babfcbd8fbafbdd0d 2024-04-03T16:23:18.4362625Z + grep download.pytorch.org /etc/hosts 2024-04-03T16:23:18.4365103Z + docker exec -i b758c44c85badb76c4c382a8895cd5e1bfdf37df294e5f1babfcbd8fbafbdd0d sudo bash -c '/bin/cat >> /etc/hosts' 2024-04-03T16:23:18.5116044Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-04-03T16:23:18.5167776Z + echo DOCKER_CONTAINER_ID=b758c44c85badb76c4c382a8895cd5e1bfdf37df294e5f1babfcbd8fbafbdd0d 2024-04-03T16:23:18.5172666Z ++ echo dist/torch-2.4.0a0+gitd253bd7-cp310-cp310-linux_x86_64.whl 2024-04-03T16:23:18.5175477Z + docker exec -t b758c44c85badb76c4c382a8895cd5e1bfdf37df294e5f1babfcbd8fbafbdd0d sh -c 'pip install dist/torch-2.4.0a0+gitd253bd7-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-04-03T16:23:19.0629758Z Processing ./dist/torch-2.4.0a0+gitd253bd7-cp310-cp310-linux_x86_64.whl 2024-04-03T16:23:20.1701135Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (3.13.1) 2024-04-03T16:23:20.1710475Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (4.10.0) 2024-04-03T16:23:20.1715472Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (1.12) 2024-04-03T16:23:20.1720514Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (2.8.8) 2024-04-03T16:23:20.1725698Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (3.1.3) 2024-04-03T16:23:20.1730916Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (2024.2.0) 2024-04-03T16:23:20.1746322Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitd253bd7) (3.3.0) 2024-04-03T16:23:20.1829633Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.4.0a0+gitd253bd7) (1.21.2) 2024-04-03T16:23:20.2433667Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.4.0a0+gitd253bd7) (2.1.5) 2024-04-03T16:23:20.2658068Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.4.0a0+gitd253bd7) (1.3.0) 2024-04-03T16:23:21.2768244Z Installing collected packages: torch 2024-04-03T16:23:33.4907031Z Successfully installed torch-2.4.0a0+gitd253bd7 2024-04-03T16:23:33.5592223Z + echo 'Environment variables:' 2024-04-03T16:23:33.5592827Z Environment variables: 2024-04-03T16:23:33.5593214Z + env 2024-04-03T16:23:33.5600569Z INSTALLED_DB=yes 2024-04-03T16:23:33.5601168Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-04-03T16:23:33.5601922Z NVIDIA_VISIBLE_DEVICES=all 2024-04-03T16:23:33.5602415Z NV_NVML_DEV_VERSION=11.8.86-1 2024-04-03T16:23:33.5603509Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:23:33.5604394Z CONTINUE_THROUGH_ERROR=False 2024-04-03T16:23:33.5605212Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-04-03T16:23:33.5606131Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-04-03T16:23:33.5606874Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-04-03T16:23:33.5607814Z HOSTNAME=b758c44c85ba 2024-04-03T16:23:33.5609706Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_5f7e7b29-9ba5-4a89-a5f0-6f545ab3aae6 2024-04-03T16:23:33.5611171Z GITHUB_ACTION=__self 2024-04-03T16:23:33.5611835Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-04-03T16:23:33.5615285Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-04-03T16:23:33.5619108Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-04-03T16:23:33.5619771Z NV_NVTX_VERSION=11.8.86-1 2024-04-03T16:23:33.5620471Z GITHUB_RUN_NUMBER=192590 2024-04-03T16:23:33.5620915Z TEST_CONFIG=distributed 2024-04-03T16:23:33.5621550Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-04-03T16:23:33.5622219Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-04-03T16:23:33.5623122Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-04-03T16:23:33.5623781Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-04-03T16:23:33.5624488Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-04-03T16:23:33.5625065Z GITHUB_TRIGGERING_ACTOR=soulitzer 2024-04-03T16:23:33.5625794Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-04-03T16:23:33.5626501Z GITHUB_REF_TYPE=branch 2024-04-03T16:23:33.5626967Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-04-03T16:23:33.5627691Z NCCL_VERSION=2.16.2-1 2024-04-03T16:23:33.5628162Z BASE_SHA=1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:23:33.5629006Z XLA_CUDA= 2024-04-03T16:23:33.5629362Z HUGGING_FACE_HUB_TOKEN= 2024-04-03T16:23:33.5632043Z *** 2024-04-03T16:23:33.5632400Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-04-03T16:23:33.5633203Z GITHUB_REPOSITORY_ID=65600975 2024-04-03T16:23:33.5633682Z GITHUB_ACTIONS=true 2024-04-03T16:23:33.5634209Z NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:33.5635016Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-04-03T16:23:33.5635975Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-04-03T16:23:33.5636582Z SHA1=d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:23:33.5637523Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-04-03T16:23:33.5638609Z GITHUB_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:23:33.5639602Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/123154/merge 2024-04-03T16:23:33.5640695Z UCC_HOME=/usr 2024-04-03T16:23:33.5641276Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-04-03T16:23:33.5641900Z VERBOSE_TEST_LOGS=False 2024-04-03T16:23:33.5642334Z NVIDIA_PRODUCT_NAME=CUDA 2024-04-03T16:23:33.5643205Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-04-03T16:23:33.5643798Z GITHUB_REF=refs/pull/123154/merge 2024-04-03T16:23:33.5644597Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-04-03T16:23:33.5645072Z SHARD_NUMBER=1 2024-04-03T16:23:33.5645687Z GITHUB_REF_PROTECTED=false 2024-04-03T16:23:33.5646157Z HOME=/var/lib/jenkins 2024-04-03T16:23:33.5646654Z GITHUB_API_URL=https://api.github.com 2024-04-03T16:23:33.5647445Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-04-03T16:23:33.5648139Z UCX_COMMIT=00bcc6bb18fc282eb160623b4c0d300147f579af 2024-04-03T16:23:33.5648932Z SCCACHE_S3_KEY_PREFIX=pull 2024-04-03T16:23:33.5649920Z CUDA_VERSION=11.8.0 2024-04-03T16:23:33.5650499Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-04-03T16:23:33.5651289Z NUM_TEST_SHARDS=3 2024-04-03T16:23:33.5651691Z UCX_HOME=/usr 2024-04-03T16:23:33.5652476Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-04-03T16:23:33.5654297Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_5f7e7b29-9ba5-4a89-a5f0-6f545ab3aae6 2024-04-03T16:23:33.5656144Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:23:33.5658254Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_5f7e7b29-9ba5-4a89-a5f0-6f545ab3aae6 2024-04-03T16:23:33.5660158Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-04-03T16:23:33.5661330Z GITHUB_EVENT_NAME=pull_request 2024-04-03T16:23:33.5661801Z DASHBOARD_TAG= 2024-04-03T16:23:33.5662424Z GITHUB_RUN_ID=8536950919 2024-04-03T16:23:33.5663024Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-04-03T16:23:33.5664057Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-04-03T16:23:33.5665644Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_5f7e7b29-9ba5-4a89-a5f0-6f545ab3aae6 2024-04-03T16:23:33.5667039Z GITHUB_ACTOR=caic99 2024-04-03T16:23:33.5667529Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-04-03T16:23:33.5668252Z PR_NUMBER=123154 2024-04-03T16:23:33.5668620Z GITHUB_RUN_ATTEMPT=2 2024-04-03T16:23:33.5669234Z ANACONDA_PYTHON_VERSION=3.10 2024-04-03T16:23:33.5669880Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-04-03T16:23:33.5670443Z TERM=xterm 2024-04-03T16:23:33.5671131Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-04-03T16:23:33.5671666Z INSTALLED_VISION=yes 2024-04-03T16:23:33.5672058Z BRANCH=pull/123154 2024-04-03T16:23:33.5672464Z OPENSSL_ROOT_DIR=/opt/openssl 2024-04-03T16:23:33.5672968Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-04-03T16:23:33.5673799Z CUDA_PATH=/usr/local/cuda 2024-04-03T16:23:33.5674771Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-04-03T16:23:33.5675735Z GITHUB_SERVER_URL=https://github.com 2024-04-03T16:23:33.5676331Z UCC_COMMIT=7cb07a76ccedad7e56ceb136b865eb9319c258ea 2024-04-03T16:23:33.5676927Z REENABLED_ISSUES= 2024-04-03T16:23:33.5677418Z SHLVL=1 2024-04-03T16:23:33.5677882Z MAX_JOBS=30 2024-04-03T16:23:33.5678283Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-04-03T16:23:33.5678731Z NVARCH=x86_64 2024-04-03T16:23:33.5679082Z GITHUB_ACTOR_ID=78061359 2024-04-03T16:23:33.5679666Z GITHUB_WORKFLOW_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:23:33.5680332Z GITHUB_REF_NAME=123154/merge 2024-04-03T16:23:33.5680858Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-04-03T16:23:33.5681734Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-04-03T16:23:33.5682463Z GITHUB_JOB=test 2024-04-03T16:23:33.5682963Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-04-03T16:23:33.5683854Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-04-03T16:23:33.5684528Z NO_TEST_TIMEOUT=False 2024-04-03T16:23:33.5685301Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-04-03T16:23:33.5685840Z GITHUB_REPOSITORY=pytorch/pytorch 2024-04-03T16:23:33.5686367Z NV_NVPROF_VERSION=11.8.87-1 2024-04-03T16:23:33.5686819Z GITHUB_RETENTION_DAYS=90 2024-04-03T16:23:33.5687241Z OPENSSL_DIR=/opt/openssl 2024-04-03T16:23:33.5687682Z GITHUB_ACTION_REPOSITORY= 2024-04-03T16:23:33.5689018Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:23:33.5690580Z GITHUB_BASE_REF=main 2024-04-03T16:23:33.5691010Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-04-03T16:23:33.5691475Z CI=true 2024-04-03T16:23:33.5691860Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-04-03T16:23:33.5692384Z GITHUB_REPOSITORY_OWNER=pytorch 2024-04-03T16:23:33.5692855Z JOB_ID=23402705639 2024-04-03T16:23:33.5693235Z INSTALLED_PROTOBUF=yes 2024-04-03T16:23:33.5693689Z GITHUB_HEAD_REF=patch-1 2024-04-03T16:23:33.5694104Z GITHUB_ACTION_REF= 2024-04-03T16:23:33.5694631Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-04-03T16:23:33.5695220Z GITHUB_WORKFLOW=pull 2024-04-03T16:23:33.5695647Z DEBIAN_FRONTEND=noninteractive 2024-04-03T16:23:33.5696783Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_5f7e7b29-9ba5-4a89-a5f0-6f545ab3aae6 2024-04-03T16:23:33.5697831Z NO_TD=False 2024-04-03T16:23:33.5698213Z SKIP_SCCACHE_INITIALIZATION=1 2024-04-03T16:23:33.5698953Z _=/usr/bin/env 2024-04-03T16:23:33.5699566Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-04-03T16:23:33.5822366Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-04-03T16:23:33.5823955Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-04-03T16:23:33.5825482Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-04-03T16:23:33.5826948Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-04-03T16:23:33.5827814Z + BUILD_DIR=build 2024-04-03T16:23:33.5828441Z + BUILD_RENAMED_DIR=build_renamed 2024-04-03T16:23:33.5828991Z + BUILD_BIN_DIR=build/bin 2024-04-03T16:23:33.5829430Z + SHARD_NUMBER=1 2024-04-03T16:23:33.5830086Z + NUM_TEST_SHARDS=3 2024-04-03T16:23:33.5830431Z + export VALGRIND=ON 2024-04-03T16:23:33.5830906Z + VALGRIND=ON 2024-04-03T16:23:33.5831643Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-04-03T16:23:33.5832323Z + [[ 0 == \1 ]] 2024-04-03T16:23:33.5832868Z + [[ False == \1 ]] 2024-04-03T16:23:33.5833430Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-04-03T16:23:33.5834379Z ++ realpath build/custom_test_artifacts 2024-04-03T16:23:33.5838597Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-04-03T16:23:33.5839765Z + [[ -n '' ]] 2024-04-03T16:23:33.5841615Z ++ dirname .ci/pytorch/test.sh 2024-04-03T16:23:33.5849512Z + source .ci/pytorch/common.sh 2024-04-03T16:23:33.5853224Z +++ dirname .ci/pytorch/common.sh 2024-04-03T16:23:33.5861265Z ++ source .ci/pytorch/common_utils.sh 2024-04-03T16:23:33.5863266Z +++ declare -f -t trap_add 2024-04-03T16:23:33.5870171Z ++ set -ex 2024-04-03T16:23:33.5870957Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-04-03T16:23:33.5871555Z ++ BUILD_TEST_LIBTORCH=0 2024-04-03T16:23:33.5872284Z + echo 'Environment variables' 2024-04-03T16:23:33.5872770Z Environment variables 2024-04-03T16:23:33.5873407Z + env 2024-04-03T16:23:33.5878313Z INSTALLED_DB=yes 2024-04-03T16:23:33.5879207Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-04-03T16:23:33.5879695Z NVIDIA_VISIBLE_DEVICES=all 2024-04-03T16:23:33.5880344Z NV_NVML_DEV_VERSION=11.8.86-1 2024-04-03T16:23:33.5881212Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:23:33.5882241Z CONTINUE_THROUGH_ERROR=False 2024-04-03T16:23:33.5882852Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-04-03T16:23:33.5884161Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-04-03T16:23:33.5885171Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-04-03T16:23:33.5885781Z HOSTNAME=b758c44c85ba 2024-04-03T16:23:33.5887205Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_5f7e7b29-9ba5-4a89-a5f0-6f545ab3aae6 2024-04-03T16:23:33.5888514Z GITHUB_ACTION=__self 2024-04-03T16:23:33.5889045Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-04-03T16:23:33.5892718Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-04-03T16:23:33.5895896Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-04-03T16:23:33.5896855Z NV_NVTX_VERSION=11.8.86-1 2024-04-03T16:23:33.5897315Z GITHUB_RUN_NUMBER=192590 2024-04-03T16:23:33.5897736Z TEST_CONFIG=distributed 2024-04-03T16:23:33.5898185Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-04-03T16:23:33.5898778Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-04-03T16:23:33.5899345Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-04-03T16:23:33.5899943Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-04-03T16:23:33.5900774Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-04-03T16:23:33.5901248Z GITHUB_TRIGGERING_ACTOR=soulitzer 2024-04-03T16:23:33.5901832Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-04-03T16:23:33.5902433Z GITHUB_REF_TYPE=branch 2024-04-03T16:23:33.5902847Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-04-03T16:23:33.5903328Z NCCL_VERSION=2.16.2-1 2024-04-03T16:23:33.5903818Z BASE_SHA=1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:23:33.5904529Z XLA_CUDA= 2024-04-03T16:23:33.5904911Z HUGGING_FACE_HUB_TOKEN= 2024-04-03T16:23:33.5905413Z *** 2024-04-03T16:23:33.5905760Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-04-03T16:23:33.5906251Z GITHUB_REPOSITORY_ID=65600975 2024-04-03T16:23:33.5906713Z GITHUB_ACTIONS=true 2024-04-03T16:23:33.5907133Z NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T16:23:33.5908035Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-04-03T16:23:33.5908729Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-04-03T16:23:33.5909328Z SHA1=d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:23:33.5909956Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-04-03T16:23:33.5910593Z GITHUB_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:23:33.5911509Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/123154/merge 2024-04-03T16:23:33.5912304Z UCC_HOME=/usr 2024-04-03T16:23:33.5912731Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-04-03T16:23:33.5913243Z VERBOSE_TEST_LOGS=False 2024-04-03T16:23:33.5913667Z NVIDIA_PRODUCT_NAME=CUDA 2024-04-03T16:23:33.5914246Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-04-03T16:23:33.5914899Z GITHUB_REF=refs/pull/123154/merge 2024-04-03T16:23:33.5915417Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-04-03T16:23:33.5915891Z SHARD_NUMBER=1 2024-04-03T16:23:33.5916277Z GITHUB_REF_PROTECTED=false 2024-04-03T16:23:33.5916702Z HOME=/var/lib/jenkins 2024-04-03T16:23:33.5917163Z GITHUB_API_URL=https://api.github.com 2024-04-03T16:23:33.5917720Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-04-03T16:23:33.5918304Z UCX_COMMIT=00bcc6bb18fc282eb160623b4c0d300147f579af 2024-04-03T16:23:33.5918912Z SCCACHE_S3_KEY_PREFIX=pull 2024-04-03T16:23:33.5919353Z CUDA_VERSION=11.8.0 2024-04-03T16:23:33.5919867Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-04-03T16:23:33.5920430Z NUM_TEST_SHARDS=3 2024-04-03T16:23:33.5920808Z UCX_HOME=/usr 2024-04-03T16:23:33.5921731Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-04-03T16:23:33.5923152Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_5f7e7b29-9ba5-4a89-a5f0-6f545ab3aae6 2024-04-03T16:23:33.5924715Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-04-03T16:23:33.5926423Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_5f7e7b29-9ba5-4a89-a5f0-6f545ab3aae6 2024-04-03T16:23:33.5927858Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-04-03T16:23:33.5928713Z GITHUB_EVENT_NAME=pull_request 2024-04-03T16:23:33.5929418Z DASHBOARD_TAG= 2024-04-03T16:23:33.5929771Z GITHUB_RUN_ID=8536950919 2024-04-03T16:23:33.5930353Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-04-03T16:23:33.5931013Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-04-03T16:23:33.5932290Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_5f7e7b29-9ba5-4a89-a5f0-6f545ab3aae6 2024-04-03T16:23:33.5933396Z GITHUB_ACTOR=caic99 2024-04-03T16:23:33.5933850Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-04-03T16:23:33.5934318Z PR_NUMBER=123154 2024-04-03T16:23:33.5934709Z GITHUB_RUN_ATTEMPT=2 2024-04-03T16:23:33.5935106Z VALGRIND=ON 2024-04-03T16:23:33.5935472Z ANACONDA_PYTHON_VERSION=3.10 2024-04-03T16:23:33.5936040Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-04-03T16:23:33.5936609Z TERM=xterm 2024-04-03T16:23:33.5937025Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-04-03T16:23:33.5937544Z INSTALLED_VISION=yes 2024-04-03T16:23:33.5937936Z BRANCH=pull/123154 2024-04-03T16:23:33.5938350Z OPENSSL_ROOT_DIR=/opt/openssl 2024-04-03T16:23:33.5938856Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-04-03T16:23:33.5939374Z CUDA_PATH=/usr/local/cuda 2024-04-03T16:23:33.5940337Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-04-03T16:23:33.5941304Z GITHUB_SERVER_URL=https://github.com 2024-04-03T16:23:33.5941922Z UCC_COMMIT=7cb07a76ccedad7e56ceb136b865eb9319c258ea 2024-04-03T16:23:33.5942503Z REENABLED_ISSUES= 2024-04-03T16:23:33.5943014Z SHLVL=1 2024-04-03T16:23:33.5943342Z MAX_JOBS=30 2024-04-03T16:23:33.5943776Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-04-03T16:23:33.5944225Z NVARCH=x86_64 2024-04-03T16:23:33.5944612Z GITHUB_ACTOR_ID=78061359 2024-04-03T16:23:33.5945182Z GITHUB_WORKFLOW_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:23:33.5945846Z GITHUB_REF_NAME=123154/merge 2024-04-03T16:23:33.5946396Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-04-03T16:23:33.5947221Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-04-03T16:23:33.5948208Z GITHUB_JOB=test 2024-04-03T16:23:33.5948773Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-04-03T16:23:33.5949462Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-04-03T16:23:33.5950113Z NO_TEST_TIMEOUT=False 2024-04-03T16:23:33.5950600Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-04-03T16:23:33.5951141Z GITHUB_REPOSITORY=pytorch/pytorch 2024-04-03T16:23:33.5951673Z NV_NVPROF_VERSION=11.8.87-1 2024-04-03T16:23:33.5952143Z GITHUB_RETENTION_DAYS=90 2024-04-03T16:23:33.5952565Z OPENSSL_DIR=/opt/openssl 2024-04-03T16:23:33.5953011Z GITHUB_ACTION_REPOSITORY= 2024-04-03T16:23:33.5954362Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:23:33.5955728Z GITHUB_BASE_REF=main 2024-04-03T16:23:33.5956157Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-04-03T16:23:33.5956627Z CI=true 2024-04-03T16:23:33.5957006Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-04-03T16:23:33.5957537Z GITHUB_REPOSITORY_OWNER=pytorch 2024-04-03T16:23:33.5958009Z JOB_ID=23402705639 2024-04-03T16:23:33.5958389Z INSTALLED_PROTOBUF=yes 2024-04-03T16:23:33.5958836Z GITHUB_HEAD_REF=patch-1 2024-04-03T16:23:33.5959260Z GITHUB_ACTION_REF= 2024-04-03T16:23:33.5959789Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-04-03T16:23:33.5960387Z GITHUB_WORKFLOW=pull 2024-04-03T16:23:33.5960820Z DEBIAN_FRONTEND=noninteractive 2024-04-03T16:23:33.5961972Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_5f7e7b29-9ba5-4a89-a5f0-6f545ab3aae6 2024-04-03T16:23:33.5963020Z NO_TD=False 2024-04-03T16:23:33.5963561Z SKIP_SCCACHE_INITIALIZATION=1 2024-04-03T16:23:33.5964006Z _=/usr/bin/env 2024-04-03T16:23:33.5964435Z + echo 'Testing pytorch' 2024-04-03T16:23:33.5964870Z Testing pytorch 2024-04-03T16:23:33.5965260Z + export LANG=C.UTF-8 2024-04-03T16:23:33.5965687Z + LANG=C.UTF-8 2024-04-03T16:23:33.5966061Z + PR_NUMBER=123154 2024-04-03T16:23:33.5966468Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-04-03T16:23:33.5967039Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-04-03T16:23:33.5967753Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-04-03T16:23:33.5968347Z + [[ distributed == \s\l\o\w ]] 2024-04-03T16:23:33.5969049Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-04-03T16:23:33.5970144Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-04-03T16:23:33.5970798Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-04-03T16:23:33.5971398Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-04-03T16:23:33.5971942Z + [[ distributed == *crossref* ]] 2024-04-03T16:23:33.5972575Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-04-03T16:23:33.5973307Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-04-03T16:23:33.5974066Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-04-03T16:23:33.5974721Z + pip_install --user ninja==1.10.2 2024-04-03T16:23:33.5975392Z + pip install --progress-bar off --user ninja==1.10.2 2024-04-03T16:23:34.1069099Z Collecting ninja==1.10.2 2024-04-03T16:23:34.1253026Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-04-03T16:23:34.1433021Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-04-03T16:23:35.1653397Z Installing collected packages: ninja 2024-04-03T16:23:35.1752665Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-04-03T16:23:35.1754345Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-04-03T16:23:35.1817666Z Successfully installed ninja-1.10.2 2024-04-03T16:23:35.2535393Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:23:35.2538127Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:23:35.2539716Z + install_tlparse 2024-04-03T16:23:35.2540214Z + pip_install --user tlparse==0.3.7 2024-04-03T16:23:35.2540909Z + pip install --progress-bar off --user tlparse==0.3.7 2024-04-03T16:23:35.8046849Z Collecting tlparse==0.3.7 2024-04-03T16:23:35.8238040Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-04-03T16:23:35.8663343Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-04-03T16:23:36.9541759Z Installing collected packages: tlparse 2024-04-03T16:23:37.0043336Z Successfully installed tlparse-0.3.7 2024-04-03T16:23:37.0750024Z ++ python -m site --user-base 2024-04-03T16:23:37.0976591Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:23:37.0978979Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-04-03T16:23:37.0979780Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-04-03T16:23:37.0980639Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-04-03T16:23:37.0981749Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-04-03T16:23:37.0983137Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-04-03T16:23:37.0983957Z + cd test 2024-04-03T16:23:37.0984618Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-04-03T16:23:38.9305818Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-04-03T16:23:38.9306499Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-04-03T16:23:38.9310071Z + DYNAMO_BENCHMARK_FLAGS=() 2024-04-03T16:23:38.9310714Z + [[ distributed == *dynamo_eager* ]] 2024-04-03T16:23:38.9311232Z + [[ distributed == *aot_eager* ]] 2024-04-03T16:23:38.9311704Z + [[ distributed == *aot_inductor* ]] 2024-04-03T16:23:38.9312239Z + [[ distributed == *inductor* ]] 2024-04-03T16:23:38.9312733Z + [[ distributed == *dynamic* ]] 2024-04-03T16:23:38.9313252Z + [[ distributed == *cpu_inductor* ]] 2024-04-03T16:23:38.9313975Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-04-03T16:23:38.9326591Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *tbb* ]] 2024-04-03T16:23:38.9345431Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-04-03T16:23:38.9346152Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-04-03T16:23:38.9348249Z + cd test 2024-04-03T16:23:38.9348882Z + python -c 'import torch; print(torch.__config__.show())' 2024-04-03T16:23:40.5599686Z PyTorch built with: 2024-04-03T16:23:40.5600949Z - GCC 9.4 2024-04-03T16:23:40.5601835Z - C++ Version: 201703 2024-04-03T16:23:40.5604324Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-04-03T16:23:40.5607546Z - Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361) 2024-04-03T16:23:40.5609994Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-04-03T16:23:40.5611480Z - LAPACK is enabled (usually provided by MKL) 2024-04-03T16:23:40.5612869Z - NNPACK is enabled 2024-04-03T16:23:40.5613968Z - CPU capability usage: AVX2 2024-04-03T16:23:40.5615259Z - CUDA Runtime 11.8 2024-04-03T16:23:40.5617275Z - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52 2024-04-03T16:23:40.5618940Z - CuDNN 8.7 2024-04-03T16:23:40.5619809Z - Magma 2.6.1 2024-04-03T16:23:40.5642941Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-04-03T16:23:40.5661436Z 2024-04-03T16:23:40.8099653Z + cd test 2024-04-03T16:23:40.8100529Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-04-03T16:23:42.4061177Z ATen/Parallel: 2024-04-03T16:23:42.4061792Z at::get_num_threads() : 16 2024-04-03T16:23:42.4062332Z at::get_num_interop_threads() : 32 2024-04-03T16:23:42.4062871Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-04-03T16:23:42.4063279Z omp_get_max_threads() : 16 2024-04-03T16:23:42.4064470Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-04-03T16:23:42.4081658Z mkl_get_max_threads() : 16 2024-04-03T16:23:42.4082531Z Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361) 2024-04-03T16:23:42.4083663Z std::thread::hardware_concurrency() : 32 2024-04-03T16:23:42.4084212Z Environment variables: 2024-04-03T16:23:42.4084654Z OMP_NUM_THREADS : [not set] 2024-04-03T16:23:42.4085181Z MKL_NUM_THREADS : [not set] 2024-04-03T16:23:42.4085585Z ATen parallel backend: OpenMP 2024-04-03T16:23:42.4085910Z 2024-04-03T16:23:42.6390218Z + [[ distributed == *backward* ]] 2024-04-03T16:23:42.6390850Z + [[ distributed == *xla* ]] 2024-04-03T16:23:42.6391252Z + [[ distributed == *executorch* ]] 2024-04-03T16:23:42.6391852Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-04-03T16:23:42.6392653Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-04-03T16:23:42.6393309Z + [[ distributed == distributed ]] 2024-04-03T16:23:42.6393785Z + test_distributed 2024-04-03T16:23:42.6394297Z + echo 'Testing distributed python tests' 2024-04-03T16:23:42.6394882Z Testing distributed python tests 2024-04-03T16:23:42.6395638Z + python test/run_test.py --distributed-tests --shard 1 3 --verbose 2024-04-03T16:23:42.7516190Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-04-03T16:23:42.7517632Z import pkg_resources 2024-04-03T16:23:45.8674253Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-04-03T16:23:45.9455902Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-04-03T16:23:46.0016707Z Ignoring disabled issues: [''] 2024-04-03T16:23:46.0104171Z Found test times from artifacts 2024-04-03T16:23:46.0439378Z Found test times from artifacts 2024-04-03T16:23:46.0450023Z Running all tests 2024-04-03T16:23:46.0587685Z Name: tests to run (est. time: 72.36min) 2024-04-03T16:23:46.0588591Z Serial tests (73): 2024-04-03T16:23:46.0589149Z distributed/_tensor/debug/test_op_coverage 1/1 2024-04-03T16:23:46.0589791Z distributed/_spmd/test_graph_utils 1/1 2024-04-03T16:23:46.0590428Z distributed/_composable/test_contract 1/1 2024-04-03T16:23:46.0591024Z distributed/checkpoint/test_traverse 1/1 2024-04-03T16:23:46.0591593Z distributed/fsdp/test_fsdp_fx 1/1 2024-04-03T16:23:46.0592231Z distributed/_composable/fsdp/test_fully_shard_state 1/1 2024-04-03T16:23:46.0592955Z distributed/checkpoint/test_dedup_tensors 1/1 2024-04-03T16:23:46.0593626Z distributed/checkpoint/test_compatibility 1/1 2024-04-03T16:23:46.0594265Z distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 2024-04-03T16:23:46.0594908Z distributed/_tensor/test_common_rules 1/1 2024-04-03T16:23:46.0595534Z distributed/test_c10d_functional_native 1/1 2024-04-03T16:23:46.0596140Z distributed/_tensor/test_dtensor_ops 1/1 2024-04-03T16:23:46.0596755Z distributed/test_distributed_spawn 2/7 2024-04-03T16:23:46.0597348Z distributed/test_distributed_spawn 5/7 2024-04-03T16:23:46.0597960Z distributed/checkpoint/test_tp_checkpoint 1/1 2024-04-03T16:23:46.0598625Z distributed/_tensor/test_tensor_ops 1/1 2024-04-03T16:23:46.0599307Z distributed/_composable/fsdp/test_fully_shard_init 1/1 2024-04-03T16:23:46.0600012Z distributed/_tensor/test_xla_integration 1/1 2024-04-03T16:23:46.0600638Z distributed/checkpoint/test_format_utils 1/1 2024-04-03T16:23:46.0601269Z distributed/test_dynamo_distributed 1/1 2024-04-03T16:23:46.0601846Z distributed/test_device_mesh 1/1 2024-04-03T16:23:46.0602485Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 2024-04-03T16:23:46.0603201Z distributed/_tools/test_memory_tracker 1/1 2024-04-03T16:23:46.0603828Z distributed/_tensor/test_pointwise_ops 1/1 2024-04-03T16:23:46.0604459Z distributed/tensor/parallel/test_tp_style 1/1 2024-04-03T16:23:46.0605160Z distributed/checkpoint/test_dtensor_checkpoint 1/1 2024-04-03T16:23:46.0605834Z distributed/_tensor/test_op_strategy 1/1 2024-04-03T16:23:46.0606516Z distributed/_composable/fsdp/test_fully_shard_memory 1/1 2024-04-03T16:23:46.0607440Z distributed/checkpoint/e2e/test_pipeline 1/1 2024-04-03T16:23:46.0608082Z distributed/_tensor/test_embedding_ops 1/1 2024-04-03T16:23:46.0608717Z distributed/checkpoint/test_hsdp_checkpoint 1/1 2024-04-03T16:23:46.0609682Z distributed/_spmd/test_tracing 1/1 2024-04-03T16:23:46.0610290Z distributed/checkpoint/e2e/test_fine_tuning 1/1 2024-04-03T16:23:46.0610915Z distributed/_tensor/test_matrix_ops 1/1 2024-04-03T16:23:46.0611709Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 1/1 2024-04-03T16:23:46.0612647Z distributed/_composable/fully_shard/test_fully_shard_compile 1/1 2024-04-03T16:23:46.0613432Z distributed/checkpoint/test_fsdp_model_state 1/1 2024-04-03T16:23:46.0614120Z distributed/fsdp/test_shard_utils 1/1 2024-04-03T16:23:46.0614812Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-04-03T16:23:46.0615507Z distributed/fsdp/test_fsdp_pure_fp16 1/1 2024-04-03T16:23:46.0616136Z distributed/fsdp/test_fsdp_optim_state 1/1 2024-04-03T16:23:46.0616787Z distributed/fsdp/test_checkpoint_wrapper 1/1 2024-04-03T16:23:46.0617417Z distributed/fsdp/test_fsdp_grad_acc 1/1 2024-04-03T16:23:46.0618051Z distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 2024-04-03T16:23:46.0618721Z distributed/fsdp/test_fsdp_state_dict 2/2 2024-04-03T16:23:46.0619507Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 2024-04-03T16:23:46.0620083Z distributed/fsdp/test_wrap 1/1 2024-04-03T16:23:46.0620745Z distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 2024-04-03T16:23:46.0621473Z distributed/fsdp/test_fsdp_comm_hooks 1/1 2024-04-03T16:23:46.0622046Z distributed/fsdp/test_fsdp_meta 1/1 2024-04-03T16:23:46.0622656Z distributed/fsdp/test_fsdp_use_orig_params 1/1 2024-04-03T16:23:46.0623479Z distributed/fsdp/test_fsdp_multiple_forward 1/1 2024-04-03T16:23:46.0624104Z distributed/fsdp/test_fsdp_comm 1/1 2024-04-03T16:23:46.0624664Z distributed/fsdp/test_fsdp_misc 1/1 2024-04-03T16:23:46.0625280Z distributed/pipeline/sync/test_microbatch 1/1 2024-04-03T16:23:46.0625923Z distributed/pipeline/sync/test_checkpoint 1/1 2024-04-03T16:23:46.0626589Z distributed/pipeline/sync/skip/test_portal 1/1 2024-04-03T16:23:46.0627287Z distributed/_shard/sharded_tensor/ops/test_init 1/1 2024-04-03T16:23:46.0627945Z distributed/rpc/test_tensorpipe_agent 1/1 2024-04-03T16:23:46.0628564Z distributed/pipeline/sync/test_copy 1/1 2024-04-03T16:23:46.0629202Z distributed/fsdp/test_fsdp_mixed_precision 1/1 2024-04-03T16:23:46.0629918Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 2024-04-03T16:23:46.0630655Z distributed/pipeline/sync/skip/test_tracker 1/1 2024-04-03T16:23:46.0631336Z distributed/pipeline/sync/skip/test_leak 1/1 2024-04-03T16:23:46.0632006Z distributed/pipeline/sync/skip/test_stash_pop 1/1 2024-04-03T16:23:46.0632652Z distributed/test_c10d_spawn_gloo 1/1 2024-04-03T16:23:46.0633199Z distributed/test_c10d_ucc 1/1 2024-04-03T16:23:46.0633921Z distributed/pipeline/sync/skip/test_inspect_skip_layout 1/1 2024-04-03T16:23:46.0634654Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 2024-04-03T16:23:46.0635314Z distributed/test_launcher 1/1 2024-04-03T16:23:46.0635920Z distributed/optim/test_zero_redundancy_optimizer 1/1 2024-04-03T16:23:46.0636553Z distributed/test_c10d_nccl 1/2 2024-04-03T16:23:46.0637097Z distributed/elastic/timer/api_test 1/1 2024-04-03T16:23:46.0637629Z Parallel tests (0): 2024-04-03T16:23:46.0638078Z Name: excluded (est. time: 0.0min) 2024-04-03T16:23:46.0638572Z Serial tests (0): 2024-04-03T16:23:46.0638959Z Parallel tests (0): 2024-04-03T16:23:46.0639725Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-04-03T16:23:46.0658256Z Running distributed/_tensor/debug/test_op_coverage 1/1 ... [2024-04-03 16:23:46.065473] 2024-04-03T16:23:46.0665357Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/debug/test_op_coverage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:23:46.066059] 2024-04-03T16:23:49.8888123Z 2024-04-03T16:23:49.8890786Z distributed/_tensor/debug/test_op_coverage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_1a50b1bcadad1c56_.log 2024-04-03T16:23:49.8896139Z Running 1 items in this shard: test/distributed/_tensor/debug/test_op_coverage.py::TestOpCoverage::test_trace_with_inductor_decomp 2024-04-03T16:23:49.8897398Z 2024-04-03T16:23:49.8898021Z Running distributed/_spmd/test_graph_utils 1/1 ... [2024-04-03 16:23:49.889238] 2024-04-03T16:23:49.8900641Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_spmd/test_graph_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:23:49.889643] 2024-04-03T16:23:52.8619484Z 2024-04-03T16:23:52.8621526Z distributed/_spmd/test_graph_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._spmd.test_graph_utils_1.1_0271ed49474f7cc1_.log 2024-04-03T16:23:52.8622935Z 2024-04-03T16:23:52.8625288Z Running distributed/_composable/test_contract 1/1 ... [2024-04-03 16:23:52.862175] 2024-04-03T16:23:52.8630022Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_contract.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:23:52.862587] 2024-04-03T16:23:57.4867082Z 2024-04-03T16:23:57.4870118Z distributed/_composable/test_contract 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_contract_1.1_cbebec6dfee90cd3_.log 2024-04-03T16:23:57.4874534Z Running 4 items in this shard: test/distributed/_composable/test_contract.py::TestContract::test_add_hooks, test/distributed/_composable/test_contract.py::TestContract::test_modify_fqn, test/distributed/_composable/test_contract.py::TestContract::test_registry, test/distributed/_composable/test_contract.py::TestContract::test_state 2024-04-03T16:23:57.4877639Z 2024-04-03T16:23:57.4878243Z Running distributed/checkpoint/test_traverse 1/1 ... [2024-04-03 16:23:57.486872] 2024-04-03T16:23:57.4880699Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_traverse.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:23:57.487278] 2024-04-03T16:24:01.1093184Z 2024-04-03T16:24:01.1095175Z distributed/checkpoint/test_traverse 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_traverse_1.1_24daab968d1b6f89_.log 2024-04-03T16:24:01.1100739Z Running 7 items in this shard: test/distributed/checkpoint/test_traverse.py::TestTraverse::test_get_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_set_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_doesnt_ignore_intermediate_collections, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_dict, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_list, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_shallow, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_with_ordered_dict 2024-04-03T16:24:01.1104996Z 2024-04-03T16:24:01.1105491Z Running distributed/fsdp/test_fsdp_fx 1/1 ... [2024-04-03 16:24:01.109684] 2024-04-03T16:24:01.1115904Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:24:01.111137] 2024-04-03T16:24:04.7332063Z 2024-04-03T16:24:04.7333982Z distributed/fsdp/test_fsdp_fx 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_fx_1.1_9ea34eeace00f196_.log 2024-04-03T16:24:04.7336035Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_fx.py::TestSymbolicTracing::test_symbolic_tracing_outputs 2024-04-03T16:24:04.7336968Z 2024-04-03T16:24:04.7337595Z Running distributed/_composable/fsdp/test_fully_shard_state 1/1 ... [2024-04-03 16:24:04.733471] 2024-04-03T16:24:04.7343417Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:24:04.733891] 2024-04-03T16:24:08.5560127Z 2024-04-03T16:24:08.5562467Z distributed/_composable/fsdp/test_fully_shard_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_bb6c00d2dd342206_.log 2024-04-03T16:24:08.5567665Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_cls, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_deepcopy, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_reapply, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_state, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_unsupported_module_cls 2024-04-03T16:24:08.5573679Z 2024-04-03T16:24:08.5574286Z Running distributed/checkpoint/test_dedup_tensors 1/1 ... [2024-04-03 16:24:08.556289] 2024-04-03T16:24:08.5576840Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dedup_tensors.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:24:08.556746] 2024-04-03T16:24:12.1787314Z 2024-04-03T16:24:12.1789520Z distributed/checkpoint/test_dedup_tensors 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_0363f49263bc932a_.log 2024-04-03T16:24:12.1791815Z Running 1 items in this shard: test/distributed/checkpoint/test_dedup_tensors.py::TestDedupTensor::test_dedup_shards 2024-04-03T16:24:12.1792634Z 2024-04-03T16:24:12.1793239Z Running distributed/checkpoint/test_compatibility 1/1 ... [2024-04-03 16:24:12.178952] 2024-04-03T16:24:12.1798199Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_compatibility.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:24:12.179363] 2024-04-03T16:24:15.9018130Z 2024-04-03T16:24:15.9021105Z distributed/checkpoint/test_compatibility 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_compatibility_1.1_c8765490a36f84fb_.log 2024-04-03T16:24:15.9024899Z Running 2 items in this shard: test/distributed/checkpoint/test_compatibility.py::TestDCPCompatbility::test_metadata, test/distributed/checkpoint/test_compatibility.py::TestDCPCompatbility::test_sharded_tensor_dependency 2024-04-03T16:24:15.9026558Z 2024-04-03T16:24:15.9027834Z Running distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 ... [2024-04-03 16:24:15.902024] 2024-04-03T16:24:15.9030553Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/fsdp/test_fsdp_dsd.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:24:15.902433] 2024-04-03T16:24:29.9463868Z 2024-04-03T16:24:29.9466153Z distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.fsdp.test_fsdp_dsd_1.1_d2f8c7f0e958b7a5_.log 2024-04-03T16:24:29.9470132Z Running 2 items in this shard: test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_1d_fsdp_cpu_offload_full_model_state_dict, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_1d_fsdp_get_model_state_dict 2024-04-03T16:24:29.9472555Z 2024-04-03T16:24:29.9473233Z Running distributed/_tensor/test_common_rules 1/1 ... [2024-04-03 16:24:29.946633] 2024-04-03T16:24:29.9475739Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_common_rules.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:24:29.947042] 2024-04-03T16:25:07.3988487Z 2024-04-03T16:25:07.3990337Z distributed/_tensor/test_common_rules 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_common_rules_1.1_ee667e9ed72dbc56_.log 2024-04-03T16:25:07.4000814Z Running 10 items in this shard: test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_basic_propagation, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_errors, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_linearity, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_merge_sharding, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_pointwise_propagation, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_enforce_sharding_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_broadcasting, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_suggestion 2024-04-03T16:25:07.4007174Z 2024-04-03T16:25:07.4007839Z Running distributed/test_c10d_functional_native 1/1 ... [2024-04-03 16:25:07.399118] 2024-04-03T16:25:07.4010596Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_functional_native.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:25:07.399535] 2024-04-03T16:25:56.1663229Z 2024-04-03T16:25:56.1665263Z distributed/test_c10d_functional_native 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_functional_native_1.1_8091fbb27d8a0715_.log 2024-04-03T16:25:56.1679890Z Running 23 items in this shard: test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_to_all_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_broadcast, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_threading, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_unwaited, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_to_all_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_broadcast, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_inplace_op_on_view, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reuse_buffer_after_inplace_collective, test/distributed/test_c10d_functional_native.py::CompileTest::test_ranks_and_tag 2024-04-03T16:25:56.1694366Z 2024-04-03T16:25:56.1694989Z Running distributed/_tensor/test_dtensor_ops 1/1 ... [2024-04-03 16:25:56.166531] 2024-04-03T16:25:56.1697340Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:25:56.166938] 2024-04-03T16:26:00.6925508Z 2024-04-03T16:26:00.6927715Z distributed/_tensor/test_dtensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_ops_1.1_d778597415f28c45_.log 2024-04-03T16:26:00.6930115Z Running 0 items in this shard: 2024-04-03T16:26:00.6930522Z 2024-04-03T16:26:00.6931336Z Running distributed/test_distributed_spawn 2/7 ... [2024-04-03 16:26:00.692762] 2024-04-03T16:26:00.6941071Z /usr/bin/mpiexec 2024-04-03T16:26:00.6942231Z MPI not available -- MPI backend tests will be skipped 2024-04-03T16:26:00.6943292Z Running distributed tests for the test backend with env init_method 2024-04-03T16:26:00.6951500Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:26:00.694696] 2024-04-03T16:26:04.2577167Z 2024-04-03T16:26:04.2579015Z distributed/test_distributed_spawn 2/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.7_300eb5a82d2641f4_.log 2024-04-03T16:26:04.2580526Z Running 0 items in this shard: 2024-04-03T16:26:04.2580877Z 2024-04-03T16:26:04.2588655Z Running distributed tests for the test backend with file init_method 2024-04-03T16:26:04.2596172Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:26:04.259181] 2024-04-03T16:26:07.7462175Z 2024-04-03T16:26:07.7464396Z distributed/test_distributed_spawn 2/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.7_5d2ab21067d12ed5_.log 2024-04-03T16:26:07.7465925Z Running 0 items in this shard: 2024-04-03T16:26:07.7466273Z 2024-04-03T16:26:07.7473385Z Running distributed tests for the nccl backend with env init_method 2024-04-03T16:26:07.7480800Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:26:07.747624] 2024-04-03T16:30:02.3883779Z 2024-04-03T16:30:02.3885590Z distributed/test_distributed_spawn 2/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.7_4cd359dbf8fd866e_.log 2024-04-03T16:30:02.3919202Z Running 41 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-04-03T16:30:02.3947850Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2024-04-03T16:30:02.3950028Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2024-04-03T16:30:02.3952060Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2024-04-03T16:30:02.3954039Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2024-04-03T16:30:02.3955891Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2024-04-03T16:30:02.3957726Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2024-04-03T16:30:02.3959603Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2024-04-03T16:30:02.3961598Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2024-04-03T16:30:02.3963367Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max 2024-04-03T16:30:02.3965107Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2024-04-03T16:30:02.3966835Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2024-04-03T16:30:02.3968501Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2024-04-03T16:30:02.3970671Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2024-04-03T16:30:02.3972294Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum 2024-04-03T16:30:02.3973974Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2024-04-03T16:30:02.3975801Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2024-04-03T16:30:02.3977755Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2024-04-03T16:30:02.3979579Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2024-04-03T16:30:02.3981174Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier 2024-04-03T16:30:02.3982780Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2024-04-03T16:30:02.3984421Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2024-04-03T16:30:02.3986128Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2024-04-03T16:30:02.3988112Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2024-04-03T16:30:02.3989768Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2024-04-03T16:30:02.3991333Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2024-04-03T16:30:02.3992980Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2024-04-03T16:30:02.3994610Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv 2024-04-03T16:30:02.3996394Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size 2024-04-03T16:30:02.3998311Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2024-04-03T16:30:02.4000123Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2024-04-03T16:30:02.4001896Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2024-04-03T16:30:02.4003591Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2024-04-03T16:30:02.4005304Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2024-04-03T16:30:02.4007072Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2024-04-03T16:30:02.4008714Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2024-04-03T16:30:02.4010795Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2024-04-03T16:30:02.4012489Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2024-04-03T16:30:02.4014279Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2024-04-03T16:30:02.4016031Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2024-04-03T16:30:02.4017813Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2024-04-03T16:30:02.4019685Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-04-03T16:30:02.4020721Z 2024-04-03T16:30:02.4021114Z Running distributed tests for the nccl backend with file init_method 2024-04-03T16:30:02.4023598Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:30:02.391274] 2024-04-03T16:33:56.5626543Z 2024-04-03T16:33:56.5630407Z distributed/test_distributed_spawn 2/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.7_fb4e089d400a7867_.log 2024-04-03T16:33:56.5662463Z Running 41 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-04-03T16:33:56.5691233Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2024-04-03T16:33:56.5693577Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2024-04-03T16:33:56.5695580Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2024-04-03T16:33:56.5697537Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2024-04-03T16:33:56.5699430Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2024-04-03T16:33:56.5701316Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2024-04-03T16:33:56.5703173Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2024-04-03T16:33:56.5704989Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2024-04-03T16:33:56.5706734Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max 2024-04-03T16:33:56.5708466Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2024-04-03T16:33:56.5710165Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2024-04-03T16:33:56.5711955Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2024-04-03T16:33:56.5713583Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2024-04-03T16:33:56.5715180Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum 2024-04-03T16:33:56.5716827Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2024-04-03T16:33:56.5718662Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2024-04-03T16:33:56.5720605Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2024-04-03T16:33:56.5722415Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2024-04-03T16:33:56.5723991Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier 2024-04-03T16:33:56.5725608Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2024-04-03T16:33:56.5727232Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2024-04-03T16:33:56.5728934Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2024-04-03T16:33:56.5731198Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2024-04-03T16:33:56.5732873Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2024-04-03T16:33:56.5734449Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2024-04-03T16:33:56.5736070Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2024-04-03T16:33:56.5737853Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv 2024-04-03T16:33:56.5739639Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size 2024-04-03T16:33:56.5741538Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2024-04-03T16:33:56.5743321Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2024-04-03T16:33:56.5745108Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2024-04-03T16:33:56.5746807Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2024-04-03T16:33:56.5748466Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2024-04-03T16:33:56.5750101Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2024-04-03T16:33:56.5751744Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2024-04-03T16:33:56.5753382Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2024-04-03T16:33:56.5755069Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2024-04-03T16:33:56.5756934Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2024-04-03T16:33:56.5758706Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2024-04-03T16:33:56.5760477Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2024-04-03T16:33:56.5762344Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-04-03T16:33:56.5763360Z 2024-04-03T16:33:56.5763725Z Running distributed tests for the gloo backend with env init_method 2024-04-03T16:33:56.5766193Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:33:56.565783] 2024-04-03T16:38:49.5153637Z 2024-04-03T16:38:49.5155500Z distributed/test_distributed_spawn 2/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.7_d05238268882bd15_.log 2024-04-03T16:38:49.5187114Z Running 41 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-04-03T16:38:49.5215687Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2024-04-03T16:38:49.5217888Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2024-04-03T16:38:49.5219911Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2024-04-03T16:38:49.5221870Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2024-04-03T16:38:49.5223727Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2024-04-03T16:38:49.5225740Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2024-04-03T16:38:49.5227610Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2024-04-03T16:38:49.5229398Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2024-04-03T16:38:49.5231148Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max 2024-04-03T16:38:49.5232879Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2024-04-03T16:38:49.5234592Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2024-04-03T16:38:49.5236259Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2024-04-03T16:38:49.5237864Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2024-04-03T16:38:49.5239466Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum 2024-04-03T16:38:49.5241118Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2024-04-03T16:38:49.5243072Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2024-04-03T16:38:49.5245036Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2024-04-03T16:38:49.5246873Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2024-04-03T16:38:49.5248448Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier 2024-04-03T16:38:49.5250561Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2024-04-03T16:38:49.5252192Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2024-04-03T16:38:49.5253906Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2024-04-03T16:38:49.5255701Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2024-04-03T16:38:49.5257378Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2024-04-03T16:38:49.5258951Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2024-04-03T16:38:49.5260598Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2024-04-03T16:38:49.5262200Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv 2024-04-03T16:38:49.5263996Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size 2024-04-03T16:38:49.5265908Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2024-04-03T16:38:49.5267714Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2024-04-03T16:38:49.5269638Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2024-04-03T16:38:49.5271333Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2024-04-03T16:38:49.5272992Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2024-04-03T16:38:49.5274643Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2024-04-03T16:38:49.5276268Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2024-04-03T16:38:49.5277911Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2024-04-03T16:38:49.5279598Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2024-04-03T16:38:49.5281367Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2024-04-03T16:38:49.5283130Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2024-04-03T16:38:49.5284900Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2024-04-03T16:38:49.5286764Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-04-03T16:38:49.5287785Z 2024-04-03T16:38:49.5288281Z Running distributed tests for the gloo backend with file init_method 2024-04-03T16:38:49.5291333Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:38:49.518566] 2024-04-03T16:43:42.7495404Z 2024-04-03T16:43:42.7497228Z distributed/test_distributed_spawn 2/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.7_96c97bb3235c70e2_.log 2024-04-03T16:43:42.7531030Z Running 41 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-04-03T16:43:42.7559130Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2024-04-03T16:43:42.7561299Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2024-04-03T16:43:42.7563326Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2024-04-03T16:43:42.7565274Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2024-04-03T16:43:42.7567132Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2024-04-03T16:43:42.7568964Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2024-04-03T16:43:42.7571326Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2024-04-03T16:43:42.7573123Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2024-04-03T16:43:42.7574876Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max 2024-04-03T16:43:42.7576778Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2024-04-03T16:43:42.7578489Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2024-04-03T16:43:42.7580144Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2024-04-03T16:43:42.7581746Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2024-04-03T16:43:42.7583342Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum 2024-04-03T16:43:42.7585011Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2024-04-03T16:43:42.7586877Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2024-04-03T16:43:42.7588822Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2024-04-03T16:43:42.7590640Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2024-04-03T16:43:42.7592226Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier 2024-04-03T16:43:42.7593829Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2024-04-03T16:43:42.7595576Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2024-04-03T16:43:42.7597299Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2024-04-03T16:43:42.7599095Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2024-04-03T16:43:42.7600761Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2024-04-03T16:43:42.7602322Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2024-04-03T16:43:42.7603961Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2024-04-03T16:43:42.7605564Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv 2024-04-03T16:43:42.7607362Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size 2024-04-03T16:43:42.7609793Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2024-04-03T16:43:42.7611628Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2024-04-03T16:43:42.7613400Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2024-04-03T16:43:42.7615095Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2024-04-03T16:43:42.7616750Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2024-04-03T16:43:42.7618406Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2024-04-03T16:43:42.7620030Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2024-04-03T16:43:42.7621834Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2024-04-03T16:43:42.7623521Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2024-04-03T16:43:42.7625277Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2024-04-03T16:43:42.7627038Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2024-04-03T16:43:42.7628807Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2024-04-03T16:43:42.7630669Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-04-03T16:43:42.7631697Z 2024-04-03T16:43:42.7632058Z Running distributed tests for the ucc backend with env init_method 2024-04-03T16:43:42.7634542Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:43:42.752214] 2024-04-03T16:48:05.6512604Z 2024-04-03T16:48:05.6517419Z distributed/test_distributed_spawn 2/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.7_091797c278c884aa_.log 2024-04-03T16:48:05.6550724Z Running 41 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-04-03T16:48:05.6579190Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2024-04-03T16:48:05.6581416Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2024-04-03T16:48:05.6583422Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2024-04-03T16:48:05.6585423Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2024-04-03T16:48:05.6587282Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2024-04-03T16:48:05.6589124Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2024-04-03T16:48:05.6590998Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2024-04-03T16:48:05.6592801Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2024-04-03T16:48:05.6594614Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max 2024-04-03T16:48:05.6596339Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2024-04-03T16:48:05.6598065Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2024-04-03T16:48:05.6599728Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2024-04-03T16:48:05.6601335Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2024-04-03T16:48:05.6603114Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum 2024-04-03T16:48:05.6604763Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2024-04-03T16:48:05.6606597Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2024-04-03T16:48:05.6608541Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2024-04-03T16:48:05.6610830Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2024-04-03T16:48:05.6612415Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier 2024-04-03T16:48:05.6614031Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2024-04-03T16:48:05.6615664Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2024-04-03T16:48:05.6617367Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2024-04-03T16:48:05.6619156Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2024-04-03T16:48:05.6620830Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2024-04-03T16:48:05.6622525Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2024-04-03T16:48:05.6624188Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2024-04-03T16:48:05.6625790Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv 2024-04-03T16:48:05.6627579Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size 2024-04-03T16:48:05.6629480Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2024-04-03T16:48:05.6631281Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2024-04-03T16:48:05.6633054Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2024-04-03T16:48:05.6634750Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2024-04-03T16:48:05.6636417Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2024-04-03T16:48:05.6638052Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2024-04-03T16:48:05.6639696Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2024-04-03T16:48:05.6641335Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2024-04-03T16:48:05.6643023Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2024-04-03T16:48:05.6644793Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2024-04-03T16:48:05.6646552Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2024-04-03T16:48:05.6648473Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2024-04-03T16:48:05.6650815Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-04-03T16:48:05.6651837Z 2024-04-03T16:48:05.6652197Z Running distributed tests for the ucc backend with file init_method 2024-04-03T16:48:05.6654670Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:48:05.654472] 2024-04-03T16:52:28.3259765Z 2024-04-03T16:52:28.3261595Z distributed/test_distributed_spawn 2/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.7_fc5e2274bae7b13e_.log 2024-04-03T16:52:28.3294255Z Running 41 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-04-03T16:52:28.3323129Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2024-04-03T16:52:28.3325294Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2024-04-03T16:52:28.3327316Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2024-04-03T16:52:28.3329832Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2024-04-03T16:52:28.3331743Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2024-04-03T16:52:28.3333584Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2024-04-03T16:52:28.3335457Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2024-04-03T16:52:28.3337239Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2024-04-03T16:52:28.3338984Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max 2024-04-03T16:52:28.3340717Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2024-04-03T16:52:28.3342435Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2024-04-03T16:52:28.3344098Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2024-04-03T16:52:28.3345703Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2024-04-03T16:52:28.3347297Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum 2024-04-03T16:52:28.3348969Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2024-04-03T16:52:28.3350790Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2024-04-03T16:52:28.3352740Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2024-04-03T16:52:28.3354700Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2024-04-03T16:52:28.3356289Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier 2024-04-03T16:52:28.3357893Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2024-04-03T16:52:28.3359519Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2024-04-03T16:52:28.3361227Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2024-04-03T16:52:28.3363021Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2024-04-03T16:52:28.3364697Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2024-04-03T16:52:28.3366271Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2024-04-03T16:52:28.3367909Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2024-04-03T16:52:28.3369986Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv 2024-04-03T16:52:28.3371801Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size 2024-04-03T16:52:28.3373699Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2024-04-03T16:52:28.3375674Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2024-04-03T16:52:28.3377476Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2024-04-03T16:52:28.3379168Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2024-04-03T16:52:28.3380826Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2024-04-03T16:52:28.3382476Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2024-04-03T16:52:28.3384101Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2024-04-03T16:52:28.3385745Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2024-04-03T16:52:28.3387437Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2024-04-03T16:52:28.3389205Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2024-04-03T16:52:28.3390969Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2024-04-03T16:52:28.3392740Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2024-04-03T16:52:28.3394607Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-04-03T16:52:28.3395630Z 2024-04-03T16:52:28.3396200Z Running distributed/test_distributed_spawn 5/7 ... [2024-04-03 16:52:28.329076] 2024-04-03T16:52:28.3396959Z /usr/bin/mpiexec 2024-04-03T16:52:28.3397534Z MPI not available -- MPI backend tests will be skipped 2024-04-03T16:52:28.3398362Z Running distributed tests for the test backend with env init_method 2024-04-03T16:52:28.3400929Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:52:28.330921] 2024-04-03T16:52:31.8787250Z 2024-04-03T16:52:31.8788982Z distributed/test_distributed_spawn 5/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.7_a7aae86745f3466c_.log 2024-04-03T16:52:31.8790486Z Running 0 items in this shard: 2024-04-03T16:52:31.8790816Z 2024-04-03T16:52:31.8797679Z Running distributed tests for the test backend with file init_method 2024-04-03T16:52:31.8804901Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:52:31.880067] 2024-04-03T16:52:35.3402088Z 2024-04-03T16:52:35.3404139Z distributed/test_distributed_spawn 5/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.7_715d09f971c889c2_.log 2024-04-03T16:52:35.3405658Z Running 0 items in this shard: 2024-04-03T16:52:35.3405990Z 2024-04-03T16:52:35.3411929Z Running distributed tests for the nccl backend with env init_method 2024-04-03T16:52:35.3420205Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:52:35.341512] 2024-04-03T16:57:34.6214652Z 2024-04-03T16:57:34.6218430Z distributed/test_distributed_spawn 5/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.7_62e08a1a18ddadaf_.log 2024-04-03T16:57:34.6257411Z Running 48 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-04-03T16:57:34.6290848Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value 2024-04-03T16:57:34.6293128Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2024-04-03T16:57:34.6295143Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2024-04-03T16:57:34.6296834Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2024-04-03T16:57:34.6298437Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2024-04-03T16:57:34.6300807Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg 2024-04-03T16:57:34.6304351Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2024-04-03T16:57:34.6307039Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2024-04-03T16:57:34.6308884Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2024-04-03T16:57:34.6310670Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2024-04-03T16:57:34.6312477Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2024-04-03T16:57:34.6314264Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2024-04-03T16:57:34.6315963Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2024-04-03T16:57:34.6317598Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2024-04-03T16:57:34.6319363Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2024-04-03T16:57:34.6321246Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2024-04-03T16:57:34.6323272Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2024-04-03T16:57:34.6325015Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2024-04-03T16:57:34.6326840Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err 2024-04-03T16:57:34.6328652Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2024-04-03T16:57:34.6331085Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2024-04-03T16:57:34.6332907Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2024-04-03T16:57:34.6334726Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2024-04-03T16:57:34.6336533Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg 2024-04-03T16:57:34.6338220Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2024-04-03T16:57:34.6339979Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2024-04-03T16:57:34.6341760Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged 2024-04-03T16:57:34.6343532Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2024-04-03T16:57:34.6345340Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params 2024-04-03T16:57:34.6347163Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2024-04-03T16:57:34.6349090Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2024-04-03T16:57:34.6350786Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group 2024-04-03T16:57:34.6352387Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2024-04-03T16:57:34.6354040Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo 2024-04-03T16:57:34.6355880Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2024-04-03T16:57:34.6357793Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2024-04-03T16:57:34.6359811Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2024-04-03T16:57:34.6361780Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2024-04-03T16:57:34.6363399Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2024-04-03T16:57:34.6364976Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2024-04-03T16:57:34.6366573Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2024-04-03T16:57:34.6368383Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2024-04-03T16:57:34.6370402Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-04-03T16:57:34.6372114Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2024-04-03T16:57:34.6373953Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2024-04-03T16:57:34.6375735Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-04-03T16:57:34.6377467Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2024-04-03T16:57:34.6379259Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-04-03T16:57:34.6380333Z 2024-04-03T16:57:34.6380699Z Running distributed tests for the nccl backend with file init_method 2024-04-03T16:57:34.6383162Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:57:34.624719] 2024-04-03T17:02:34.8689644Z 2024-04-03T17:02:34.8693823Z distributed/test_distributed_spawn 5/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.7_915ce8a80924b639_.log 2024-04-03T17:02:34.8730305Z Running 48 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-04-03T17:02:34.8763407Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value 2024-04-03T17:02:34.8765675Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2024-04-03T17:02:34.8767682Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2024-04-03T17:02:34.8769750Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2024-04-03T17:02:34.8771371Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2024-04-03T17:02:34.8773093Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg 2024-04-03T17:02:34.8774945Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2024-04-03T17:02:34.8776801Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2024-04-03T17:02:34.8778795Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2024-04-03T17:02:34.8780598Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2024-04-03T17:02:34.8782393Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2024-04-03T17:02:34.8784168Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2024-04-03T17:02:34.8785864Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2024-04-03T17:02:34.8787493Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2024-04-03T17:02:34.8789259Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2024-04-03T17:02:34.8791145Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2024-04-03T17:02:34.8792958Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2024-04-03T17:02:34.8794684Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2024-04-03T17:02:34.8796502Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err 2024-04-03T17:02:34.8798314Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2024-04-03T17:02:34.8800180Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2024-04-03T17:02:34.8801988Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2024-04-03T17:02:34.8803936Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2024-04-03T17:02:34.8805745Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg 2024-04-03T17:02:34.8807417Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2024-04-03T17:02:34.8809481Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2024-04-03T17:02:34.8811325Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged 2024-04-03T17:02:34.8813100Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2024-04-03T17:02:34.8814919Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params 2024-04-03T17:02:34.8816730Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2024-04-03T17:02:34.8818500Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2024-04-03T17:02:34.8820207Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group 2024-04-03T17:02:34.8821811Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2024-04-03T17:02:34.8823588Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo 2024-04-03T17:02:34.8825451Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2024-04-03T17:02:34.8827366Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2024-04-03T17:02:34.8829381Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2024-04-03T17:02:34.8831341Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2024-04-03T17:02:34.8832966Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2024-04-03T17:02:34.8834547Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2024-04-03T17:02:34.8836138Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2024-04-03T17:02:34.8837783Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2024-04-03T17:02:34.8839383Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-04-03T17:02:34.8841102Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2024-04-03T17:02:34.8842919Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2024-04-03T17:02:34.8844698Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-04-03T17:02:34.8846432Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2024-04-03T17:02:34.8848376Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-04-03T17:02:34.8849800Z 2024-04-03T17:02:34.8850176Z Running distributed tests for the gloo backend with env init_method 2024-04-03T17:02:34.8852644Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:02:34.871628] 2024-04-03T17:08:17.2933388Z 2024-04-03T17:08:17.2937384Z distributed/test_distributed_spawn 5/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.7_856cb86d1ee1daf9_.log 2024-04-03T17:08:17.2974728Z Running 48 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-04-03T17:08:17.3007792Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value 2024-04-03T17:08:17.3010580Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2024-04-03T17:08:17.3012587Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2024-04-03T17:08:17.3014291Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2024-04-03T17:08:17.3015886Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2024-04-03T17:08:17.3017604Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg 2024-04-03T17:08:17.3019451Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2024-04-03T17:08:17.3021302Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2024-04-03T17:08:17.3023132Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2024-04-03T17:08:17.3024928Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2024-04-03T17:08:17.3026729Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2024-04-03T17:08:17.3028499Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2024-04-03T17:08:17.3030187Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2024-04-03T17:08:17.3032015Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2024-04-03T17:08:17.3033783Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2024-04-03T17:08:17.3035646Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2024-04-03T17:08:17.3037435Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2024-04-03T17:08:17.3039187Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2024-04-03T17:08:17.3040989Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err 2024-04-03T17:08:17.3042813Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2024-04-03T17:08:17.3044670Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2024-04-03T17:08:17.3046491Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2024-04-03T17:08:17.3048282Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2024-04-03T17:08:17.3050525Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg 2024-04-03T17:08:17.3052392Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2024-04-03T17:08:17.3054176Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2024-04-03T17:08:17.3055944Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged 2024-04-03T17:08:17.3057723Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2024-04-03T17:08:17.3059551Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params 2024-04-03T17:08:17.3061360Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2024-04-03T17:08:17.3063127Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2024-04-03T17:08:17.3064823Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group 2024-04-03T17:08:17.3066447Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2024-04-03T17:08:17.3068097Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo 2024-04-03T17:08:17.3069917Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2024-04-03T17:08:17.3071824Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2024-04-03T17:08:17.3073871Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2024-04-03T17:08:17.3075836Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2024-04-03T17:08:17.3077563Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2024-04-03T17:08:17.3079140Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2024-04-03T17:08:17.3080756Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2024-04-03T17:08:17.3082403Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2024-04-03T17:08:17.3083985Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-04-03T17:08:17.3085710Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2024-04-03T17:08:17.3087549Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2024-04-03T17:08:17.3089710Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-04-03T17:08:17.3091469Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2024-04-03T17:08:17.3093266Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-04-03T17:08:17.3094314Z 2024-04-03T17:08:17.3094701Z Running distributed tests for the gloo backend with file init_method 2024-04-03T17:08:17.3097291Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:08:17.296608] 2024-04-03T17:14:04.5898967Z 2024-04-03T17:14:04.5900942Z distributed/test_distributed_spawn 5/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.7_b32bbab13b3eec33_.log 2024-04-03T17:14:04.5940765Z Running 48 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-04-03T17:14:04.5974209Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value 2024-04-03T17:14:04.5976470Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2024-04-03T17:14:04.5978464Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2024-04-03T17:14:04.5980163Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2024-04-03T17:14:04.5981755Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2024-04-03T17:14:04.5983624Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg 2024-04-03T17:14:04.5985473Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2024-04-03T17:14:04.5987336Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2024-04-03T17:14:04.5989167Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2024-04-03T17:14:04.5990953Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2024-04-03T17:14:04.5992760Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2024-04-03T17:14:04.5994547Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2024-04-03T17:14:04.5996232Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2024-04-03T17:14:04.5997876Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2024-04-03T17:14:04.5999613Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2024-04-03T17:14:04.6001609Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2024-04-03T17:14:04.6003416Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2024-04-03T17:14:04.6005156Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2024-04-03T17:14:04.6006953Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err 2024-04-03T17:14:04.6008766Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2024-04-03T17:14:04.6011121Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2024-04-03T17:14:04.6012951Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2024-04-03T17:14:04.6014727Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2024-04-03T17:14:04.6016542Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg 2024-04-03T17:14:04.6018236Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2024-04-03T17:14:04.6019991Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2024-04-03T17:14:04.6021747Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged 2024-04-03T17:14:04.6023512Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2024-04-03T17:14:04.6025342Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params 2024-04-03T17:14:04.6027316Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2024-04-03T17:14:04.6029067Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2024-04-03T17:14:04.6030762Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group 2024-04-03T17:14:04.6032379Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2024-04-03T17:14:04.6034032Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo 2024-04-03T17:14:04.6035855Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2024-04-03T17:14:04.6037757Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2024-04-03T17:14:04.6039796Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2024-04-03T17:14:04.6041750Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2024-04-03T17:14:04.6043349Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2024-04-03T17:14:04.6044970Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2024-04-03T17:14:04.6046756Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2024-04-03T17:14:04.6048403Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2024-04-03T17:14:04.6050404Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-04-03T17:14:04.6052118Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2024-04-03T17:14:04.6053960Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2024-04-03T17:14:04.6055713Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-04-03T17:14:04.6057441Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2024-04-03T17:14:04.6059255Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-04-03T17:14:04.6060309Z 2024-04-03T17:14:04.6060684Z Running distributed tests for the ucc backend with env init_method 2024-04-03T17:14:04.6063119Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:14:04.592962] 2024-04-03T17:19:07.9330865Z 2024-04-03T17:19:07.9332714Z distributed/test_distributed_spawn 5/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.7_d4c0c027f37bbc54_.log 2024-04-03T17:19:07.9370423Z Running 48 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-04-03T17:19:07.9403447Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value 2024-04-03T17:19:07.9405743Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2024-04-03T17:19:07.9407739Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2024-04-03T17:19:07.9409832Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2024-04-03T17:19:07.9411425Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2024-04-03T17:19:07.9413125Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg 2024-04-03T17:19:07.9414966Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2024-04-03T17:19:07.9416943Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2024-04-03T17:19:07.9418805Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2024-04-03T17:19:07.9420588Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2024-04-03T17:19:07.9422387Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2024-04-03T17:19:07.9424137Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2024-04-03T17:19:07.9425814Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2024-04-03T17:19:07.9427464Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2024-04-03T17:19:07.9429199Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2024-04-03T17:19:07.9431073Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2024-04-03T17:19:07.9432850Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2024-04-03T17:19:07.9434579Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2024-04-03T17:19:07.9436365Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err 2024-04-03T17:19:07.9438227Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2024-04-03T17:19:07.9440080Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2024-04-03T17:19:07.9442044Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2024-04-03T17:19:07.9443821Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2024-04-03T17:19:07.9445609Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg 2024-04-03T17:19:07.9447290Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2024-04-03T17:19:07.9449052Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2024-04-03T17:19:07.9451109Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged 2024-04-03T17:19:07.9452878Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2024-04-03T17:19:07.9454703Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params 2024-04-03T17:19:07.9456517Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2024-04-03T17:19:07.9458267Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2024-04-03T17:19:07.9459967Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group 2024-04-03T17:19:07.9461703Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2024-04-03T17:19:07.9463380Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo 2024-04-03T17:19:07.9465186Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2024-04-03T17:19:07.9467085Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2024-04-03T17:19:07.9469120Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2024-04-03T17:19:07.9471077Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2024-04-03T17:19:07.9472677Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2024-04-03T17:19:07.9474249Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2024-04-03T17:19:07.9475862Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2024-04-03T17:19:07.9477483Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2024-04-03T17:19:07.9479078Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-04-03T17:19:07.9480797Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2024-04-03T17:19:07.9482641Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2024-04-03T17:19:07.9484387Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-04-03T17:19:07.9486252Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2024-04-03T17:19:07.9488053Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-04-03T17:19:07.9489388Z 2024-04-03T17:19:07.9489735Z Running distributed tests for the ucc backend with file init_method 2024-04-03T17:19:07.9492163Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:19:07.936145] 2024-04-03T17:24:11.1181795Z 2024-04-03T17:24:11.1183637Z distributed/test_distributed_spawn 5/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.7_8ff465ec967bfa97_.log 2024-04-03T17:24:11.1217398Z Running 48 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-04-03T17:24:11.1283169Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value 2024-04-03T17:24:11.1287893Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2024-04-03T17:24:11.1292347Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2024-04-03T17:24:11.1295781Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2024-04-03T17:24:11.1299073Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2024-04-03T17:24:11.1302577Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg 2024-04-03T17:24:11.1306473Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2024-04-03T17:24:11.1310361Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2024-04-03T17:24:11.1314211Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2024-04-03T17:24:11.1317939Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2024-04-03T17:24:11.1321834Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2024-04-03T17:24:11.1325526Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2024-04-03T17:24:11.1329790Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2024-04-03T17:24:11.1333254Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2024-04-03T17:24:11.1336890Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2024-04-03T17:24:11.1340823Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2024-04-03T17:24:11.1344526Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2024-04-03T17:24:11.1348043Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2024-04-03T17:24:11.1351789Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err 2024-04-03T17:24:11.1355554Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2024-04-03T17:24:11.1359416Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2024-04-03T17:24:11.1363270Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2024-04-03T17:24:11.1367087Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2024-04-03T17:24:11.1371457Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg 2024-04-03T17:24:11.1374982Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2024-04-03T17:24:11.1378688Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2024-04-03T17:24:11.1382390Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged 2024-04-03T17:24:11.1386061Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2024-04-03T17:24:11.1389860Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params 2024-04-03T17:24:11.1393510Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2024-04-03T17:24:11.1397195Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2024-04-03T17:24:11.1400726Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group 2024-04-03T17:24:11.1404091Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2024-04-03T17:24:11.1407484Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo 2024-04-03T17:24:11.1411659Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2024-04-03T17:24:11.1415624Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2024-04-03T17:24:11.1419861Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2024-04-03T17:24:11.1424182Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2024-04-03T17:24:11.1427540Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2024-04-03T17:24:11.1430930Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2024-04-03T17:24:11.1434284Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2024-04-03T17:24:11.1437668Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2024-04-03T17:24:11.1441005Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-04-03T17:24:11.1444705Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2024-04-03T17:24:11.1448704Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2024-04-03T17:24:11.1452710Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-04-03T17:24:11.1456343Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2024-04-03T17:24:11.1460127Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-04-03T17:24:11.1462280Z 2024-04-03T17:24:11.1463779Z Running distributed/checkpoint/test_tp_checkpoint 1/1 ... [2024-04-03 17:24:11.121306] 2024-04-03T17:24:11.1468866Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_tp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:24:11.121708] 2024-04-03T17:24:23.8137082Z 2024-04-03T17:24:23.8139042Z distributed/checkpoint/test_tp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_tp_checkpoint_1.1_87ebbf6392b481be_.log 2024-04-03T17:24:23.8142281Z Running 2 items in this shard: test/distributed/checkpoint/test_tp_checkpoint.py::TestTpCheckpoint::test_tp_checkpoint, test/distributed/checkpoint/test_tp_checkpoint.py::TestTpCheckpoint::test_tp_checkpoint_load_on_meta_device 2024-04-03T17:24:23.8144085Z 2024-04-03T17:24:23.8144801Z Running distributed/_tensor/test_tensor_ops 1/1 ... [2024-04-03 17:24:23.813931] 2024-04-03T17:24:23.8147578Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_tensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:24:23.814342] 2024-04-03T17:26:10.3030533Z 2024-04-03T17:26:10.3032340Z distributed/_tensor/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_tensor_ops_1.1_a64503e4a4f6c248_.log 2024-04-03T17:26:10.3044883Z Running 22 items in this shard: test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_aten_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_clone, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_detach, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_dtensor_dtype_conversion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_empty_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_equal, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_full_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_gather, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_index, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_inplace_op, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_full, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_op_out_variant, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_stack, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_where_type_promotion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zero_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like_partial_sum 2024-04-03T17:26:10.3057381Z 2024-04-03T17:26:10.3058051Z Running distributed/_composable/fsdp/test_fully_shard_init 1/1 ... [2024-04-03 17:26:10.303316] 2024-04-03T17:26:10.3060654Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:26:10.303723] 2024-04-03T17:26:16.9821658Z 2024-04-03T17:26:16.9824079Z distributed/_composable/fsdp/test_fully_shard_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_init_1.1_0a3062f88c4293eb_.log 2024-04-03T17:26:16.9841714Z Running 20 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardDeviceTensor::test_move_states_to_device_tensor, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardDeviceDTensor::test_move_states_to_device_dtensor_invalid, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardDeviceDTensor::test_move_states_to_device_dtensor_valid, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMeshArg::test_invalid_mesh_ndim, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_duplicate, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_nested, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_nested_fully_shard_and_replicate, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_single, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_states_nested_fully_shard, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_states_shared_params_and_buffers, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardParamModuleInfos::test_get_param_module_infos_duplicates, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardParamModuleInfos::test_get_param_module_infos_shared_params, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardedParameterTensor::test_shard_tensor_parameters, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardedParameterDTensor::test_shard_dtensor_parameters, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_fully_shard_double_lazy_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_fully_shard_is_root, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_fully_shard_module_and_param_fqns, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMetaDeviceInit::test_invalid_meta_device_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMetaDeviceInit::test_meta_device_1d_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMetaDeviceInit::test_meta_device_2d_init 2024-04-03T17:26:16.9858545Z 2024-04-03T17:26:16.9859145Z Running distributed/_tensor/test_xla_integration 1/1 ... [2024-04-03 17:26:16.982441] 2024-04-03T17:26:16.9861600Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_xla_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:26:16.982887] 2024-04-03T17:26:20.5543530Z 2024-04-03T17:26:20.5545440Z distributed/_tensor/test_xla_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_xla_integration_1.1_19b4f89c81ed5ddb_.log 2024-04-03T17:26:20.5549122Z Running 3 items in this shard: test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_replicate, test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_shard, test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_2d 2024-04-03T17:26:20.5551683Z 2024-04-03T17:26:20.5552262Z Running distributed/checkpoint/test_format_utils 1/1 ... [2024-04-03 17:26:20.554613] 2024-04-03T17:26:20.5555054Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_format_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:26:20.555025] 2024-04-03T17:26:35.1997831Z 2024-04-03T17:26:35.1999762Z distributed/checkpoint/test_format_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_format_utils_1.1_1488c45e86ed8da0_.log 2024-04-03T17:26:35.2003129Z Running 3 items in this shard: test/distributed/checkpoint/test_format_utils.py::TestFormatUtils::test_dcp_to_torch_save, test/distributed/checkpoint/test_format_utils.py::TestFormatUtils::test_online_torch_save_to_dcp, test/distributed/checkpoint/test_format_utils.py::TestFormatUtils::test_torch_save_to_dcp 2024-04-03T17:26:35.2005250Z 2024-04-03T17:26:35.2005780Z Running distributed/test_dynamo_distributed 1/1 ... [2024-04-03 17:26:35.200078] 2024-04-03T17:26:35.2009452Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:26:35.200525] 2024-04-03T17:27:21.6574697Z 2024-04-03T17:27:21.6578646Z distributed/test_dynamo_distributed 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_dynamo_distributed_1.1_be94fbcf6ae657f3_.log 2024-04-03T17:27:21.6601451Z Running 37 items in this shard: test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_call_method_forward, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_issue90375, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_symbol_splitting, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_baseline_aot_eager_multiprocess, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_aot_autograd, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_async_subclass_no_specialize, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_custom_layer, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_aot_eager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_empty_graph_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_diff_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_same_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_orig_params_assert, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_guards, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_register_attr_or_module, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_staticmethod, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_ctx_manager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_inference, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_training, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_transpose, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_higher_order_op, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ignored_parameters, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_no_split 2024-04-03T17:27:21.6624258Z 2024-04-03T17:27:21.6624738Z Running distributed/test_device_mesh 1/1 ... [2024-04-03 17:27:21.657702] 2024-04-03T17:27:21.6627108Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_device_mesh.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:27:21.658119] 2024-04-03T17:30:23.5061503Z 2024-04-03T17:30:23.5065544Z distributed/test_device_mesh 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_device_mesh_1.1_8d033f1f4ead02be_.log 2024-04-03T17:30:23.5103302Z Running 56 items in this shard: test/distributed/test_device_mesh.py::DeviceMeshTest::test_assert_invalid_mesh_tensor, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_2d_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_2d_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshTest::test_fake_pg_device_mesh_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshTest::test_fake_pg_device_mesh_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_group_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_group_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank_raises_exception_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank_raises_exception_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshTest::test_init_process_group_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshTest::test_init_process_group_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_nd, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_init_device_mesh, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_duplicate_mesh_dim_names, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_mesh_shape_mesh_dim_names_mismatch, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_cache_and_reuse_submesh_slice_result_use_native_funcol_False, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_cache_and_reuse_submesh_slice_result_use_native_funcol_True, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_1d_use_native_funcol_False, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_1d_use_native_funcol_True, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_use_native_funcol_False, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_use_native_funcol_True, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_invalid_mesh_dim_name_use_native_funcol_False, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_invalid_mesh_dim_name_use_native_funcol_True, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_no_mesh_dim_found_use_native_funcol_False, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_no_mesh_dim_found_use_native_funcol_True, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_mesh_dim_by_name_use_native_funcol_False, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_mesh_dim_by_name_use_native_funcol_True, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_parent_mesh_dim_exist_use_native_funcol_False, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_parent_mesh_dim_exist_use_native_funcol_True, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_parent_mesh_dim_not_exist_use_native_funcol_False, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_parent_mesh_dim_not_exist_use_native_funcol_True, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_parent_mesh_use_native_funcol_False, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_parent_mesh_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_gather_uneven_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_gather_uneven_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_to_all_1d_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_to_all_1d_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_to_all_nd_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_to_all_nd_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_1d_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_1d_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_nd_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_nd_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_contiguous_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_contiguous_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_uneven_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_uneven_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_1d_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_1d_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_nd_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_nd_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_uneven_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_uneven_use_native_funcol_True 2024-04-03T17:30:23.5139881Z 2024-04-03T17:30:23.5140566Z Running distributed/_composable/fsdp/test_fully_shard_overlap 1/1 ... [2024-04-03 17:30:23.506521] 2024-04-03T17:30:23.5143218Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_overlap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:30:23.507056] 2024-04-03T17:30:31.7389241Z 2024-04-03T17:30:31.7391870Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_overlap_1.1_fd6e6144e195c7b7_.log 2024-04-03T17:30:31.7394447Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_training_overlap 2024-04-03T17:30:31.7395513Z 2024-04-03T17:30:31.7396059Z Running distributed/_tools/test_memory_tracker 1/1 ... [2024-04-03 17:30:31.739158] 2024-04-03T17:30:31.7401084Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_memory_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:30:31.739605] 2024-04-03T17:30:35.8122025Z 2024-04-03T17:30:35.8124104Z distributed/_tools/test_memory_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_memory_tracker_1.1_3275b5adcd8c41c3_.log 2024-04-03T17:30:35.8126389Z Running 1 items in this shard: test/distributed/_tools/test_memory_tracker.py::TestMemoryTracker::test_local_model 2024-04-03T17:30:35.8127421Z 2024-04-03T17:30:35.8128535Z Running distributed/_tensor/test_pointwise_ops 1/1 ... [2024-04-03 17:30:35.812458] 2024-04-03T17:30:35.8134670Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_pointwise_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:30:35.812904] 2024-04-03T17:30:39.7855882Z 2024-04-03T17:30:39.7857910Z distributed/_tensor/test_pointwise_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_pointwise_ops_1.1_d548ab8fbd32d761_.log 2024-04-03T17:30:39.7863362Z Running 7 items in this shard: test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_activations, test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_dropout, test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_dropout_backward, test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_dropout_errors, test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_mul_out, test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_partial_add, test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_partial_mul 2024-04-03T17:30:39.7867955Z 2024-04-03T17:30:39.7868591Z Running distributed/tensor/parallel/test_tp_style 1/1 ... [2024-04-03 17:30:39.785880] 2024-04-03T17:30:39.7871111Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_style.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:30:39.786362] 2024-04-03T17:31:16.8787567Z 2024-04-03T17:31:16.8789490Z distributed/tensor/parallel/test_tp_style 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_style_1.1_2d21dafd9d43b722_.log 2024-04-03T17:31:16.8797108Z Running 8 items in this shard: test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input_multiple_inputs, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_output, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_sequence_parallel_style 2024-04-03T17:31:16.8803004Z 2024-04-03T17:31:16.8803659Z Running distributed/checkpoint/test_dtensor_checkpoint 1/1 ... [2024-04-03 17:31:16.879014] 2024-04-03T17:31:16.8806224Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dtensor_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:31:16.879460] 2024-04-03T17:31:24.9612459Z 2024-04-03T17:31:24.9614879Z distributed/checkpoint/test_dtensor_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dtensor_checkpoint_1.1_6a5b91b09f5dbe12_.log 2024-04-03T17:31:24.9617275Z Running 1 items in this shard: test/distributed/checkpoint/test_dtensor_checkpoint.py::DTensorPlanner::test_distributed_tensor_planner 2024-04-03T17:31:24.9618316Z 2024-04-03T17:31:24.9618844Z Running distributed/_tensor/test_op_strategy 1/1 ... [2024-04-03 17:31:24.961066] 2024-04-03T17:31:24.9621286Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_op_strategy.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:31:24.961494] 2024-04-03T17:31:28.6841865Z 2024-04-03T17:31:28.6843791Z distributed/_tensor/test_op_strategy 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_op_strategy_1.1_a1be2495db9726d7_.log 2024-04-03T17:31:28.6853434Z Running 15 items in this shard: test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_batch_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_bmm_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_free_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_mm_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_bmm_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_bmm_2d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_linearity_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_mm_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_mm_2d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_pointwise_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_bmm_strategies, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_mm_strategies, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_redistribute_cost_latency, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_redistribute_cost_mesh_1d, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_redistribute_cost_mesh_2d 2024-04-03T17:31:28.6862147Z 2024-04-03T17:31:28.6862813Z Running distributed/_composable/fsdp/test_fully_shard_memory 1/1 ... [2024-04-03 17:31:28.684392] 2024-04-03T17:31:28.6865444Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:31:28.684833] 2024-04-03T17:31:37.8188302Z 2024-04-03T17:31:37.8190588Z distributed/_composable/fsdp/test_fully_shard_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_memory_1.1_f1a19e0da63a0ab3_.log 2024-04-03T17:31:37.8193074Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_memory.py::TestFullyShardMemory::test_fully_shard_training_memory 2024-04-03T17:31:37.8194116Z 2024-04-03T17:31:37.8195546Z Running distributed/checkpoint/e2e/test_pipeline 1/1 ... [2024-04-03 17:31:37.819103] 2024-04-03T17:31:37.8200334Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_pipeline.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:31:37.819538] 2024-04-03T17:31:45.6004180Z 2024-04-03T17:31:45.6006229Z distributed/checkpoint/e2e/test_pipeline 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_pipeline_1.1_472b6598bd660329_.log 2024-04-03T17:31:45.6008392Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_pipeline.py::TestPipeline::test_pipeline 2024-04-03T17:31:45.6009522Z 2024-04-03T17:31:45.6010096Z Running distributed/_tensor/test_embedding_ops 1/1 ... [2024-04-03 17:31:45.600638] 2024-04-03T17:31:45.6016207Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_embedding_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:31:45.601131] 2024-04-03T17:32:02.0496764Z 2024-04-03T17:32:02.0499316Z distributed/_tensor/test_embedding_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_embedding_ops_1.1_40395176b4c17540_.log 2024-04-03T17:32:02.0502889Z Running 3 items in this shard: test/distributed/_tensor/test_embedding_ops.py::TestEmbeddingOp::test_sharded_embedding_colwise, test/distributed/_tensor/test_embedding_ops.py::TestEmbeddingOp::test_sharded_embedding_colwise_max_norm_errors, test/distributed/_tensor/test_embedding_ops.py::TestEmbeddingOp::test_sharded_embedding_rowwise 2024-04-03T17:32:02.0506073Z 2024-04-03T17:32:02.0506714Z Running distributed/checkpoint/test_hsdp_checkpoint 1/1 ... [2024-04-03 17:32:02.049973] 2024-04-03T17:32:02.0510422Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_hsdp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:32:02.050496] 2024-04-03T17:32:19.1526581Z 2024-04-03T17:32:19.1528549Z distributed/checkpoint/test_hsdp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_hsdp_checkpoint_1.1_3b8bd579070614c7_.log 2024-04-03T17:32:19.1533753Z Running 4 items in this shard: test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_checkpoint_is_even_sharded_model_False, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_checkpoint_is_even_sharded_model_True, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_fsdp_checkpoint_conversion_is_even_sharded_model_False, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_fsdp_checkpoint_conversion_is_even_sharded_model_True 2024-04-03T17:32:19.1537465Z 2024-04-03T17:32:19.1537986Z Running distributed/_spmd/test_tracing 1/1 ... [2024-04-03 17:32:19.152886] 2024-04-03T17:32:19.1540310Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_spmd/test_tracing.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:32:19.153329] 2024-04-03T17:32:22.2577212Z 2024-04-03T17:32:22.2579148Z distributed/_spmd/test_tracing 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._spmd.test_tracing_1.1_d5f30c847a02f129_.log 2024-04-03T17:32:22.2580427Z 2024-04-03T17:32:22.2583080Z Running distributed/checkpoint/e2e/test_fine_tuning 1/1 ... [2024-04-03 17:32:22.257979] 2024-04-03T17:32:22.2588668Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_fine_tuning.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:32:22.258398] 2024-04-03T17:32:29.2371473Z 2024-04-03T17:32:29.2373666Z distributed/checkpoint/e2e/test_fine_tuning 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_fine_tuning_1.1_e6e7dbe809df9c08_.log 2024-04-03T17:32:29.2375896Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_fine_tuning.py::TestFineTuning::test_fine_tuning 2024-04-03T17:32:29.2377150Z 2024-04-03T17:32:29.2378003Z Running distributed/_tensor/test_matrix_ops 1/1 ... [2024-04-03 17:32:29.237449] 2024-04-03T17:32:29.2383817Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_matrix_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:32:29.237930] 2024-04-03T17:33:17.4556468Z 2024-04-03T17:33:17.4559210Z distributed/_tensor/test_matrix_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_matrix_ops_1.1_76dfade85d5f6b25_.log 2024-04-03T17:33:17.4567770Z Running 10 items in this shard: test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm_auto_redistribute, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm_empty_operand, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_baddbmm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_bmm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_mm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_scaled_dot_product_attention, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_t, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_t_partial_use_native_funcol_False, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_t_partial_use_native_funcol_True 2024-04-03T17:33:17.4574647Z 2024-04-03T17:33:17.4575451Z Running distributed/_composable/fully_shard/test_fully_shard_mixed_precision 1/1 ... [2024-04-03 17:33:17.455989] 2024-04-03T17:33:17.4578268Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:33:17.456508] 2024-04-03T17:33:24.9367078Z 2024-04-03T17:33:24.9370302Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_mixed_precision_1.1_f6e2fd1841674e45_.log 2024-04-03T17:33:24.9373883Z Running 1 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_mixed_precision.py::TestMixedPrecision::test_float16_cast_forward 2024-04-03T17:33:24.9375262Z 2024-04-03T17:33:24.9376101Z Running distributed/_composable/fully_shard/test_fully_shard_compile 1/1 ... [2024-04-03 17:33:24.936979] 2024-04-03T17:33:24.9379215Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:33:24.937397] 2024-04-03T17:33:28.3087166Z 2024-04-03T17:33:28.3089570Z distributed/_composable/fully_shard/test_fully_shard_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_compile_1.1_b093cef8e7e7c1c3_.log 2024-04-03T17:33:28.3092530Z Running 1 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_compile.py::TestCompile::test_compile 2024-04-03T17:33:28.3093512Z 2024-04-03T17:33:28.3094126Z Running distributed/checkpoint/test_fsdp_model_state 1/1 ... [2024-04-03 17:33:28.308963] 2024-04-03T17:33:28.3098643Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_model_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:33:28.309381] 2024-04-03T17:33:39.7478234Z 2024-04-03T17:33:39.7480144Z distributed/checkpoint/test_fsdp_model_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_model_state_1.1_8e94ffe223e2e77a_.log 2024-04-03T17:33:39.7483782Z Running 2 items in this shard: test/distributed/checkpoint/test_fsdp_model_state.py::FsdpModelStateCheckpoint::test_fsdp_model_state_no_resharding, test/distributed/checkpoint/test_fsdp_model_state.py::FsdpModelStateCheckpoint::test_fsdp_model_state_with_resharding 2024-04-03T17:33:39.7485651Z 2024-04-03T17:33:39.7486515Z Running distributed/fsdp/test_shard_utils 1/1 ... [2024-04-03 17:33:39.748035] 2024-04-03T17:33:39.7488868Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_shard_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:33:39.748450] 2024-04-03T17:33:51.8378854Z 2024-04-03T17:33:51.8380761Z distributed/fsdp/test_shard_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_shard_utils_1.1_269e2c9fdad190d0_.log 2024-04-03T17:33:51.8383671Z Running 2 items in this shard: test/distributed/fsdp/test_shard_utils.py::TestShardUtilsDistributed::test_create_chunk_sharded_tensor, test/distributed/fsdp/test_shard_utils.py::TestShardUtilsDistributedDTensor::test_create_chunk_dtensor 2024-04-03T17:33:51.8385359Z 2024-04-03T17:33:51.8386119Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-04-03 17:33:51.838090] 2024-04-03T17:33:51.8390297Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:33:51.838501] 2024-04-03T17:36:58.8717283Z 2024-04-03T17:36:58.8719425Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_1fabdb19266e44bd_.log 2024-04-03T17:36:58.8756447Z Running 17 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::test_compiled_fsdp, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardForwardInputs::test_root_move_forward_input_to_device, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardRegisteredParams::test_param_registration_after_backward, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardRegisteredParams::test_param_registration_after_forward, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardCastAfterInit::test_to_float64_after_init, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_multi_forward_module, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_non_root_forward_backward, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group_compile, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group_eager, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_single_group, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCompose::test_train_parity_with_activation_checkpointing, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardSharedParams::test_train_parity_with_shared_params_ac, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardSharedParams::test_train_parity_with_shared_params_no_ac, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardGradientAccumulation::test_set_requires_gradient_sync, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard2DTraining::test_train_parity_2d_mlp, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard2DTraining::test_train_parity_2d_transformer_checkpoint_resume, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardHSDPTraining::test_train_parity_hsdp 2024-04-03T17:36:58.8925322Z 2024-04-03T17:36:58.8933212Z Running distributed/fsdp/test_fsdp_pure_fp16 1/1 ... [2024-04-03 17:36:58.892989] 2024-04-03T17:36:58.8938655Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_pure_fp16.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:36:58.893429] 2024-04-03T17:37:11.2841355Z 2024-04-03T17:37:11.2843361Z distributed/fsdp/test_fsdp_pure_fp16 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_pure_fp16_1.1_4aa4b9e70a8d30a4_.log 2024-04-03T17:37:11.2846255Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_pure_fp16.py::TestPureFP16::test_fp16_dtypes, test/distributed/fsdp/test_fsdp_pure_fp16.py::TestPureFP16::test_pure_fp16_training 2024-04-03T17:37:11.2847568Z 2024-04-03T17:37:11.2848178Z Running distributed/fsdp/test_fsdp_optim_state 1/1 ... [2024-04-03 17:37:11.284378] 2024-04-03T17:37:11.2853523Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_optim_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:37:11.284854] 2024-04-03T17:41:51.2377788Z 2024-04-03T17:41:51.2379874Z distributed/fsdp/test_fsdp_optim_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_optim_state_1.1_33ed4a25d596538d_.log 2024-04-03T17:41:51.2437856Z Running 60 items in this shard: test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_compatible_with_trec, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_flatten_sharded_optim_state_dict_nested, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_flatten_sharded_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_full_optim_state_dict_keys, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_full_optim_state_dict_nested_invalid, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_interface_arguments, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_no_grad, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_input_warning, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_without_param_groups, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type0_use_multiple_param_groups_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type0_use_multiple_param_groups_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type1_use_multiple_param_groups_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type1_use_multiple_param_groups_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_names, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_save_load_without_0th_param_state_state_dict_type0, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_save_load_without_0th_param_state_state_dict_type1, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_halve_world_size, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_halve_world_size, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type0_add_to_fsdp_module_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type0_add_to_fsdp_module_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type1_add_to_fsdp_module_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type1_add_to_fsdp_module_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_state_dict_with_none_tensor_state, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_use_orig_params, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_with_empty_optimizer_state, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_with_no_shard 2024-04-03T17:41:51.2494553Z 2024-04-03T17:41:51.2495150Z Running distributed/fsdp/test_checkpoint_wrapper 1/1 ... [2024-04-03 17:41:51.238766] 2024-04-03T17:41:51.2497852Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:41:51.239256] 2024-04-03T17:41:55.1617839Z 2024-04-03T17:41:55.1619976Z distributed/fsdp/test_checkpoint_wrapper 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_checkpoint_wrapper_1.1_e901cad03dbf70f8_.log 2024-04-03T17:41:55.1627491Z Running 8 items in this shard: test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_apply_activation_checkpointing, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_args_kwargs, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_cpu_offload, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_kwarg_support, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_parity, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_forward_missing_attributes, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_fqn, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_load_activation_checkpointed_module 2024-04-03T17:41:55.1633450Z 2024-04-03T17:41:55.1634010Z Running distributed/fsdp/test_fsdp_grad_acc 1/1 ... [2024-04-03 17:41:55.162020] 2024-04-03T17:41:55.1636421Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:41:55.162476] 2024-04-03T17:42:38.2666409Z 2024-04-03T17:42:38.2668408Z distributed/fsdp/test_fsdp_grad_acc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_grad_acc_1.1_c6645c6ae19e08c7_.log 2024-04-03T17:42:38.2673740Z Running 6 items in this shard: test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs0_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs0_use_orig_params_True, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs1_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs1_use_orig_params_True, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_cpu_offload_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_cpu_offload_use_orig_params_True 2024-04-03T17:42:38.2678269Z 2024-04-03T17:42:38.2678888Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 ... [2024-04-03 17:42:38.267078] 2024-04-03T17:42:38.2681452Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:42:38.267521] 2024-04-03T17:44:25.5968679Z 2024-04-03T17:44:25.5970910Z distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_sharded_grad_scaler_1.1_2fa04a26791b357d_.log 2024-04-03T17:44:25.5993950Z Running 20 items in this shard: test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_grad_scaling, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_inf_gradients_skip_optim_step, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_scaling_unscaling_sparse, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_sharded_grad_scaler_found_inf 2024-04-03T17:44:25.6014745Z 2024-04-03T17:44:25.6015355Z Running distributed/fsdp/test_fsdp_state_dict 2/2 ... [2024-04-03 17:44:25.597348] 2024-04-03T17:44:25.6017786Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '--shard-id=2', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:44:25.597832] 2024-04-03T17:51:43.0979795Z 2024-04-03T17:51:43.0981794Z distributed/fsdp/test_fsdp_state_dict 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_state_dict_2.2_dad4170aff65df97_.log 2024-04-03T17:51:43.1102975Z Running 101 items in this shard: test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_local_state_dict_with_empty_ranks, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_sharded_load_multi_backend_pg, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_shared_module_and_shared_parameter, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_rank0_offload_save_load_flow_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_world_size_one, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_wrong_state_dict_config 2024-04-03T17:51:43.1221345Z 2024-04-03T17:51:43.1222074Z Running distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 ... [2024-04-03 17:51:43.098836] 2024-04-03T17:51:43.1224702Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:51:43.099255] 2024-04-03T17:52:13.9796707Z 2024-04-03T17:52:13.9798899Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks_1.1_512b80199acde4d7_.log 2024-04-03T17:52:13.9806140Z Running 6 items in this shard: test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_allreduce_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_fp16compress_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_noop_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_quantize_per_channel_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_quantize_per_tensor_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_is_last_hook 2024-04-03T17:52:13.9812236Z 2024-04-03T17:52:13.9812737Z Running distributed/fsdp/test_wrap 1/1 ... [2024-04-03 17:52:13.979937] 2024-04-03T17:52:13.9815049Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:52:13.980537] 2024-04-03T17:54:03.9329043Z 2024-04-03T17:54:03.9331218Z distributed/fsdp/test_wrap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_wrap_1.1_989e6a50fe1d14e9_.log 2024-04-03T17:54:03.9367908Z Running 51 items in this shard: test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_bn_always_wrapped_individually, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_False_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_False_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_True_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_True_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_False_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_False_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_True_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_True_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_False_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_False_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_True_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_True_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_False_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_False_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_True_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_True_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_False_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_False_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_True_cuda_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_True_cuda_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_wrap_batchnorm_individually_use_or_policy_False, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_wrap_batchnorm_individually_use_or_policy_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap_with_ignored_modules_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap_with_ignored_modules_wrap_method1, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_api, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_exclude_wrap, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_exclude_wrap_include_children, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_force_leaf, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_force_leaf_custom, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_cuda_init_mode0_cpu_offload0_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_cuda_init_mode0_cpu_offload0_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_cuda_init_mode0_cpu_offload1_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_cuda_init_mode0_cpu_offload1_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_cuda_init_mode1_cpu_offload0_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_cuda_init_mode1_cpu_offload0_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_cuda_init_mode1_cpu_offload1_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_cuda_init_mode1_cpu_offload1_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_with_ignored_modules_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_with_ignored_modules_wrap_method1, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_custom_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_frozen_params, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_module_wrap_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_module_wrap_policy_callable, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_transformer_auto_wrap_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_disabled_outside_context, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_override_defaults, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_wrap_method1, test/distributed/fsdp/test_wrap.py::TestWrapUtils::test_validate_frozen_params 2024-04-03T17:54:03.9403006Z 2024-04-03T17:54:03.9403711Z Running distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 ... [2024-04-03 17:54:03.933365] 2024-04-03T17:54:03.9406390Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint_cpu.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:54:03.933800] 2024-04-03T17:54:42.1775331Z 2024-04-03T17:54:42.1777379Z distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_cpu_1.1_7254c26b5ab6c16b_.log 2024-04-03T17:54:42.1790451Z Running 12 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_save_load_bytes_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_save_load_bytes_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor_thread_count_2 2024-04-03T17:54:42.1802046Z 2024-04-03T17:54:42.1802613Z Running distributed/fsdp/test_fsdp_comm_hooks 1/1 ... [2024-04-03 17:54:42.177869] 2024-04-03T17:54:42.1805051Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm_hooks.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:54:42.178295] 2024-04-03T17:56:43.8497560Z 2024-04-03T17:56:43.8499598Z distributed/fsdp/test_fsdp_comm_hooks 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_comm_hooks_1.1_17b16091b18d084c_.log 2024-04-03T17:56:43.8524867Z Running 28 items in this shard: test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_False_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_True_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_behavior_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_behavior_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_behavior_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_False_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_True_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_hybrid_strategy, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_non_root_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_non_root_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_non_root_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_submodules_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_submodules_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_submodules_sharding_strategy2 2024-04-03T17:56:43.8548118Z 2024-04-03T17:56:43.8548669Z Running distributed/fsdp/test_fsdp_meta 1/1 ... [2024-04-03 17:56:43.850165] 2024-04-03T17:56:43.8551054Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:56:43.850624] 2024-04-03T17:57:22.1052649Z 2024-04-03T17:57:22.1054571Z distributed/fsdp/test_fsdp_meta 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_meta_1.1_a454565346ba189b_.log 2024-04-03T17:57:22.1067166Z Running 15 items in this shard: test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_bad_arg_meta, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_bad_arg_torchdistx, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_meta_device_with_mixed_precision, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_default_init_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_default_init_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_reset_params_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_reset_params_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_default_init_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_default_init_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_init_fn_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_init_fn_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_meta_device_default_init, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_meta_device_reset_params, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_torchdistX_default_init, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_torchdistX_init_fn 2024-04-03T17:57:22.1078549Z 2024-04-03T17:57:22.1079133Z Running distributed/fsdp/test_fsdp_use_orig_params 1/1 ... [2024-04-03 17:57:22.105602] 2024-04-03T17:57:22.1081623Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_use_orig_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:57:22.106025] 2024-04-03T17:59:05.2868519Z 2024-04-03T17:59:05.2870409Z distributed/fsdp/test_fsdp_use_orig_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_use_orig_params_1.1_7c24337a26eef6dc_.log 2024-04-03T17:59:05.2894400Z Running 25 items in this shard: test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_full_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_no_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_shard_grad_op, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_full_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_no_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_shard_grad_op, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_trainability, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_fsdp_compile, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_multiple_optimizers, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_multiple_forward_offload_params_False, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_multiple_forward_offload_params_True, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_summon_between_two_forwards_offload_params_False, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_summon_between_two_forwards_offload_params_True, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsParamAccess::test_access_params_after_forward, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_grad_writeback, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_no_reshard_and_mixed_precision, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_param_writeback, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_writeback_between_fwd_and_bwd_for_no_reshard_raises, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_writeback_shape_mismatch, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsFQNs::test_named_parameters_in_forward, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsNoSync::test_no_sync_correctness, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsNoSync::test_no_sync_mixed_precision, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsInit::test_non_uniform_requires_grad, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestMultiTensorApply::test_multi_tensor_apply_size0_tensors_cpu, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestMultiTensorApply::test_multi_tensor_apply_size0_tensors_cuda 2024-04-03T17:59:05.2915451Z 2024-04-03T17:59:05.2916113Z Running distributed/fsdp/test_fsdp_multiple_forward 1/1 ... [2024-04-03 17:59:05.287519] 2024-04-03T17:59:05.2918640Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:59:05.287987] 2024-04-03T17:59:13.1700479Z 2024-04-03T17:59:13.1702672Z distributed/fsdp/test_fsdp_multiple_forward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_multiple_forward_1.1_71962f4b9c85ba99_.log 2024-04-03T17:59:13.1704888Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_multiple_forward.py::TestMultiForward::test_multi_forward 2024-04-03T17:59:13.1705817Z 2024-04-03T17:59:13.1706561Z Running distributed/fsdp/test_fsdp_comm 1/1 ... [2024-04-03 17:59:13.170372] 2024-04-03T17:59:13.1713147Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:59:13.170860] 2024-04-03T17:59:51.7217748Z 2024-04-03T17:59:51.7219548Z distributed/fsdp/test_fsdp_comm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_comm_1.1_09165abdbe44cf1c_.log 2024-04-03T17:59:51.7228010Z Running 8 items in this shard: test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_False_use_no_sync_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_False_use_no_sync_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_False_use_no_sync_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_False_use_no_sync_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_True_use_no_sync_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_True_use_no_sync_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_True_use_no_sync_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_True_use_no_sync_True_sharding_strategy1 2024-04-03T17:59:51.7235538Z 2024-04-03T17:59:51.7236101Z Running distributed/fsdp/test_fsdp_misc 1/1 ... [2024-04-03 17:59:51.722005] 2024-04-03T17:59:51.7238483Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_misc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:59:51.722424] 2024-04-03T18:00:56.6220665Z 2024-04-03T18:00:56.6222967Z distributed/fsdp/test_fsdp_misc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_misc_1.1_4452d7a2a205935b_.log 2024-04-03T18:00:56.6247049Z Running 27 items in this shard: test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_cpu_init_with_sync_module_states, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_cpu_init_stays_on_cpu, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_cpu_training, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_device_id_use_index_False, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_device_id_use_index_True, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_not_all_outputs_used_in_loss, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_optim_overlap_no_use_orig_params_error, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_optimizer_overlap, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_zero2_eval_with_prefetch, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_cpu_gpu_module, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_device_id_auto_wrap, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_device_id_cpu_offload, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_device_id_no_move_ignored_params_and_bufs, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_ignored_module_meta, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_namedtuple, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_same_model_across_ranks, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_homogeneous_attributes, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_module_device_mismatches_device_id, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_multigpu_module, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_no_params, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_training_device_mismatch_errors, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_unsafe_setattr, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_world_size_1_sharding_strategy_warning 2024-04-03T18:00:56.6270142Z 2024-04-03T18:00:56.6270904Z Running distributed/pipeline/sync/test_microbatch 1/1 ... [2024-04-03 18:00:56.622386] 2024-04-03T18:00:56.6274025Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_microbatch.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:00:56.622804] 2024-04-03T18:00:58.9922708Z 2024-04-03T18:00:58.9924720Z distributed/pipeline/sync/test_microbatch 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_microbatch_1.1_8d1843845297fa95_.log 2024-04-03T18:00:58.9931435Z Running 10 items in this shard: test/distributed/pipeline/sync/test_microbatch.py::test_batch_atomic, test/distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic, test/distributed/pipeline/sync/test_microbatch.py::test_batch_call, test/distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index, test/distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice, test/distributed/pipeline/sync/test_microbatch.py::test_check, test/distributed/pipeline/sync/test_microbatch.py::test_gather_tensors, test/distributed/pipeline/sync/test_microbatch.py::test_gather_tuples, test/distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor, test/distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors 2024-04-03T18:00:58.9936613Z 2024-04-03T18:00:58.9937207Z Running distributed/pipeline/sync/test_checkpoint 1/1 ... [2024-04-03 18:00:58.992677] 2024-04-03T18:00:58.9940059Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:00:58.993203] 2024-04-03T18:01:02.0641327Z 2024-04-03T18:01:02.0643490Z distributed/pipeline/sync/test_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_checkpoint_1.1_63ca0b1fc50a3584_.log 2024-04-03T18:01:02.0650410Z Running 9 items in this shard: test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda], test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad, test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter, test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda], test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing, test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint, test/distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output 2024-04-03T18:01:02.0655721Z 2024-04-03T18:01:02.0656351Z Running distributed/pipeline/sync/skip/test_portal 1/1 ... [2024-04-03 18:01:02.064497] 2024-04-03T18:01:02.0658908Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_portal.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:01:02.065041] 2024-04-03T18:01:04.5851622Z 2024-04-03T18:01:04.5853746Z distributed/pipeline/sync/skip/test_portal 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_portal_1.1_cc20ba5afe23ded7_.log 2024-04-03T18:01:04.5861099Z Running 10 items in this shard: test/distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad, test/distributed/pipeline/sync/skip/test_portal.py::test_use_grad, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 2024-04-03T18:01:04.5867450Z 2024-04-03T18:01:04.5868082Z Running distributed/_shard/sharded_tensor/ops/test_init 1/1 ... [2024-04-03 18:01:04.585417] 2024-04-03T18:01:04.5870639Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:01:04.585831] 2024-04-03T18:01:18.2322914Z 2024-04-03T18:01:18.2325220Z distributed/_shard/sharded_tensor/ops/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_init_1.1_8a5aed4219eff876_.log 2024-04-03T18:01:18.2330355Z Running 3 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_kaiming_uniform, test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_normal, test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_uniform 2024-04-03T18:01:18.2333046Z 2024-04-03T18:01:18.2333678Z Running distributed/rpc/test_tensorpipe_agent 1/1 ... [2024-04-03 18:01:18.232604] 2024-04-03T18:01:18.2336739Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/rpc/test_tensorpipe_agent.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:01:18.233143] 2024-04-03T18:01:21.9963362Z 2024-04-03T18:01:21.9965504Z distributed/rpc/test_tensorpipe_agent 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.test_tensorpipe_agent_1.1_3341a2dc731117d2_.log 2024-04-03T18:01:21.9967065Z Running 0 items in this shard: 2024-04-03T18:01:21.9967407Z 2024-04-03T18:01:21.9970995Z Running distributed/pipeline/sync/test_copy 1/1 ... [2024-04-03 18:01:21.996711] 2024-04-03T18:01:21.9976917Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_copy.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:01:21.997247] 2024-04-03T18:01:25.3685500Z 2024-04-03T18:01:25.3687626Z distributed/pipeline/sync/test_copy 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_copy_1.1_9ca68a953fcc5127_.log 2024-04-03T18:01:25.3691800Z Running 5 items in this shard: test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda, test/distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors 2024-04-03T18:01:25.3694445Z 2024-04-03T18:01:25.3698023Z Running distributed/fsdp/test_fsdp_mixed_precision 1/1 ... [2024-04-03 18:01:25.369413] 2024-04-03T18:01:25.3706077Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:01:25.370107] 2024-04-03T18:06:22.1163826Z 2024-04-03T18:06:22.1165860Z distributed/fsdp/test_fsdp_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_mixed_precision_1.1_6f32697801a011a3_.log 2024-04-03T18:06:22.1228863Z Running 65 items in this shard: test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_eval_root_cast_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval_buffers, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval_comm, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_grads_reduced_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_input_grads_with_param_mixed_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_no_reshard_after_forward, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_resnet, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_batchnorm_convert_sync_bn_False, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_batchnorm_convert_sync_bn_True, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_default, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_only_params_and_bufs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_params_and_reduce_diff, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_reduce, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_grads_reduced_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_mixed_precision_e2e_full_shard, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_mixed_precision_no_reshard_after_forward, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionIgnoredModules::test_mixed_precision_with_ignored_module, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule_skip_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule_skip_inputs_error, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_different_precisions, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_different_precisions_error, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_external_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPTrainEval::test_train_ema_eval_flow 2024-04-03T18:06:22.1290040Z 2024-04-03T18:06:22.1290728Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 ... [2024-04-03 18:06:22.117367] 2024-04-03T18:06:22.1293375Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:06:22.117783] 2024-04-03T18:06:39.1699822Z 2024-04-03T18:06:39.1702223Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_binary_cmp_1.1_602bea0828bd0340_.log 2024-04-03T18:06:39.1706867Z Running 4 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_allclose, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_allclose_tensor_specs, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_equal, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_equal_tensor_specs 2024-04-03T18:06:39.1710156Z 2024-04-03T18:06:39.1710790Z Running distributed/pipeline/sync/skip/test_tracker 1/1 ... [2024-04-03 18:06:39.170179] 2024-04-03T18:06:39.1713339Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:06:39.170620] 2024-04-03T18:06:41.7403744Z 2024-04-03T18:06:41.7405767Z distributed/pipeline/sync/skip/test_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_tracker_1.1_c76e5079f4873f80_.log 2024-04-03T18:06:41.7411910Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker, test/distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel, test/distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal, test/distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal, test/distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing, test/distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing 2024-04-03T18:06:41.7415677Z 2024-04-03T18:06:41.7416312Z Running distributed/pipeline/sync/skip/test_leak 1/1 ... [2024-04-03 18:06:41.740633] 2024-04-03T18:06:41.7418847Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_leak.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:06:41.741123] 2024-04-03T18:06:44.8121194Z 2024-04-03T18:06:44.8123217Z distributed/pipeline/sync/skip/test_leak 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_leak_1.1_7a5939bbe6d2696a_.log 2024-04-03T18:06:44.8131342Z Running 8 items in this shard: test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] 2024-04-03T18:06:44.8136620Z 2024-04-03T18:06:44.8137283Z Running distributed/pipeline/sync/skip/test_stash_pop 1/1 ... [2024-04-03 18:06:44.812426] 2024-04-03T18:06:44.8139870Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_stash_pop.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:06:44.812895] 2024-04-03T18:06:47.1823223Z 2024-04-03T18:06:47.1825753Z distributed/pipeline/sync/skip/test_stash_pop 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_stash_pop_1.1_0d607cdf81cc8845_.log 2024-04-03T18:06:47.1830957Z Running 7 items in this shard: test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none 2024-04-03T18:06:47.1834900Z 2024-04-03T18:06:47.1835416Z Running distributed/test_c10d_spawn_gloo 1/1 ... [2024-04-03 18:06:47.182636] 2024-04-03T18:06:47.1837870Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:06:47.183077] 2024-04-03T18:08:00.4859746Z 2024-04-03T18:08:00.4861426Z distributed/test_c10d_spawn_gloo 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_gloo_1.1_b518259205201e28_.log 2024-04-03T18:08:00.4870835Z Running 11 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cpu, test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cuda, test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_rnn, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all_single, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_allreduce, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_broadcast, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_gather, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_reduce, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_scatter 2024-04-03T18:08:00.4878809Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cpu 2024-04-03T18:08:00.4880545Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cuda 2024-04-03T18:08:00.4882282Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_rnn 2024-04-03T18:08:00.4883929Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather 2024-04-03T18:08:00.4885743Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all 2024-04-03T18:08:00.4887446Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all_single 2024-04-03T18:08:00.4889582Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_allreduce 2024-04-03T18:08:00.4891200Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_broadcast 2024-04-03T18:08:00.4892786Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_gather 2024-04-03T18:08:00.4894348Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_reduce 2024-04-03T18:08:00.4895911Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_scatter 2024-04-03T18:08:00.4896805Z 2024-04-03T18:08:00.4897279Z Running distributed/test_c10d_ucc 1/1 ... [2024-04-03 18:08:00.486645] 2024-04-03T18:08:00.4899619Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_ucc.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:08:00.487085] 2024-04-03T18:15:52.1117780Z 2024-04-03T18:15:52.1120778Z distributed/test_c10d_ucc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_ucc_1.1_5752da3391f6adfd_.log 2024-04-03T18:15:52.1172035Z Running 71 items in this shard: test/distributed/test_c10d_ucc.py::RendezvousEnvTest::test_logging_init, test/distributed/test_c10d_ucc.py::TimeoutTest::test_default_store_timeout_ucc, test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_allgather_basics, test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_allreduce_basics, test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_barrier_implies_wait, test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_broadcast_basics, test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_empty_tensors, test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_reduce_basics, test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_send_recv_all_to_all, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_ucc, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_global_local_unused_params_grad, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ignored_output, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_save_load_checkpoint, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sparse_gradients, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_2gpu_module, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_4gpu_module, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_cpu_module, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_cpu_module_grad_is_view, test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_incremented_ucc_default, test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_incremented_ucc_subgroup, test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_set_default_pg_ucc, test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_set_ucc_new_group, test/distributed/test_c10d_ucc.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_ucc.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_ucc.py::CommTest::test_ucc_barrier_device_ids, test/distributed/test_c10d_ucc.py::CommTest::test_ucc_rank_membership, test/distributed/test_c10d_ucc.py::CommTest::test_ucc_warn_not_in_group, test/distributed/test_c10d_ucc.py::CompilerTest::test_allgather_work_wait_cpu, test/distributed/test_c10d_ucc.py::CompilerTest::test_allgather_work_wait_gpu, test/distributed/test_c10d_ucc.py::CompilerTest::test_allreduce_work_wait_cpu, test/distributed/test_c10d_ucc.py::CompilerTest::test_allreduce_work_wait_gpu, test/distributed/test_c10d_ucc.py::CompilerTest::test_broadcast_work_wait_cpu, test/distributed/test_c10d_ucc.py::CompilerTest::test_broadcast_work_wait_gpu, test/distributed/test_c10d_ucc.py::CompilerTest::test_consecutive_comm_work_wait_cpu, test/distributed/test_c10d_ucc.py::CompilerTest::test_consecutive_comm_work_wait_gpu, test/distributed/test_c10d_ucc.py::CompilerTest::test_nested_comm_tensor_wrapping_cpu, test/distributed/test_c10d_ucc.py::CompilerTest::test_nested_comm_tensor_wrapping_gpu, test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_allgather_base, test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-04-03T18:15:52.1216072Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::RendezvousEnvTest::test_logging_init 2024-04-03T18:15:52.1217500Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::TimeoutTest::test_default_store_timeout_ucc 2024-04-03T18:15:52.1218948Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_allgather_basics 2024-04-03T18:15:52.1220447Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_allreduce_basics 2024-04-03T18:15:52.1221910Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_barrier_implies_wait 2024-04-03T18:15:52.1223382Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_broadcast_basics 2024-04-03T18:15:52.1224824Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_empty_tensors 2024-04-03T18:15:52.1226208Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_reduce_basics 2024-04-03T18:15:52.1227652Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_send_recv_all_to_all 2024-04-03T18:15:52.1229185Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_dataclass_output 2024-04-03T18:15:52.1230836Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_dataclass_output_unused_param 2024-04-03T18:15:52.1232570Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2024-04-03T18:15:52.1234396Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2024-04-03T18:15:52.1236445Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2024-04-03T18:15:52.1238322Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2024-04-03T18:15:52.1240270Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False 2024-04-03T18:15:52.1242324Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2024-04-03T18:15:52.1244299Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2024-04-03T18:15:52.1246192Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2024-04-03T18:15:52.1248050Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing 2024-04-03T18:15:52.1250461Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False 2024-04-03T18:15:52.1252456Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2024-04-03T18:15:52.1254447Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False 2024-04-03T18:15:52.1256557Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2024-04-03T18:15:52.1258517Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu 2024-04-03T18:15:52.1260318Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_ucc 2024-04-03T18:15:52.1262106Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once 2024-04-03T18:15:52.1263840Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients 2024-04-03T18:15:52.1265550Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init 2024-04-03T18:15:52.1267284Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type 2024-04-03T18:15:52.1269171Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty 2024-04-03T18:15:52.1271038Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_global_local_unused_params_grad 2024-04-03T18:15:52.1272888Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view 2024-04-03T18:15:52.1274853Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph 2024-04-03T18:15:52.1276577Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ignored_output 2024-04-03T18:15:52.1278268Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters 2024-04-03T18:15:52.1280005Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_invalid_powerSGD_state 2024-04-03T18:15:52.1281790Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_save_load_checkpoint 2024-04-03T18:15:52.1283360Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sparse_gradients 2024-04-03T18:15:52.1285004Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view 2024-04-03T18:15:52.1286718Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2024-04-03T18:15:52.1288450Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input 2024-04-03T18:15:52.1290873Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_1gpu_module_device_ids_integer_list 2024-04-03T18:15:52.1292856Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_1gpu_module_device_ids_torch_device_list 2024-04-03T18:15:52.1294675Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_2gpu_module 2024-04-03T18:15:52.1296328Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_4gpu_module 2024-04-03T18:15:52.1297950Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_cpu_module 2024-04-03T18:15:52.1299660Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_cpu_module_grad_is_view 2024-04-03T18:15:52.1301457Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_incremented_ucc_default 2024-04-03T18:15:52.1303018Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_incremented_ucc_subgroup 2024-04-03T18:15:52.1304513Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_set_default_pg_ucc 2024-04-03T18:15:52.1305975Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_set_ucc_new_group 2024-04-03T18:15:52.1307379Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_tensor_dtype_complex 2024-04-03T18:15:52.1308706Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_tensor_dtype_mismatch 2024-04-03T18:15:52.1310053Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_ucc_barrier_device_ids 2024-04-03T18:15:52.1311399Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_ucc_rank_membership 2024-04-03T18:15:52.1312733Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_ucc_warn_not_in_group 2024-04-03T18:15:52.1314112Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_allgather_work_wait_cpu 2024-04-03T18:15:52.1315543Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_allgather_work_wait_gpu 2024-04-03T18:15:52.1316973Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_allreduce_work_wait_cpu 2024-04-03T18:15:52.1318383Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_allreduce_work_wait_gpu 2024-04-03T18:15:52.1319805Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_broadcast_work_wait_cpu 2024-04-03T18:15:52.1321222Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_broadcast_work_wait_gpu 2024-04-03T18:15:52.1322691Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_consecutive_comm_work_wait_cpu 2024-04-03T18:15:52.1324180Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_consecutive_comm_work_wait_gpu 2024-04-03T18:15:52.1325832Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_nested_comm_tensor_wrapping_cpu 2024-04-03T18:15:52.1327363Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_nested_comm_tensor_wrapping_gpu 2024-04-03T18:15:52.1329048Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_allgather_base 2024-04-03T18:15:52.1331613Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_collectives 2024-04-03T18:15:52.1333582Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-04-03T18:15:52.1335680Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-04-03T18:15:52.1336835Z 2024-04-03T18:15:52.1337571Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout 1/1 ... [2024-04-03 18:15:52.114351] 2024-04-03T18:15:52.1340264Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:15:52.114848] 2024-04-03T18:15:54.5343721Z 2024-04-03T18:15:54.5346168Z distributed/pipeline/sync/skip/test_inspect_skip_layout 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_inspect_skip_layout_1.1_9e1d6c75a7047875_.log 2024-04-03T18:15:54.5352463Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace 2024-04-03T18:15:54.5356520Z 2024-04-03T18:15:54.5357238Z Running distributed/_shard/sharded_optim/test_sharded_optim 1/1 ... [2024-04-03 18:15:54.534611] 2024-04-03T18:15:54.5359855Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:15:54.535029] 2024-04-03T18:16:04.7747030Z 2024-04-03T18:16:04.7750224Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_optim.test_sharded_optim_1.1_0eb7e660423b38d1_.log 2024-04-03T18:16:04.7754656Z Running 2 items in this shard: test/distributed/_shard/sharded_optim/test_sharded_optim.py::TestShardedOptimizer::test_named_params_with_sharded_tensor, test/distributed/_shard/sharded_optim/test_sharded_optim.py::TestShardedOptimizer::test_sharded_optim 2024-04-03T18:16:04.7756405Z 2024-04-03T18:16:04.7756972Z Running distributed/test_launcher 1/1 ... [2024-04-03 18:16:04.774852] 2024-04-03T18:16:04.7760329Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_launcher.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:16:04.775271] 2024-04-03T18:16:09.3998612Z 2024-04-03T18:16:09.4000505Z distributed/test_launcher 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_launcher_1.1_b45e3f774af04074_.log 2024-04-03T18:16:09.4002856Z Running 1 items in this shard: test/distributed/test_launcher.py::TestDistributedLaunch::test_launch_user_script 2024-04-03T18:16:09.4003690Z 2024-04-03T18:16:09.4004286Z Running distributed/optim/test_zero_redundancy_optimizer 1/1 ... [2024-04-03 18:16:09.400060] 2024-04-03T18:16:09.4009794Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_zero_redundancy_optimizer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:16:09.400487] 2024-04-03T18:19:22.7729745Z 2024-04-03T18:19:22.7731957Z distributed/optim/test_zero_redundancy_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_zero_redundancy_optimizer_1.1_6e5f0d7c773bd51a_.log 2024-04-03T18:19:22.7778728Z Running 42 items in this shard: test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_constructor, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_lr_scheduler, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_same_dense_param_type, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_state_dict, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_with_extra_inner_key, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_with_kwargs, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_without_closure, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_zero_grad, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_add_param_group, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_collect_shards, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_lr_scheduler, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_multiple_param_groups, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_nondefault_process_group, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_sharding, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_step, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_step_with_closure, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_join_cpu, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_join_gpu, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_model_parallel_parameters_as_bucket_view_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_model_parallel_parameters_as_bucket_view_True 2024-04-03T18:19:22.7823674Z 2024-04-03T18:19:22.7824174Z Running distributed/test_c10d_nccl 1/2 ... [2024-04-03 18:19:22.773387] 2024-04-03T18:19:22.7826712Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=1', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:19:22.773801] 2024-04-03T18:30:55.3473627Z 2024-04-03T18:30:55.3475626Z distributed/test_c10d_nccl 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_1.2_f89881d3fdec9ff8_.log 2024-04-03T18:30:55.3530513Z Running 83 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_abort_in_destroy_multi_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_abort_in_destroy_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_abort_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_allreduce_in_cudagraph, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_allreduce_ops, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_broadcast_ops, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_close_multi_pg_unordered, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_close_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_comm_split_subgroup, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_destruct_before_terminate_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_empty_tensors, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_gather_checks, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_get_uid, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_non_blocking_init, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_non_blocking_with_eager_init, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_reduce_ops, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_reduce_scatter_base_ops, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_scatter_stress, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_set_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_set_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_sparse_allreduce_ops, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking, test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl, test/distributed/test_c10d_nccl.py::CompilerTest::test_allgather_into_tensor_work_wait_gpu, test/distributed/test_c10d_nccl.py::CompilerTest::test_alltoall_work_wait_gpu, test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_base_k, test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_nccl.py::CompilerTest::test_scatter_work_wait_gpu, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTestDumpOnTimeout::test_timeout_dumps_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTestDumpOnTimeout::test_timeout_dumps_timing_enabled_True, test/distributed/test_c10d_nccl.py::NcclErrorDumpTest::test_nccl_errors_dump 2024-04-03T18:30:55.3580108Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors 2024-04-03T18:30:55.3581638Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus 2024-04-03T18:30:55.3583203Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_abort_in_destroy_multi_pgs 2024-04-03T18:30:55.3584765Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_abort_in_destroy_pg 2024-04-03T18:30:55.3586190Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_abort_pg 2024-04-03T18:30:55.3587659Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_allreduce_in_cudagraph 2024-04-03T18:30:55.3589164Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_allreduce_ops 2024-04-03T18:30:55.3590609Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_broadcast_ops 2024-04-03T18:30:55.3592101Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_close_multi_pg_unordered 2024-04-03T18:30:55.3593567Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_close_pg 2024-04-03T18:30:55.3595000Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_comm_split_subgroup 2024-04-03T18:30:55.3596705Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_destruct_before_terminate_pg 2024-04-03T18:30:55.3598268Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_empty_tensors 2024-04-03T18:30:55.3599706Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_gather_checks 2024-04-03T18:30:55.3601099Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_get_uid 2024-04-03T18:30:55.3602498Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_non_blocking_init 2024-04-03T18:30:55.3604042Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_non_blocking_with_eager_init 2024-04-03T18:30:55.3605550Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_reduce_ops 2024-04-03T18:30:55.3607020Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_reduce_scatter_base_ops 2024-04-03T18:30:55.3608520Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_scatter_stress 2024-04-03T18:30:55.3610461Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_set_nccl_pg_timeout_backend0 2024-04-03T18:30:55.3612115Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_set_nccl_pg_timeout_backend_nccl 2024-04-03T18:30:55.3613700Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLTest::test_sparse_allreduce_ops 2024-04-03T18:30:55.3615342Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value 2024-04-03T18:30:55.3617177Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view 2024-04-03T18:30:55.3619014Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view 2024-04-03T18:30:55.3620801Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view 2024-04-03T18:30:55.3622737Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig 2024-04-03T18:30:55.3624495Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2024-04-03T18:30:55.3626459Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2024-04-03T18:30:55.3628396Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing 2024-04-03T18:30:55.3630342Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False 2024-04-03T18:30:55.3632257Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl 2024-04-03T18:30:55.3634121Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl 2024-04-03T18:30:55.3636011Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl 2024-04-03T18:30:55.3637809Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config 2024-04-03T18:30:55.3639483Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing 2024-04-03T18:30:55.3641222Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery 2024-04-03T18:30:55.3642953Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info 2024-04-03T18:30:55.3644813Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off 2024-04-03T18:30:55.3646753Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail 2024-04-03T18:30:55.3648777Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info 2024-04-03T18:30:55.3651205Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off 2024-04-03T18:30:55.3652930Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16 2024-04-03T18:30:55.3654471Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state 2024-04-03T18:30:55.3656266Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None 2024-04-03T18:30:55.3658226Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None 2024-04-03T18:30:55.3659951Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp 2024-04-03T18:30:55.3661536Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env 2024-04-03T18:30:55.3663203Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit 2024-04-03T18:30:55.3664876Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill 2024-04-03T18:30:55.3666510Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm 2024-04-03T18:30:55.3668285Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking 2024-04-03T18:30:55.3669756Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce 2024-04-03T18:30:55.3671085Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier 2024-04-03T18:30:55.3672405Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids 2024-04-03T18:30:55.3673896Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument 2024-04-03T18:30:55.3675424Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off 2024-04-03T18:30:55.3676953Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup 2024-04-03T18:30:55.3678491Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl 2024-04-03T18:30:55.3680009Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_allgather_into_tensor_work_wait_gpu 2024-04-03T18:30:55.3681514Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_alltoall_work_wait_gpu 2024-04-03T18:30:55.3682944Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_base_k 2024-04-03T18:30:55.3684426Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_tensor_coalesced 2024-04-03T18:30:55.3686003Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_scatter_work_wait_gpu 2024-04-03T18:30:55.3687678Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base 2024-04-03T18:30:55.3689963Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives 2024-04-03T18:30:55.3691755Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0 2024-04-03T18:30:55.3693413Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1 2024-04-03T18:30:55.3694941Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup 2024-04-03T18:30:55.3696415Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_True 2024-04-03T18:30:55.3698136Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True 2024-04-03T18:30:55.3699989Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False 2024-04-03T18:30:55.3701783Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False 2024-04-03T18:30:55.3703424Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_timing_enabled_False 2024-04-03T18:30:55.3704890Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_timing_enabled_True 2024-04-03T18:30:55.3706453Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False 2024-04-03T18:30:55.3708103Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True 2024-04-03T18:30:55.3709742Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False 2024-04-03T18:30:55.3711518Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True 2024-04-03T18:30:55.3713217Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTestDumpOnTimeout::test_timeout_dumps_timing_enabled_False 2024-04-03T18:30:55.3714988Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTestDumpOnTimeout::test_timeout_dumps_timing_enabled_True 2024-04-03T18:30:55.3716574Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorDumpTest::test_nccl_errors_dump 2024-04-03T18:30:55.3717387Z 2024-04-03T18:30:55.3717960Z Running distributed/elastic/timer/api_test 1/1 ... [2024-04-03 18:30:55.351291] 2024-04-03T18:30:55.3720389Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/timer/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 18:30:55.351722] 2024-04-03T18:30:57.2022095Z 2024-04-03T18:30:57.2023954Z distributed/elastic/timer/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.api_test_1.1_c5873c44dc3fc7d9_.log 2024-04-03T18:30:57.2025339Z 2024-04-03T18:30:57.7958765Z 2024-04-03T18:30:57.7960088Z real 127m15.157s 2024-04-03T18:30:57.7960971Z user 223m6.397s 2024-04-03T18:30:57.7961686Z sys 91m58.081s 2024-04-03T18:30:57.7962347Z + assert_git_not_dirty 2024-04-03T18:30:57.7963680Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-04-03T18:30:57.7964490Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-04-03T18:30:57.7965355Z ++ git status --porcelain 2024-04-03T18:30:57.7967005Z ++ grep -v '?? third_party' 2024-04-03T18:30:59.3730700Z ++ true 2024-04-03T18:30:59.3731871Z + git_status= 2024-04-03T18:30:59.3732467Z + [[ -n '' ]] 2024-04-03T18:30:59.3733246Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-04-03T18:30:59.3733878Z + [[ 1 == 1 ]] 2024-04-03T18:30:59.3734633Z + echo 'Testing distributed C++ tests' 2024-04-03T18:30:59.3735227Z Testing distributed C++ tests 2024-04-03T18:30:59.3740547Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-04-03T18:30:59.3750779Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-04-03T18:30:59.3759267Z + export CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-04-03T18:30:59.3760451Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-04-03T18:30:59.3761467Z + python test/run_test.py --cpp --verbose -i cpp/FileStoreTest 2024-04-03T18:30:59.4888322Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-04-03T18:30:59.4890099Z import pkg_resources 2024-04-03T18:31:02.7502223Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-04-03T18:31:02.7504393Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-04-03T18:31:02.7585722Z Found test times from artifacts 2024-04-03T18:31:02.7916977Z Found test times from artifacts 2024-04-03T18:31:02.7927761Z Running all tests 2024-04-03T18:31:02.7929382Z Name: tests to run (est. time: 0.0min) 2024-04-03T18:31:02.7929948Z Serial tests (1): 2024-04-03T18:31:02.7930372Z cpp/FileStoreTest 1/1 2024-04-03T18:31:02.7930814Z Parallel tests (0): 2024-04-03T18:31:02.7931233Z Name: excluded (est. time: 0.0min) 2024-04-03T18:31:02.7931734Z Serial tests (0): 2024-04-03T18:31:02.7932115Z Parallel tests (0): 2024-04-03T18:31:02.7932888Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-04-03T18:31:02.7994548Z Running cpp/FileStoreTest 1/1 ... [2024-04-03 18:31:02.799127] 2024-04-03T18:31:02.8003804Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/FileStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-b0ee8f362d1378d5.xml', '-x', '--reruns=2'] ... [2024-04-03 18:31:02.799914] 2024-04-03T18:31:08.0674676Z 2024-04-03T18:31:08.0676216Z cpp/FileStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.FileStoreTest_1.1_c488c8fb6ae9e202_.log 2024-04-03T18:31:08.0677516Z 2024-04-03T18:31:08.7036823Z + python test/run_test.py --cpp --verbose -i cpp/HashStoreTest 2024-04-03T18:31:08.8163081Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-04-03T18:31:08.8164550Z import pkg_resources 2024-04-03T18:31:11.9732125Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-04-03T18:31:11.9734631Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-04-03T18:31:11.9816221Z Found test times from artifacts 2024-04-03T18:31:12.0147309Z Found test times from artifacts 2024-04-03T18:31:12.0158021Z Running all tests 2024-04-03T18:31:12.0158730Z Name: tests to run (est. time: 0.0min) 2024-04-03T18:31:12.0159291Z Serial tests (1): 2024-04-03T18:31:12.0159719Z cpp/HashStoreTest 1/1 2024-04-03T18:31:12.0160140Z Parallel tests (0): 2024-04-03T18:31:12.0160592Z Name: excluded (est. time: 0.0min) 2024-04-03T18:31:12.0161064Z Serial tests (0): 2024-04-03T18:31:12.0161446Z Parallel tests (0): 2024-04-03T18:31:12.0162224Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-04-03T18:31:12.0221273Z Running cpp/HashStoreTest 1/1 ... [2024-04-03 18:31:12.021743] 2024-04-03T18:31:12.0230193Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/HashStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-fce664ef67b6bc2e.xml', '-x', '--reruns=2'] ... [2024-04-03 18:31:12.022539] 2024-04-03T18:31:15.3666946Z 2024-04-03T18:31:15.3668431Z cpp/HashStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.HashStoreTest_1.1_a437887af7a7d024_.log 2024-04-03T18:31:15.3669591Z 2024-04-03T18:31:15.9939003Z + python test/run_test.py --cpp --verbose -i cpp/TCPStoreTest 2024-04-03T18:31:16.1057434Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-04-03T18:31:16.1058855Z import pkg_resources 2024-04-03T18:31:19.2366096Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-04-03T18:31:19.2368284Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-04-03T18:31:19.2449594Z Found test times from artifacts 2024-04-03T18:31:19.2780335Z Found test times from artifacts 2024-04-03T18:31:19.2790291Z Running all tests 2024-04-03T18:31:19.2791909Z Name: tests to run (est. time: 0.0min) 2024-04-03T18:31:19.2792439Z Serial tests (1): 2024-04-03T18:31:19.2793047Z cpp/TCPStoreTest 1/1 2024-04-03T18:31:19.2793508Z Parallel tests (0): 2024-04-03T18:31:19.2793940Z Name: excluded (est. time: 0.0min) 2024-04-03T18:31:19.2794412Z Serial tests (0): 2024-04-03T18:31:19.2794807Z Parallel tests (0): 2024-04-03T18:31:19.2795590Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-04-03T18:31:19.2852693Z Running cpp/TCPStoreTest 1/1 ... [2024-04-03 18:31:19.284884] 2024-04-03T18:31:19.2861271Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/TCPStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-bd53757de6d12bb2.xml', '-x', '--reruns=2'] ... [2024-04-03 18:31:19.285637] 2024-04-03T18:31:37.1144026Z 2024-04-03T18:31:37.1146057Z cpp/TCPStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.TCPStoreTest_1.1_75e357ac54058d35_.log 2024-04-03T18:31:37.1147198Z 2024-04-03T18:31:37.7462606Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-04-03T18:31:37.7464501Z ++ command -v mpiexec 2024-04-03T18:31:37.7468290Z + MPIEXEC=/usr/bin/mpiexec 2024-04-03T18:31:37.7468852Z + [[ -n /usr/bin/mpiexec ]] 2024-04-03T18:31:37.7469993Z + MPICMD='/usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest' 2024-04-03T18:31:37.7471605Z + eval '/usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest' 2024-04-03T18:31:37.7473246Z ++ /usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest 2024-04-03T18:31:40.7959428Z Test successful 2024-04-03T18:31:40.8014219Z Test successful 2024-04-03T18:31:40.9504518Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupGlooTest 2024-04-03T18:31:41.0626882Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-04-03T18:31:41.0628346Z import pkg_resources 2024-04-03T18:31:44.1418661Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-04-03T18:31:44.1420825Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-04-03T18:31:44.1500790Z Found test times from artifacts 2024-04-03T18:31:44.1827819Z Found test times from artifacts 2024-04-03T18:31:44.1837443Z Running all tests 2024-04-03T18:31:44.1838335Z Name: tests to run (est. time: 0.0min) 2024-04-03T18:31:44.1838895Z Serial tests (1): 2024-04-03T18:31:44.1839327Z cpp/ProcessGroupGlooTest 1/1 2024-04-03T18:31:44.1839819Z Parallel tests (0): 2024-04-03T18:31:44.1840490Z Name: excluded (est. time: 0.0min) 2024-04-03T18:31:44.1841240Z Serial tests (0): 2024-04-03T18:31:44.1841628Z Parallel tests (0): 2024-04-03T18:31:44.1842415Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-04-03T18:31:44.1898040Z Running cpp/ProcessGroupGlooTest 1/1 ... [2024-04-03 18:31:44.189469] 2024-04-03T18:31:44.1906429Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupGlooTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-640ef59b3f9998ac.xml', '-x', '--reruns=2'] ... [2024-04-03 18:31:44.190173] 2024-04-03T18:31:52.7377375Z 2024-04-03T18:31:52.7379332Z cpp/ProcessGroupGlooTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupGlooTest_1.1_54857242c1147efe_.log 2024-04-03T18:31:52.7381317Z 2024-04-03T18:31:53.3486354Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupNCCLTest 2024-04-03T18:31:53.4879469Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-04-03T18:31:53.4880920Z import pkg_resources 2024-04-03T18:31:56.6389749Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-04-03T18:31:56.6391933Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-04-03T18:31:56.6472602Z Found test times from artifacts 2024-04-03T18:31:56.6802864Z Found test times from artifacts 2024-04-03T18:31:56.6813359Z Running all tests 2024-04-03T18:31:56.6814421Z Name: tests to run (est. time: 0.0min) 2024-04-03T18:31:56.6814932Z Serial tests (1): 2024-04-03T18:31:56.6815430Z cpp/ProcessGroupNCCLTest 1/1 2024-04-03T18:31:56.6815902Z Parallel tests (0): 2024-04-03T18:31:56.6816328Z Name: excluded (est. time: 0.0min) 2024-04-03T18:31:56.6816824Z Serial tests (0): 2024-04-03T18:31:56.6817214Z Parallel tests (0): 2024-04-03T18:31:56.6817982Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-04-03T18:31:56.6875415Z Running cpp/ProcessGroupNCCLTest 1/1 ... [2024-04-03 18:31:56.687203] 2024-04-03T18:31:56.6884659Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-baf903786c4b47bd.xml', '-x', '--reruns=2'] ... [2024-04-03 18:31:56.687940] 2024-04-03T18:32:22.4043922Z 2024-04-03T18:32:22.4045611Z cpp/ProcessGroupNCCLTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupNCCLTest_1.1_f74d0a8259aa79d9_.log 2024-04-03T18:32:22.4047188Z 2024-04-03T18:32:23.0149854Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupNCCLErrorsTest 2024-04-03T18:32:23.1269416Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-04-03T18:32:23.1270890Z import pkg_resources 2024-04-03T18:32:26.2949218Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-04-03T18:32:26.2951365Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-04-03T18:32:26.3032670Z Found test times from artifacts 2024-04-03T18:32:26.3364246Z Found test times from artifacts 2024-04-03T18:32:26.3374997Z Running all tests 2024-04-03T18:32:26.3376263Z Name: tests to run (est. time: 0.0min) 2024-04-03T18:32:26.3376792Z Serial tests (1): 2024-04-03T18:32:26.3377257Z cpp/ProcessGroupNCCLErrorsTest 1/1 2024-04-03T18:32:26.3377797Z Parallel tests (0): 2024-04-03T18:32:26.3378273Z Name: excluded (est. time: 0.0min) 2024-04-03T18:32:26.3378699Z Serial tests (0): 2024-04-03T18:32:26.3379144Z Parallel tests (0): 2024-04-03T18:32:26.3379948Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-04-03T18:32:26.3438548Z Running cpp/ProcessGroupNCCLErrorsTest 1/1 ... [2024-04-03 18:32:26.343447] 2024-04-03T18:32:26.3446801Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLErrorsTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-7f68d0f547b77d93.xml', '-x', '--reruns=2'] ... [2024-04-03 18:32:26.344198] 2024-04-03T18:32:38.2246359Z 2024-04-03T18:32:38.2248139Z cpp/ProcessGroupNCCLErrorsTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupNCCLErrorsTest_1.1_78f69ca470e83b20_.log 2024-04-03T18:32:38.2250049Z 2024-04-03T18:32:38.8489412Z + [[ 1 == 1 ]] 2024-04-03T18:32:38.8489882Z + test_rpc 2024-04-03T18:32:38.8490393Z + echo 'Testing RPC C++ tests' 2024-04-03T18:32:38.8491266Z Testing RPC C++ tests 2024-04-03T18:32:38.8496481Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-04-03T18:32:38.8508481Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-04-03T18:32:38.8519660Z + ln -sf '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtbb*' /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-04-03T18:32:38.8529617Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-04-03T18:32:38.8530916Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc 2024-04-03T18:32:38.9682337Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-04-03T18:32:38.9683763Z import pkg_resources 2024-04-03T18:32:42.1276978Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-04-03T18:32:42.1279544Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-04-03T18:32:42.1360860Z Found test times from artifacts 2024-04-03T18:32:42.1695393Z Found test times from artifacts 2024-04-03T18:32:42.1705697Z Running all tests 2024-04-03T18:32:42.1706928Z Name: tests to run (est. time: 0.0min) 2024-04-03T18:32:42.1707419Z Serial tests (1): 2024-04-03T18:32:42.1707884Z cpp/test_cpp_rpc 1/1 2024-04-03T18:32:42.1708233Z Parallel tests (0): 2024-04-03T18:32:42.1708668Z Name: excluded (est. time: 0.0min) 2024-04-03T18:32:42.1709134Z Serial tests (0): 2024-04-03T18:32:42.1709533Z Parallel tests (0): 2024-04-03T18:32:42.1710311Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-04-03T18:32:42.1767767Z Running cpp/test_cpp_rpc 1/1 ... [2024-04-03 18:32:42.176437] 2024-04-03T18:32:42.1777487Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-1fa2a2392c591fc5.xml', '-x', '--reruns=2'] ... [2024-04-03 18:32:42.177226] 2024-04-03T18:32:47.7476295Z 2024-04-03T18:32:47.7477913Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_6ed2ad04e9d1d6f0_.log 2024-04-03T18:32:47.7478964Z 2024-04-03T18:32:48.3910414Z ##[group]Run cat test/**/*_toprint.log || true 2024-04-03T18:32:48.3911104Z cat test/**/*_toprint.log || true 2024-04-03T18:32:48.3919611Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:32:48.3920242Z env: 2024-04-03T18:32:48.3920566Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:32:48.3921132Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:32:48.3922087Z DOCKER_CONTAINER_ID: b758c44c85badb76c4c382a8895cd5e1bfdf37df294e5f1babfcbd8fbafbdd0d 2024-04-03T18:32:48.3922907Z ##[endgroup] 2024-04-03T18:32:48.3996637Z cat: test/**/*_toprint.log: No such file or directory 2024-04-03T18:32:48.4035463Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-04-03T18:32:48.4051706Z kill "$MONITOR_SCRIPT_PID" 2024-04-03T18:32:48.4060089Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:32:48.4060702Z env: 2024-04-03T18:32:48.4061048Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:32:48.4061616Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:32:48.4062552Z DOCKER_CONTAINER_ID: b758c44c85badb76c4c382a8895cd5e1bfdf37df294e5f1babfcbd8fbafbdd0d 2024-04-03T18:32:48.4063419Z MONITOR_SCRIPT_PID: 102557 2024-04-03T18:32:48.4063866Z ##[endgroup] 2024-04-03T18:32:48.4277308Z Prepare all required actions 2024-04-03T18:32:48.4277899Z Getting action download info 2024-04-03T18:32:48.5629156Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-04-03T18:32:48.7570977Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-04-03T18:32:48.7571562Z with: 2024-04-03T18:32:48.7572174Z file-suffix: test-distributed-1-3-linux.8xlarge.nvidia.gpu_23402705639 2024-04-03T18:32:48.7572958Z s3-bucket: gha-artifacts 2024-04-03T18:32:48.7573382Z env: 2024-04-03T18:32:48.7573719Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:32:48.7574282Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:32:48.7575219Z DOCKER_CONTAINER_ID: b758c44c85badb76c4c382a8895cd5e1bfdf37df294e5f1babfcbd8fbafbdd0d 2024-04-03T18:32:48.7576062Z ##[endgroup] 2024-04-03T18:32:48.7608459Z ##[group]Run # Remove any previous test jsons if they exist 2024-04-03T18:32:48.7609551Z # Remove any previous test jsons if they exist 2024-04-03T18:32:48.7610191Z rm -f test-jsons-*.zip 2024-04-03T18:32:48.7610847Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-04-03T18:32:48.7619392Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:32:48.7620022Z env: 2024-04-03T18:32:48.7620347Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:32:48.7620916Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:32:48.7621876Z DOCKER_CONTAINER_ID: b758c44c85badb76c4c382a8895cd5e1bfdf37df294e5f1babfcbd8fbafbdd0d 2024-04-03T18:32:48.7623003Z FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_23402705639 2024-04-03T18:32:48.7623768Z ##[endgroup] 2024-04-03T18:32:48.7899992Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-04-03T18:32:48.7938506Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-04-03T18:32:48.7939380Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-04-03T18:32:48.7946907Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-04-03T18:32:48.7947950Z adding: test/.pytorch-slow-tests.json (deflated 59%) 2024-04-03T18:32:48.7957011Z adding: test/.pytorch-disabled-tests.json (deflated 84%) 2024-04-03T18:32:48.7993339Z ##[group]Run # Remove any previous test reports if they exist 2024-04-03T18:32:48.7994168Z # Remove any previous test reports if they exist 2024-04-03T18:32:48.7995161Z rm -f test-reports-*.zip 2024-04-03T18:32:48.7996001Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-04-03T18:32:48.8004622Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:32:48.8005257Z env: 2024-04-03T18:32:48.8005583Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:32:48.8006155Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:32:48.8007116Z DOCKER_CONTAINER_ID: b758c44c85badb76c4c382a8895cd5e1bfdf37df294e5f1babfcbd8fbafbdd0d 2024-04-03T18:32:48.8008217Z FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_23402705639 2024-04-03T18:32:48.8008975Z ##[endgroup] 2024-04-03T18:32:48.8300671Z adding: test/test-reports/python-pytest/distributed._tensor.debug.test_op_coverage/distributed._tensor.debug.test_op_coverage-0285e689d574fd03.xml (deflated 34%) 2024-04-03T18:32:48.8302984Z adding: test/test-reports/python-pytest/distributed._composable.test_contract/distributed._composable.test_contract-6edfadecd19f50e5.xml (deflated 65%) 2024-04-03T18:32:48.8305204Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_traverse/distributed.checkpoint.test_traverse-aacc8a590eb659ad.xml (deflated 75%) 2024-04-03T18:32:48.8307638Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_fx/distributed.fsdp.test_fsdp_fx-27bdfae798935cb9.xml (deflated 37%) 2024-04-03T18:32:48.8309912Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_state/distributed._composable.fsdp.test_fully_shard_state-a2c121f37fc77701.xml (deflated 73%) 2024-04-03T18:32:48.8312539Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dedup_tensors/distributed.checkpoint.test_dedup_tensors-383e1cee9f42fe52.xml (deflated 37%) 2024-04-03T18:32:48.8314992Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_compatibility/distributed.checkpoint.test_compatibility-859b741cf9aad5e2.xml (deflated 66%) 2024-04-03T18:32:48.8317376Z adding: test/test-reports/python-pytest/distributed.checkpoint.fsdp.test_fsdp_dsd/distributed.checkpoint.fsdp.test_fsdp_dsd-c74adf8fa1797e22.xml (deflated 51%) 2024-04-03T18:32:48.8319653Z adding: test/test-reports/python-pytest/distributed._tensor.test_common_rules/distributed._tensor.test_common_rules-37f52af081564b26.xml (deflated 86%) 2024-04-03T18:32:48.8321820Z adding: test/test-reports/python-pytest/distributed.test_c10d_functional_native/distributed.test_c10d_functional_native-545550538d88fd53.xml (deflated 90%) 2024-04-03T18:32:48.8323993Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_ops/distributed._tensor.test_dtensor_ops-f66e46fc3ddba25e.xml (deflated 28%) 2024-04-03T18:32:48.8326228Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_tp_checkpoint/distributed.checkpoint.test_tp_checkpoint-d1970f58adaf0edb.xml (deflated 52%) 2024-04-03T18:32:48.8328485Z adding: test/test-reports/python-pytest/distributed._tensor.test_tensor_ops/distributed._tensor.test_tensor_ops-ceb2684f89f9a368.xml (deflated 83%) 2024-04-03T18:32:48.8331133Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_init/distributed._composable.fsdp.test_fully_shard_init-afb830a9b8ec25fe.xml (deflated 82%) 2024-04-03T18:32:48.8333582Z adding: test/test-reports/python-pytest/distributed._tensor.test_xla_integration/distributed._tensor.test_xla_integration-7717710139db866b.xml (deflated 74%) 2024-04-03T18:32:48.8335873Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_format_utils/distributed.checkpoint.test_format_utils-1d72cb580e1bc8f6.xml (deflated 57%) 2024-04-03T18:32:48.8338100Z adding: test/test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-abee4fddf50daafc.xml (deflated 90%) 2024-04-03T18:32:48.8340128Z adding: test/test-reports/python-pytest/distributed.test_device_mesh/distributed.test_device_mesh-3019debece1fedb8.xml (deflated 94%) 2024-04-03T18:32:48.8342457Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_overlap/distributed._composable.fsdp.test_fully_shard_overlap-67144b26de76f7e0.xml (deflated 40%) 2024-04-03T18:32:48.8344854Z adding: test/test-reports/python-pytest/distributed._tools.test_memory_tracker/distributed._tools.test_memory_tracker-cb6de4ff85013b67.xml (deflated 63%) 2024-04-03T18:32:48.8347030Z adding: test/test-reports/python-pytest/distributed._tensor.test_pointwise_ops/distributed._tensor.test_pointwise_ops-68c681a819d4c03a.xml (deflated 71%) 2024-04-03T18:32:48.8349308Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_style/distributed.tensor.parallel.test_tp_style-34f63408036562fd.xml (deflated 76%) 2024-04-03T18:32:48.8351749Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dtensor_checkpoint/distributed.checkpoint.test_dtensor_checkpoint-5bdfd59bc849d4b3.xml (deflated 41%) 2024-04-03T18:32:48.8354055Z adding: test/test-reports/python-pytest/distributed._tensor.test_op_strategy/distributed._tensor.test_op_strategy-3ec5f5bf53cb56ef.xml (deflated 82%) 2024-04-03T18:32:48.8356441Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_memory/distributed._composable.fsdp.test_fully_shard_memory-1eda814b33046f29.xml (deflated 39%) 2024-04-03T18:32:48.8359098Z adding: test/test-reports/python-pytest/distributed.checkpoint.e2e.test_pipeline/distributed.checkpoint.e2e.test_pipeline-c59bbced355cfa28.xml (deflated 52%) 2024-04-03T18:32:48.8361477Z adding: test/test-reports/python-pytest/distributed._tensor.test_embedding_ops/distributed._tensor.test_embedding_ops-f773919cc13163d1.xml (deflated 59%) 2024-04-03T18:32:48.8363818Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_hsdp_checkpoint/distributed.checkpoint.test_hsdp_checkpoint-c8d0416b7ab53b2e.xml (deflated 78%) 2024-04-03T18:32:48.8366249Z adding: test/test-reports/python-pytest/distributed.checkpoint.e2e.test_fine_tuning/distributed.checkpoint.e2e.test_fine_tuning-cec7c385cc4eae47.xml (deflated 51%) 2024-04-03T18:32:48.8368504Z adding: test/test-reports/python-pytest/distributed._tensor.test_matrix_ops/distributed._tensor.test_matrix_ops-9fc8ff6596879e21.xml (deflated 74%) 2024-04-03T18:32:48.8371257Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_mixed_precision/distributed._composable.fully_shard.test_fully_shard_mixed_precision-ea40f0a588ac86c6.xml (deflated 39%) 2024-04-03T18:32:48.8374143Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_compile/distributed._composable.fully_shard.test_fully_shard_compile-2af2e3ae4fae2fa9.xml (deflated 49%) 2024-04-03T18:32:48.8376744Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_model_state/distributed.checkpoint.test_fsdp_model_state-16347f5ebd5ba53b.xml (deflated 55%) 2024-04-03T18:32:48.8378949Z adding: test/test-reports/python-pytest/distributed.fsdp.test_shard_utils/distributed.fsdp.test_shard_utils-470459897e953010.xml (deflated 50%) 2024-04-03T18:32:48.8381338Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-bebf8816e49f8333.xml (deflated 81%) 2024-04-03T18:32:48.8383739Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_pure_fp16/distributed.fsdp.test_fsdp_pure_fp16-6605045f5143aa2e.xml (deflated 49%) 2024-04-03T18:32:48.8385938Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_optim_state/distributed.fsdp.test_fsdp_optim_state-852089b6d5890e66.xml (deflated 93%) 2024-04-03T18:32:48.8388193Z adding: test/test-reports/python-pytest/distributed.fsdp.test_checkpoint_wrapper/distributed.fsdp.test_checkpoint_wrapper-215266451754db71.xml (deflated 70%) 2024-04-03T18:32:48.8390406Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_grad_acc/distributed.fsdp.test_fsdp_grad_acc-ec7069b5f95395a7.xml (deflated 72%) 2024-04-03T18:32:48.8392696Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_sharded_grad_scaler/distributed.fsdp.test_fsdp_sharded_grad_scaler-662c418ab66fea54.xml (deflated 89%) 2024-04-03T18:32:48.8394991Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_state_dict/distributed.fsdp.test_fsdp_state_dict-f5648ec7056088bb.xml (deflated 95%) 2024-04-03T18:32:48.8397430Z adding: test/test-reports/python-pytest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks-3b1c814f14aba7bb.xml (deflated 74%) 2024-04-03T18:32:48.8399700Z adding: test/test-reports/python-pytest/distributed.fsdp.test_wrap/distributed.fsdp.test_wrap-553daefc0d5cc27a.xml (deflated 86%) 2024-04-03T18:32:48.8402010Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_file_system_checkpoint_cpu/distributed.checkpoint.test_file_system_checkpoint_cpu-93edb2022f893686.xml (deflated 78%) 2024-04-03T18:32:48.8404428Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm_hooks/distributed.fsdp.test_fsdp_comm_hooks-c90d6dbfd689b18a.xml (deflated 90%) 2024-04-03T18:32:48.8406559Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_meta/distributed.fsdp.test_fsdp_meta-94ceb51940f28da3.xml (deflated 88%) 2024-04-03T18:32:48.8408939Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_use_orig_params/distributed.fsdp.test_fsdp_use_orig_params-d3e579df7f8295d4.xml (deflated 85%) 2024-04-03T18:32:48.8411735Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_multiple_forward/distributed.fsdp.test_fsdp_multiple_forward-6a3a000f2c737cd1.xml (deflated 40%) 2024-04-03T18:32:48.8413970Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-dfc46dce2dacdfb7.xml (deflated 80%) 2024-04-03T18:32:48.8416110Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_misc/distributed.fsdp.test_fsdp_misc-29454fb36295e31a.xml (deflated 77%) 2024-04-03T18:32:48.8418336Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_microbatch/distributed.pipeline.sync.test_microbatch-b98f118c2dc21a40.xml (deflated 80%) 2024-04-03T18:32:48.8420748Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_checkpoint/distributed.pipeline.sync.test_checkpoint-3fffbe2b404fa366.xml (deflated 77%) 2024-04-03T18:32:48.8423168Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_portal/distributed.pipeline.sync.skip.test_portal-423ddef812619a3a.xml (deflated 80%) 2024-04-03T18:32:48.8425619Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_init/distributed._shard.sharded_tensor.ops.test_init-34ae9f0a22e5369c.xml (deflated 73%) 2024-04-03T18:32:48.8427944Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_copy/distributed.pipeline.sync.test_copy-4a076e4a9065aaf3.xml (deflated 69%) 2024-04-03T18:32:48.8430230Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_mixed_precision/distributed.fsdp.test_fsdp_mixed_precision-e3a00b6ee5d09bf3.xml (deflated 92%) 2024-04-03T18:32:48.8432719Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_binary_cmp/distributed._shard.sharded_tensor.ops.test_binary_cmp-6c9405212789ab7e.xml (deflated 78%) 2024-04-03T18:32:48.8435330Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_tracker/distributed.pipeline.sync.skip.test_tracker-7839f91a0f439fd4.xml (deflated 72%) 2024-04-03T18:32:48.8437751Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_leak/distributed.pipeline.sync.skip.test_leak-dadba56373571d53.xml (deflated 75%) 2024-04-03T18:32:48.8440207Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_stash_pop/distributed.pipeline.sync.skip.test_stash_pop-ebf315ef92656716.xml (deflated 77%) 2024-04-03T18:32:48.8442484Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-d017fcd507a880c0.xml (deflated 36%) 2024-04-03T18:32:48.8444517Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-dc47b73286746577.xml (deflated 35%) 2024-04-03T18:32:48.8446523Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-3c717c72d47be044.xml (deflated 36%) 2024-04-03T18:32:48.8448573Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-a983e2941c00b5e8.xml (deflated 35%) 2024-04-03T18:32:48.8450882Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-221b8a0c7e79032c.xml (deflated 36%) 2024-04-03T18:32:48.8452911Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-45b7913f00b6c587.xml (deflated 35%) 2024-04-03T18:32:48.8454925Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-30636d219c46d98c.xml (deflated 35%) 2024-04-03T18:32:48.8456949Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-19f629db5891140a.xml (deflated 35%) 2024-04-03T18:32:48.8459147Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-efaa130bbf5afa94.xml (deflated 36%) 2024-04-03T18:32:48.8461188Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-2d1947551804e807.xml (deflated 35%) 2024-04-03T18:32:48.8463294Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-5e25832e8b7a87d1.xml (deflated 36%) 2024-04-03T18:32:48.8465260Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-f86b6a1e3168473b.xml (deflated 38%) 2024-04-03T18:32:48.8467095Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-d469f22e734aa462.xml (deflated 40%) 2024-04-03T18:32:48.8468957Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-6d05f9a6475d6b1b.xml (deflated 37%) 2024-04-03T18:32:48.8470800Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-4757efbac9428ca2.xml (deflated 37%) 2024-04-03T18:32:48.8472658Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-06aeec0a17eaf50d.xml (deflated 44%) 2024-04-03T18:32:48.8474505Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-632ce248944e9606.xml (deflated 37%) 2024-04-03T18:32:48.8476335Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-6d9a128b968bc04e.xml (deflated 37%) 2024-04-03T18:32:48.8478151Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-b8e7a052970b529b.xml (deflated 37%) 2024-04-03T18:32:48.8480002Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-a3ad15c6dbc586dd.xml (deflated 38%) 2024-04-03T18:32:48.8481849Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-9b5acc198db225a1.xml (deflated 39%) 2024-04-03T18:32:48.8483664Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-d63db01111496cdd.xml (deflated 39%) 2024-04-03T18:32:48.8485490Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-0f52ef61089611ed.xml (deflated 38%) 2024-04-03T18:32:48.8487344Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-638859260c103751.xml (deflated 38%) 2024-04-03T18:32:48.8489346Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-86b5bcda2bc6cf0a.xml (deflated 38%) 2024-04-03T18:32:48.8491197Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-7c8a4276d11373d3.xml (deflated 38%) 2024-04-03T18:32:48.8493027Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-39d43cabc480f961.xml (deflated 37%) 2024-04-03T18:32:48.8494875Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-944ef32742d2184f.xml (deflated 38%) 2024-04-03T18:32:48.8496739Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-8186f0ed3bc526ef.xml (deflated 38%) 2024-04-03T18:32:48.8498551Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-a425b6321713c20e.xml (deflated 38%) 2024-04-03T18:32:48.8500401Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-e75370d913733158.xml (deflated 38%) 2024-04-03T18:32:48.8502235Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-9fdfe3575e56e5ca.xml (deflated 38%) 2024-04-03T18:32:48.8504085Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-d2994baa6afb57c5.xml (deflated 38%) 2024-04-03T18:32:48.8506080Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-a64bc07a92f54967.xml (deflated 38%) 2024-04-03T18:32:48.8507907Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-21a5b6b0e1f5ab17.xml (deflated 38%) 2024-04-03T18:32:48.8509867Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-9637f9e3e329b479.xml (deflated 38%) 2024-04-03T18:32:48.8511768Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-805987f2074348db.xml (deflated 38%) 2024-04-03T18:32:48.8513566Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-dcdf43440dae1671.xml (deflated 38%) 2024-04-03T18:32:48.8515402Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-af7ea5788e73364a.xml (deflated 45%) 2024-04-03T18:32:48.8517225Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-3b3bdc5a9dec104b.xml (deflated 38%) 2024-04-03T18:32:48.8519104Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-c5db66d242393e26.xml (deflated 38%) 2024-04-03T18:32:48.8520911Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-2d679e98be85a9b8.xml (deflated 44%) 2024-04-03T18:32:48.8522771Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-84c5af500d853b99.xml (deflated 42%) 2024-04-03T18:32:48.8524606Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-a4bb76a162babac9.xml (deflated 43%) 2024-04-03T18:32:48.8526427Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-a0d4fbec5332313b.xml (deflated 42%) 2024-04-03T18:32:48.8528260Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-5183a24f65836419.xml (deflated 38%) 2024-04-03T18:32:48.8530301Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-bbab5bd7361f462d.xml (deflated 38%) 2024-04-03T18:32:48.8532158Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-b44f8bef022808b8.xml (deflated 38%) 2024-04-03T18:32:48.8534029Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-9c4f80042dae5daa.xml (deflated 38%) 2024-04-03T18:32:48.8535868Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-05691d5c0980a570.xml (deflated 45%) 2024-04-03T18:32:48.8537709Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-f8c375313bc508be.xml (deflated 45%) 2024-04-03T18:32:48.8539526Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-349908b7d93971c1.xml (deflated 38%) 2024-04-03T18:32:48.8541352Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-84c62e7312b36bc7.xml (deflated 38%) 2024-04-03T18:32:48.8543179Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-44dda517a7b5dee5.xml (deflated 38%) 2024-04-03T18:32:48.8545018Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-1ee9caca4c63c7d3.xml (deflated 38%) 2024-04-03T18:32:48.8546869Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-4ea0c932788a8196.xml (deflated 47%) 2024-04-03T18:32:48.8548679Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-e4495c2b448ae3a4.xml (deflated 47%) 2024-04-03T18:32:48.8550512Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-0505aa5fae489357.xml (deflated 38%) 2024-04-03T18:32:48.8552333Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-e0d2877723840990.xml (deflated 38%) 2024-04-03T18:32:48.8554320Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-d0bd0126f504c39e.xml (deflated 38%) 2024-04-03T18:32:48.8556131Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-907b56fac61d1104.xml (deflated 49%) 2024-04-03T18:32:48.8558064Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-9d6869e2e55f2431.xml (deflated 38%) 2024-04-03T18:32:48.8559904Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-2589dff11dcdc8e5.xml (deflated 38%) 2024-04-03T18:32:48.8561744Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-72f5c44991dbdfc6.xml (deflated 38%) 2024-04-03T18:32:48.8563545Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-d7a493112ac44a86.xml (deflated 37%) 2024-04-03T18:32:48.8565400Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-d4434b8b37ce0143.xml (deflated 42%) 2024-04-03T18:32:48.8567226Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-f527eefef7e25240.xml (deflated 38%) 2024-04-03T18:32:48.8569028Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-76f76c22814f4c98.xml (deflated 42%) 2024-04-03T18:32:48.8571063Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-9591abada8e0bfd5.xml (deflated 37%) 2024-04-03T18:32:48.8572897Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-88292c40a02fea3b.xml (deflated 37%) 2024-04-03T18:32:48.8574726Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-d511300d733e1fe6.xml (deflated 37%) 2024-04-03T18:32:48.8576532Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-4a2c77bbf4edb274.xml (deflated 37%) 2024-04-03T18:32:48.8578369Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-85735a81d9810965.xml (deflated 37%) 2024-04-03T18:32:48.8580214Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-0cd6ff3eea8c4f24.xml (deflated 37%) 2024-04-03T18:32:48.8582053Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-455df5f8f05fe893.xml (deflated 38%) 2024-04-03T18:32:48.8583854Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-612a9b49f22ceb86.xml (deflated 38%) 2024-04-03T18:32:48.8585697Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-556e642960ca82a9.xml (deflated 37%) 2024-04-03T18:32:48.8587531Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-5f98bc2e14790851.xml (deflated 37%) 2024-04-03T18:32:48.8589389Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-bc3b6c9b81827dd8.xml (deflated 40%) 2024-04-03T18:32:48.8591182Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-a44d30d53d3e9ba0.xml (deflated 42%) 2024-04-03T18:32:48.8593023Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-04815796fd1263f1.xml (deflated 37%) 2024-04-03T18:32:48.8594856Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-36cc085a48ae366d.xml (deflated 37%) 2024-04-03T18:32:48.8597191Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_inspect_skip_layout/distributed.pipeline.sync.skip.test_inspect_skip_layout-ae6202d819c7f793.xml (deflated 74%) 2024-04-03T18:32:48.8599898Z adding: test/test-reports/python-pytest/distributed._shard.sharded_optim.test_sharded_optim/distributed._shard.sharded_optim.test_sharded_optim-0dc55d44bb916166.xml (deflated 66%) 2024-04-03T18:32:48.8602244Z adding: test/test-reports/python-pytest/distributed.test_launcher/distributed.test_launcher-d5b1e363de7e17c4.xml (deflated 39%) 2024-04-03T18:32:48.8604456Z adding: test/test-reports/python-pytest/distributed.optim.test_zero_redundancy_optimizer/distributed.optim.test_zero_redundancy_optimizer-0c8771f1bef087f5.xml (deflated 91%) 2024-04-03T18:32:48.8606729Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8306987a6034599c.xml (deflated 36%) 2024-04-03T18:32:48.8608612Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-12b27a2ade8ae866.xml (deflated 43%) 2024-04-03T18:32:48.8610683Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6d2d48620e42c4a5.xml (deflated 34%) 2024-04-03T18:32:48.8612538Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-663648ac07f603e2.xml (deflated 34%) 2024-04-03T18:32:48.8614448Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-37e2b13971bbe943.xml (deflated 34%) 2024-04-03T18:32:48.8616296Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f2c98c3a2af4a495.xml (deflated 34%) 2024-04-03T18:32:48.8618194Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c0dd81cbfd80dd73.xml (deflated 34%) 2024-04-03T18:32:48.8620091Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-39cff48a7631a3f6.xml (deflated 34%) 2024-04-03T18:32:48.8621980Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-525fd4f8c2bb6ab9.xml (deflated 34%) 2024-04-03T18:32:48.8623845Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-07df84b4aff5aded.xml (deflated 34%) 2024-04-03T18:32:48.8625737Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-cf02fa21af10e4b5.xml (deflated 35%) 2024-04-03T18:32:48.8627619Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b17d3c8cff5e58a7.xml (deflated 33%) 2024-04-03T18:32:48.8629509Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5db366bba7216459.xml (deflated 33%) 2024-04-03T18:32:48.8631374Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a91df81b8e01acc9.xml (deflated 34%) 2024-04-03T18:32:48.8633257Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-05408ec239ac4dc8.xml (deflated 33%) 2024-04-03T18:32:48.8635183Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0bd3ded661018fa5.xml (deflated 33%) 2024-04-03T18:32:48.8637048Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c60e8c98cea6a639.xml (deflated 56%) 2024-04-03T18:32:48.8638946Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fb90946addec58c1.xml (deflated 33%) 2024-04-03T18:32:48.8640813Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4fa21634fe596211.xml (deflated 34%) 2024-04-03T18:32:48.8642691Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-34f0d6f95c425947.xml (deflated 34%) 2024-04-03T18:32:48.8644542Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3fbcb136f09e3b9a.xml (deflated 34%) 2024-04-03T18:32:48.8646434Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2f57892728dc8925.xml (deflated 35%) 2024-04-03T18:32:48.8648303Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e35f89d16c062baa.xml (deflated 34%) 2024-04-03T18:32:48.8650510Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f1cdb95256b5b8eb.xml (deflated 36%) 2024-04-03T18:32:48.8652388Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8cde82a0e6a3401e.xml (deflated 36%) 2024-04-03T18:32:48.8654458Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9d600c9b7c9d7189.xml (deflated 36%) 2024-04-03T18:32:48.8656373Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6560e56c0017a6c3.xml (deflated 36%) 2024-04-03T18:32:48.8658252Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d66a9c7719712277.xml (deflated 37%) 2024-04-03T18:32:48.8660098Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-33d456ee4c9b61fd.xml (deflated 36%) 2024-04-03T18:32:48.8661948Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-851928c00a279ae6.xml (deflated 36%) 2024-04-03T18:32:48.8663852Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6a436040aeee8dfc.xml (deflated 36%) 2024-04-03T18:32:48.8665732Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e59bafc83d2065dc.xml (deflated 36%) 2024-04-03T18:32:48.8667565Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-29cc50b1727e6926.xml (deflated 37%) 2024-04-03T18:32:48.8669452Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-259388f124e75638.xml (deflated 37%) 2024-04-03T18:32:48.8671322Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8847d88a812db5f5.xml (deflated 36%) 2024-04-03T18:32:48.8673197Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-38ddb44e8e482587.xml (deflated 43%) 2024-04-03T18:32:48.8675038Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-578c4a4b46a2959b.xml (deflated 36%) 2024-04-03T18:32:48.8676923Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-92c995cbdecfcf43.xml (deflated 38%) 2024-04-03T18:32:48.8678823Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d5b1d71fda4640ef.xml (deflated 36%) 2024-04-03T18:32:48.8680677Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1c359f920364efbd.xml (deflated 36%) 2024-04-03T18:32:48.8682543Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9f9d0812a9ead1ec.xml (deflated 36%) 2024-04-03T18:32:48.8684632Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a8f2ce1971012929.xml (deflated 36%) 2024-04-03T18:32:48.8686518Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-36e7a4991179be98.xml (deflated 35%) 2024-04-03T18:32:48.8688367Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6b211a245aec4230.xml (deflated 37%) 2024-04-03T18:32:48.8690464Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-aae8550fd7237ee0.xml (deflated 36%) 2024-04-03T18:32:48.8692343Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6c45c24f4306b669.xml (deflated 44%) 2024-04-03T18:32:48.8694221Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b8ed21de0b1870c0.xml (deflated 37%) 2024-04-03T18:32:48.8696077Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-86133064ce379a4d.xml (deflated 34%) 2024-04-03T18:32:48.8697944Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9da75e20519bd0dd.xml (deflated 43%) 2024-04-03T18:32:48.8700012Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-87daad823d06bd46.xml (deflated 35%) 2024-04-03T18:32:48.8701870Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f5c768d1a0f43a57.xml (deflated 36%) 2024-04-03T18:32:48.8703862Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-da24fcf8804d7d61.xml (deflated 36%) 2024-04-03T18:32:48.8705758Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8d18f324b49195e8.xml (deflated 45%) 2024-04-03T18:32:48.8707627Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-87eb277a833b1273.xml (deflated 49%) 2024-04-03T18:32:48.8709500Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ee0d7393a23b85cd.xml (deflated 44%) 2024-04-03T18:32:48.8711377Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9de7b7d84958b089.xml (deflated 36%) 2024-04-03T18:32:48.8713239Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2eed7f46b7b6f89f.xml (deflated 35%) 2024-04-03T18:32:48.8715140Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-98f3ba441534d7d0.xml (deflated 35%) 2024-04-03T18:32:48.8717005Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-481570997ce31947.xml (deflated 43%) 2024-04-03T18:32:48.8718861Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0eb73eb3979ad950.xml (deflated 36%) 2024-04-03T18:32:48.8720735Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2c5e1d0f0974abdf.xml (deflated 34%) 2024-04-03T18:32:48.8722611Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8e374b6f6b44cd48.xml (deflated 35%) 2024-04-03T18:32:48.8724501Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-faddb072eda01c71.xml (deflated 35%) 2024-04-03T18:32:48.8726385Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-eeece823ccff6a50.xml (deflated 36%) 2024-04-03T18:32:48.8728257Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d1ad36b5a7996358.xml (deflated 35%) 2024-04-03T18:32:48.8730327Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-810a42663cca9c9b.xml (deflated 33%) 2024-04-03T18:32:48.8732181Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-15446c1fe6848402.xml (deflated 35%) 2024-04-03T18:32:48.8734064Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d60afb7b1e0ff86d.xml (deflated 43%) 2024-04-03T18:32:48.8735951Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3973f751a69cc672.xml (deflated 43%) 2024-04-03T18:32:48.8737826Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8c923f405cd04602.xml (deflated 43%) 2024-04-03T18:32:48.8739680Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4588b787dfe5100a.xml (deflated 43%) 2024-04-03T18:32:48.8741844Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-bedec0abdf9f254e.xml (deflated 34%) 2024-04-03T18:32:48.8743728Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-38e9161d037f0671.xml (deflated 35%) 2024-04-03T18:32:48.8745590Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d57317ea2a39dc55.xml (deflated 34%) 2024-04-03T18:32:48.8747598Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-21f777d346162a7c.xml (deflated 34%) 2024-04-03T18:32:48.8749472Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-15afd90839b561d5.xml (deflated 34%) 2024-04-03T18:32:48.8751457Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-adda6a0c94237ffe.xml (deflated 34%) 2024-04-03T18:32:48.8753352Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-42171ccfa7159b06.xml (deflated 34%) 2024-04-03T18:32:48.8755200Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6661a6a199ea7d6f.xml (deflated 34%) 2024-04-03T18:32:48.8757060Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-27207605017aa241.xml (deflated 34%) 2024-04-03T18:32:48.8758932Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8f8e7ab4b2b51413.xml (deflated 34%) 2024-04-03T18:32:48.8760797Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-96461760136185f5.xml (deflated 35%) 2024-04-03T18:32:48.8762645Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-be66746baead9dc9.xml (deflated 37%) 2024-04-03T18:32:48.8764340Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-b0ee8f362d1378d5.xml (deflated 60%) 2024-04-03T18:32:48.8765861Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-fce664ef67b6bc2e.xml (deflated 60%) 2024-04-03T18:32:48.8767376Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-bd53757de6d12bb2.xml (deflated 69%) 2024-04-03T18:32:48.8768861Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-640ef59b3f9998ac.xml (deflated 78%) 2024-04-03T18:32:48.8770589Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-baf903786c4b47bd.xml (deflated 74%) 2024-04-03T18:32:48.8772090Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-7f68d0f547b77d93.xml (deflated 76%) 2024-04-03T18:32:48.8773568Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-1fa2a2392c591fc5.xml (deflated 64%) 2024-04-03T18:32:48.8775368Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e160c315b6a0c613.xml (deflated 38%) 2024-04-03T18:32:48.8777409Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7dc9c65c896e6252.xml (deflated 38%) 2024-04-03T18:32:48.8779445Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e4ba7695589c3ce.xml (deflated 38%) 2024-04-03T18:32:48.8781468Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-afcfea6c21c4001a.xml (deflated 38%) 2024-04-03T18:32:48.8783568Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6aceb3e3cba1d1f8.xml (deflated 42%) 2024-04-03T18:32:48.8785568Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e66d45ebe7817fb2.xml (deflated 44%) 2024-04-03T18:32:48.8787618Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cde9c2b8b4615ac3.xml (deflated 44%) 2024-04-03T18:32:48.8789630Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8fcd8260ae15276f.xml (deflated 44%) 2024-04-03T18:32:48.8791654Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b8e7356368162f0.xml (deflated 42%) 2024-04-03T18:32:48.8793696Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-93cb25dbc6f22bf7.xml (deflated 43%) 2024-04-03T18:32:48.8795877Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e37f8dee8cf30c5f.xml (deflated 43%) 2024-04-03T18:32:48.8797886Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-880932b1db95e991.xml (deflated 43%) 2024-04-03T18:32:48.8800028Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f2287ef80e4135bf.xml (deflated 43%) 2024-04-03T18:32:48.8802080Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-777264b8f8b0e327.xml (deflated 43%) 2024-04-03T18:32:48.8804106Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-495ea1b54c16507d.xml (deflated 38%) 2024-04-03T18:32:48.8806119Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8ff139aa134db3a3.xml (deflated 44%) 2024-04-03T18:32:48.8808177Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e574e02fae2a82dd.xml (deflated 38%) 2024-04-03T18:32:48.8810397Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-08e9ac5c3088297b.xml (deflated 38%) 2024-04-03T18:32:48.8812432Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-87df4cecc1d71962.xml (deflated 44%) 2024-04-03T18:32:48.8814436Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ba85406a684a0917.xml (deflated 37%) 2024-04-03T18:32:48.8816476Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a6795aebbfdaf628.xml (deflated 43%) 2024-04-03T18:32:48.8818511Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e678a2c39ab9619e.xml (deflated 37%) 2024-04-03T18:32:48.8820555Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-85d754a9002e80ae.xml (deflated 38%) 2024-04-03T18:32:48.8822569Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-197d645f9843f26f.xml (deflated 38%) 2024-04-03T18:32:48.8824609Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c62e12c6517cc7b.xml (deflated 38%) 2024-04-03T18:32:48.8826635Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ca5c99034536163.xml (deflated 37%) 2024-04-03T18:32:48.8828664Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ef84ad0be4a996ca.xml (deflated 43%) 2024-04-03T18:32:48.8830679Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b813aa5dd3752c84.xml (deflated 43%) 2024-04-03T18:32:48.8832716Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d145010a42df1767.xml (deflated 37%) 2024-04-03T18:32:48.8834794Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-015e1ce32ba47894.xml (deflated 37%) 2024-04-03T18:32:48.8836832Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e15d7e8c33c2de9e.xml (deflated 42%) 2024-04-03T18:32:48.8838847Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b702b5bb5ce9e77b.xml (deflated 37%) 2024-04-03T18:32:48.8840888Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9ffff2adae2495f.xml (deflated 43%) 2024-04-03T18:32:48.8842917Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf80e8b6474ac289.xml (deflated 37%) 2024-04-03T18:32:48.8845115Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dbe72a1a2b73778a.xml (deflated 43%) 2024-04-03T18:32:48.8847118Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c21b41575a2d3951.xml (deflated 44%) 2024-04-03T18:32:48.8849436Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f6154870c1363806.xml (deflated 45%) 2024-04-03T18:32:48.8851542Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-352036763157f555.xml (deflated 45%) 2024-04-03T18:32:48.8853571Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ebd2dff8c5eff653.xml (deflated 45%) 2024-04-03T18:32:48.8855597Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89e69230dd8dd268.xml (deflated 45%) 2024-04-03T18:32:48.8857645Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bb57951e4fd08d2f.xml (deflated 45%) 2024-04-03T18:32:48.8859694Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c6cac087d2fdfc5.xml (deflated 37%) 2024-04-03T18:32:48.8861734Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f8d466ba2c31750e.xml (deflated 38%) 2024-04-03T18:32:48.8863749Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ad65e40be87dd6ad.xml (deflated 37%) 2024-04-03T18:32:48.8865777Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-44f28ca3c29f15e3.xml (deflated 38%) 2024-04-03T18:32:48.8867827Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bc19a7acad5b97c1.xml (deflated 42%) 2024-04-03T18:32:48.8869840Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e48e963cd14dd46.xml (deflated 44%) 2024-04-03T18:32:48.8871869Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2fc25b7647928274.xml (deflated 44%) 2024-04-03T18:32:48.8873888Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5d003a214010c31a.xml (deflated 44%) 2024-04-03T18:32:48.8875907Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70a634a440537e55.xml (deflated 42%) 2024-04-03T18:32:48.8877909Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89fe2f9968920ff1.xml (deflated 42%) 2024-04-03T18:32:48.8879947Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c6bbbd9bad28493.xml (deflated 43%) 2024-04-03T18:32:48.8881975Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-06e390aa935a50a6.xml (deflated 43%) 2024-04-03T18:32:48.8884009Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ebd52ec19016747c.xml (deflated 43%) 2024-04-03T18:32:48.8886010Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-48465ae7317d5266.xml (deflated 43%) 2024-04-03T18:32:48.8888037Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9ca257d5451b1fe9.xml (deflated 38%) 2024-04-03T18:32:48.8890265Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e58f0fa1c79d3669.xml (deflated 44%) 2024-04-03T18:32:48.8892452Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c8e159e6f41b60d.xml (deflated 37%) 2024-04-03T18:32:48.8894463Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3e524b1a3f82da2.xml (deflated 38%) 2024-04-03T18:32:48.8896609Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2aba96f05bfbbe00.xml (deflated 44%) 2024-04-03T18:32:48.8898668Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4a71f05acec332cd.xml (deflated 37%) 2024-04-03T18:32:48.8900757Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-59406c7134af47e6.xml (deflated 43%) 2024-04-03T18:32:48.8902754Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-35025a48c1c45539.xml (deflated 37%) 2024-04-03T18:32:48.8904787Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34c462e5dbdcffac.xml (deflated 37%) 2024-04-03T18:32:48.8906823Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b8a9cadb1aa9dd51.xml (deflated 37%) 2024-04-03T18:32:48.8908870Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d18dabcc6ff7e700.xml (deflated 37%) 2024-04-03T18:32:48.8910885Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9844a1b5f096258a.xml (deflated 37%) 2024-04-03T18:32:48.8912909Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-034ca21c59ea1b0d.xml (deflated 43%) 2024-04-03T18:32:48.8914938Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-497b2c3c71ad991d.xml (deflated 43%) 2024-04-03T18:32:48.8916963Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-05c3750697963475.xml (deflated 37%) 2024-04-03T18:32:48.8918968Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aad41f04597a5660.xml (deflated 37%) 2024-04-03T18:32:48.8920994Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-472c140185b31e7f.xml (deflated 42%) 2024-04-03T18:32:48.8923023Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-60ade67310bfd14f.xml (deflated 38%) 2024-04-03T18:32:48.8925056Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9c46b56fabec22f4.xml (deflated 43%) 2024-04-03T18:32:48.8927079Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b7efabd8eae19dc1.xml (deflated 37%) 2024-04-03T18:32:48.8929530Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-37bd1f7d19857a27.xml (deflated 43%) 2024-04-03T18:32:48.8931737Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a6b1bc6a4175f1d.xml (deflated 44%) 2024-04-03T18:32:48.8933990Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7c1729269e92fb14.xml (deflated 45%) 2024-04-03T18:32:48.8936269Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bdd474623fe7c31b.xml (deflated 45%) 2024-04-03T18:32:48.8938436Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b91ca7b83d3a22b3.xml (deflated 45%) 2024-04-03T18:32:48.8940596Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-48dada3fa915c767.xml (deflated 45%) 2024-04-03T18:32:48.8942911Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a7ea77ba45f4968.xml (deflated 45%) 2024-04-03T18:32:48.8945070Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f6edff6e6433ee7b.xml (deflated 37%) 2024-04-03T18:32:48.8947332Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f7989e8eb85bdd38.xml (deflated 37%) 2024-04-03T18:32:48.8949521Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dd2c9db1d5660838.xml (deflated 39%) 2024-04-03T18:32:48.8951673Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1483a8bf634936a7.xml (deflated 43%) 2024-04-03T18:32:48.8953833Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e0edea8dc81b8f19.xml (deflated 37%) 2024-04-03T18:32:48.8955989Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ab3b3c8417d35852.xml (deflated 37%) 2024-04-03T18:32:48.8958144Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7f4414cab7317480.xml (deflated 44%) 2024-04-03T18:32:48.8960311Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4a8560e754a5e68.xml (deflated 44%) 2024-04-03T18:32:48.8962468Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-693a13fa34873839.xml (deflated 44%) 2024-04-03T18:32:48.8964605Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6c23360987255af2.xml (deflated 44%) 2024-04-03T18:32:48.8966770Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a8edc6284895fa2.xml (deflated 43%) 2024-04-03T18:32:48.8968936Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-07063e5a7e5b7779.xml (deflated 43%) 2024-04-03T18:32:48.8971289Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ac31297e196d9d2a.xml (deflated 38%) 2024-04-03T18:32:48.8973439Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b180748585019274.xml (deflated 43%) 2024-04-03T18:32:48.8975606Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1f27d4386685b80c.xml (deflated 44%) 2024-04-03T18:32:48.8977764Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f96587e14e2d8c10.xml (deflated 37%) 2024-04-03T18:32:48.8979925Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b5f0791c6dcbbb10.xml (deflated 45%) 2024-04-03T18:32:48.8982086Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-909599caf4fb4a16.xml (deflated 41%) 2024-04-03T18:32:48.8984247Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f61cc1b7f694e1a7.xml (deflated 38%) 2024-04-03T18:32:48.8986426Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-29338ef12fefb1a1.xml (deflated 37%) 2024-04-03T18:32:48.8988601Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6eae2488c6287483.xml (deflated 57%) 2024-04-03T18:32:48.8990748Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6a169bbffb1584fd.xml (deflated 38%) 2024-04-03T18:32:48.8992916Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2fc0918b475cf143.xml (deflated 37%) 2024-04-03T18:32:48.8995230Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5b32d901ffba666f.xml (deflated 37%) 2024-04-03T18:32:48.8997362Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca3344507e32932d.xml (deflated 43%) 2024-04-03T18:32:48.8999649Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-93506b9018f60d37.xml (deflated 44%) 2024-04-03T18:32:48.9001847Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-db6ee91f7c238ada.xml (deflated 38%) 2024-04-03T18:32:48.9004012Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-547a0bd724e103d2.xml (deflated 37%) 2024-04-03T18:32:48.9006155Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d35874790556298a.xml (deflated 37%) 2024-04-03T18:32:48.9008321Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-50f6c072c322ad7d.xml (deflated 37%) 2024-04-03T18:32:48.9010690Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fcda2ea7296aba80.xml (deflated 37%) 2024-04-03T18:32:48.9012870Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e46a5fd5056da68.xml (deflated 43%) 2024-04-03T18:32:48.9015026Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fd4b720d3020e3d3.xml (deflated 37%) 2024-04-03T18:32:48.9017187Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c05afecc2765a23.xml (deflated 44%) 2024-04-03T18:32:48.9019445Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd6789ba3ea4acb9.xml (deflated 38%) 2024-04-03T18:32:48.9021637Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ad2cdf17114a44bc.xml (deflated 38%) 2024-04-03T18:32:48.9050418Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-29cb7e223e5c754d.xml (deflated 47%) 2024-04-03T18:32:48.9052504Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2e1d61b86f559ed7.xml (deflated 43%) 2024-04-03T18:32:48.9054559Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-60655027eca15f27.xml (deflated 43%) 2024-04-03T18:32:48.9056593Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8eed5c0811215304.xml (deflated 43%) 2024-04-03T18:32:48.9058638Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0a725242564daaa9.xml (deflated 43%) 2024-04-03T18:32:48.9060663Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-feb3e3da9c7e215a.xml (deflated 43%) 2024-04-03T18:32:48.9062711Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bc3605ba97abcb88.xml (deflated 46%) 2024-04-03T18:32:48.9064756Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3c12cd37416d6d8d.xml (deflated 45%) 2024-04-03T18:32:48.9066790Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c95f6751480c8c33.xml (deflated 37%) 2024-04-03T18:32:48.9068804Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e8e227fb960e32b.xml (deflated 45%) 2024-04-03T18:32:48.9071059Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15320b524fb39d6a.xml (deflated 38%) 2024-04-03T18:32:48.9073100Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-57d3c1274cc77b8f.xml (deflated 37%) 2024-04-03T18:32:48.9075260Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7b613675c64983b2.xml (deflated 37%) 2024-04-03T18:32:48.9077301Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b14b7b01388919b5.xml (deflated 37%) 2024-04-03T18:32:48.9079337Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-df7a331df638f852.xml (deflated 38%) 2024-04-03T18:32:48.9081380Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3835cc3238cafa71.xml (deflated 43%) 2024-04-03T18:32:48.9083426Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-04427523d58ac3a0.xml (deflated 38%) 2024-04-03T18:32:48.9085436Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-64a3d81b539a4b52.xml (deflated 37%) 2024-04-03T18:32:48.9087487Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f2df0e9b0ed04ed.xml (deflated 44%) 2024-04-03T18:32:48.9089723Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd328868a0168647.xml (deflated 44%) 2024-04-03T18:32:48.9091769Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13f610cbd7beff7e.xml (deflated 44%) 2024-04-03T18:32:48.9093783Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d307bffa6ea19f97.xml (deflated 44%) 2024-04-03T18:32:48.9095834Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2720aec5fb5f57db.xml (deflated 43%) 2024-04-03T18:32:48.9097868Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-251d9cbc2af614d5.xml (deflated 42%) 2024-04-03T18:32:48.9099895Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6116cac1c7845093.xml (deflated 38%) 2024-04-03T18:32:48.9101911Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f5ca764e6244677.xml (deflated 43%) 2024-04-03T18:32:48.9103943Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fb7c3b4e8bbf1346.xml (deflated 44%) 2024-04-03T18:32:48.9105969Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-475796879b642dbd.xml (deflated 38%) 2024-04-03T18:32:48.9108008Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4703f2db64cdeae9.xml (deflated 44%) 2024-04-03T18:32:48.9110028Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ffd62d9cc26c32f1.xml (deflated 41%) 2024-04-03T18:32:48.9112068Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0a0a61639dfb949b.xml (deflated 38%) 2024-04-03T18:32:48.9114103Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ed8919d7d78f4fd.xml (deflated 37%) 2024-04-03T18:32:48.9116139Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fe14946079c6ed3f.xml (deflated 57%) 2024-04-03T18:32:48.9118147Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5590bb1c2130d782.xml (deflated 37%) 2024-04-03T18:32:48.9120338Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-685bce50c619c620.xml (deflated 37%) 2024-04-03T18:32:48.9122374Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3406ff9f355cc1a2.xml (deflated 38%) 2024-04-03T18:32:48.9124538Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e1e6e5446280ecc0.xml (deflated 43%) 2024-04-03T18:32:48.9126586Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ace80b35e4fd1960.xml (deflated 44%) 2024-04-03T18:32:48.9128622Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a22314fc508d77b6.xml (deflated 38%) 2024-04-03T18:32:48.9130844Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-092ce45e30dc743b.xml (deflated 37%) 2024-04-03T18:32:48.9132889Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c23b7d7bd987e99e.xml (deflated 38%) 2024-04-03T18:32:48.9134947Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e6bf5ab0d79af79.xml (deflated 37%) 2024-04-03T18:32:48.9136992Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c7bdf1c212e62aed.xml (deflated 37%) 2024-04-03T18:32:48.9139033Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fd6a45cb75975b4b.xml (deflated 43%) 2024-04-03T18:32:48.9141062Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0ad9382e637c847d.xml (deflated 38%) 2024-04-03T18:32:48.9143062Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-14290654f35820bf.xml (deflated 44%) 2024-04-03T18:32:48.9145096Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3d12cf4bf099448a.xml (deflated 37%) 2024-04-03T18:32:48.9147133Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96c92ae0b66e9915.xml (deflated 37%) 2024-04-03T18:32:48.9149169Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a7a3304fa7fb12e7.xml (deflated 47%) 2024-04-03T18:32:48.9151191Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-37400ba3daab7724.xml (deflated 43%) 2024-04-03T18:32:48.9153228Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c9af959fc96ac8d.xml (deflated 43%) 2024-04-03T18:32:48.9155263Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e9c9719e0a2c130e.xml (deflated 43%) 2024-04-03T18:32:48.9157284Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-153984de0746e7a8.xml (deflated 43%) 2024-04-03T18:32:48.9159318Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f0fef47378f79a63.xml (deflated 43%) 2024-04-03T18:32:48.9161359Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0f92d8c2573d100d.xml (deflated 46%) 2024-04-03T18:32:48.9163402Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-97a9cd3d33e2ba19.xml (deflated 45%) 2024-04-03T18:32:48.9165405Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9482336fd4326d31.xml (deflated 37%) 2024-04-03T18:32:48.9167412Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-78cc4bacd87a80b1.xml (deflated 45%) 2024-04-03T18:32:48.9169800Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-47159d6ba1d7fe4f.xml (deflated 38%) 2024-04-03T18:32:48.9171881Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a74b670959fbeee.xml (deflated 37%) 2024-04-03T18:32:48.9174011Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a61d1e0352f60012.xml (deflated 37%) 2024-04-03T18:32:48.9176067Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-88039e933e32bbc1.xml (deflated 38%) 2024-04-03T18:32:48.9178079Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-12469f4aea8b50ea.xml (deflated 38%) 2024-04-03T18:32:48.9180088Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ea592a55de80fbab.xml (deflated 38%) 2024-04-03T18:32:48.9182088Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32d554983cee5380.xml (deflated 42%) 2024-04-03T18:32:48.9184102Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2aa7743764532b97.xml (deflated 37%) 2024-04-03T18:32:48.9186159Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4593606bcea15e3c.xml (deflated 37%) 2024-04-03T18:32:48.9188178Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-221ead1fa17d2bcb.xml (deflated 37%) 2024-04-03T18:32:48.9190273Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c6d58f752335f9e.xml (deflated 37%) 2024-04-03T18:32:48.9192285Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-18ce0161fe9e5876.xml (deflated 49%) 2024-04-03T18:32:48.9194307Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-63cc47914346b865.xml (deflated 49%) 2024-04-03T18:32:48.9196346Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-25bb4650e194eafc.xml (deflated 37%) 2024-04-03T18:32:48.9198373Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a7a5eeb2fedaec7.xml (deflated 38%) 2024-04-03T18:32:48.9200480Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c57f6dc9c467a09.xml (deflated 38%) 2024-04-03T18:32:48.9202502Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-717011d8f16b7841.xml (deflated 38%) 2024-04-03T18:32:48.9204536Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8896d7a6500f30fc.xml (deflated 44%) 2024-04-03T18:32:48.9206544Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-306b54e816057efe.xml (deflated 44%) 2024-04-03T18:32:48.9208582Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-789dbd3f0dd3e584.xml (deflated 37%) 2024-04-03T18:32:48.9210824Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a6afe92db2fe7925.xml (deflated 37%) 2024-04-03T18:32:48.9212862Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32fc940166073d9d.xml (deflated 42%) 2024-04-03T18:32:48.9214856Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-63f871b619796992.xml (deflated 37%) 2024-04-03T18:32:48.9217062Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5f4e54ff0dd1cec5.xml (deflated 37%) 2024-04-03T18:32:48.9219099Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c243fddc7fc5d876.xml (deflated 38%) 2024-04-03T18:32:48.9221241Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-84be35c04aa9056a.xml (deflated 37%) 2024-04-03T18:32:48.9223284Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4b8f0ef8e588d04.xml (deflated 38%) 2024-04-03T18:32:48.9225322Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0408212a7496bf5c.xml (deflated 37%) 2024-04-03T18:32:48.9227335Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89ec6a9d441f5d25.xml (deflated 37%) 2024-04-03T18:32:48.9229345Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-00011947d1572127.xml (deflated 43%) 2024-04-03T18:32:48.9231335Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2703d6428eb75aa.xml (deflated 37%) 2024-04-03T18:32:48.9233359Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3689576e4edafe7e.xml (deflated 37%) 2024-04-03T18:32:48.9235444Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8b864152c91b98d0.xml (deflated 37%) 2024-04-03T18:32:48.9237474Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bbb03fbdfc5c4fec.xml (deflated 42%) 2024-04-03T18:32:48.9239485Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5419012a468fc005.xml (deflated 38%) 2024-04-03T18:32:48.9241497Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d02636e646a00d83.xml (deflated 42%) 2024-04-03T18:32:48.9243507Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e454df17ecca1e72.xml (deflated 49%) 2024-04-03T18:32:48.9245543Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-09164b10d5a9688f.xml (deflated 37%) 2024-04-03T18:32:48.9247559Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bb09baa531bad871.xml (deflated 37%) 2024-04-03T18:32:48.9249779Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4505c6f60f5d4f5c.xml (deflated 37%) 2024-04-03T18:32:48.9251811Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-66f295ba3041fd94.xml (deflated 37%) 2024-04-03T18:32:48.9253831Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-290eda9c9ebe8451.xml (deflated 37%) 2024-04-03T18:32:48.9255843Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e3e9a4759bd8e06.xml (deflated 37%) 2024-04-03T18:32:48.9257881Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9baf5e79095831ef.xml (deflated 37%) 2024-04-03T18:32:48.9259918Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bde61bb9e1705ea0.xml (deflated 37%) 2024-04-03T18:32:48.9261956Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17f4dac2dbb4ed7d.xml (deflated 38%) 2024-04-03T18:32:48.9263968Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b6fa64f63c030d1.xml (deflated 39%) 2024-04-03T18:32:48.9266296Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4b4d54702df67d0e.xml (deflated 42%) 2024-04-03T18:32:48.9268457Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a33943b254185a84.xml (deflated 37%) 2024-04-03T18:32:48.9270712Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dff456f887dfbc09.xml (deflated 37%) 2024-04-03T18:32:48.9272890Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-79a7d4ad16c573d4.xml (deflated 37%) 2024-04-03T18:32:48.9275047Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39fca59a2ca61290.xml (deflated 37%) 2024-04-03T18:32:48.9277219Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd682d716dcf8744.xml (deflated 49%) 2024-04-03T18:32:48.9279400Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-faf64dea1559454b.xml (deflated 48%) 2024-04-03T18:32:48.9281559Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2356acb3087ab91b.xml (deflated 38%) 2024-04-03T18:32:48.9283729Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-463896da92f4949a.xml (deflated 38%) 2024-04-03T18:32:48.9285896Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-719d0e4db659eb93.xml (deflated 37%) 2024-04-03T18:32:48.9288049Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-db96b9a96b31d4b1.xml (deflated 38%) 2024-04-03T18:32:48.9290418Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-63629631d4d4bbe3.xml (deflated 44%) 2024-04-03T18:32:48.9292604Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e79c27311343442.xml (deflated 44%) 2024-04-03T18:32:48.9294764Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6cb9eef87a89e109.xml (deflated 37%) 2024-04-03T18:32:48.9296924Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f9bc12c3a8077a3.xml (deflated 37%) 2024-04-03T18:32:48.9299079Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fa8f7f44c6c9ccd7.xml (deflated 42%) 2024-04-03T18:32:48.9301244Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d52198c8a36410ed.xml (deflated 37%) 2024-04-03T18:32:48.9303468Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d1274d6b96e03dff.xml (deflated 37%) 2024-04-03T18:32:48.9305656Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-227f39d4f1e459e3.xml (deflated 38%) 2024-04-03T18:32:48.9307808Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6283c2dc589d8aaa.xml (deflated 37%) 2024-04-03T18:32:48.9309983Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-baba3bf9c809a600.xml (deflated 38%) 2024-04-03T18:32:48.9312171Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9c559dea3ef51b85.xml (deflated 37%) 2024-04-03T18:32:48.9314329Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-527c35f991c8e692.xml (deflated 38%) 2024-04-03T18:32:48.9316471Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-01e7d17d26513381.xml (deflated 43%) 2024-04-03T18:32:48.9318790Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fa1cfcd35c3b1f5a.xml (deflated 37%) 2024-04-03T18:32:48.9320962Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fe03a62f285ffa79.xml (deflated 37%) 2024-04-03T18:32:48.9323236Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-589e91737f484840.xml (deflated 37%) 2024-04-03T18:32:48.9325408Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b8519cf52d872277.xml (deflated 42%) 2024-04-03T18:32:48.9327558Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-52321ced108ff51e.xml (deflated 38%) 2024-04-03T18:32:48.9329896Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-005f71f48f9dc009.xml (deflated 42%) 2024-04-03T18:32:48.9332094Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1bb82b3a68b5da77.xml (deflated 49%) 2024-04-03T18:32:48.9334292Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6544c3eab47ececc.xml (deflated 37%) 2024-04-03T18:32:48.9336467Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8b15e8846d84137a.xml (deflated 37%) 2024-04-03T18:32:48.9338621Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f9f471bf918e3e10.xml (deflated 37%) 2024-04-03T18:32:48.9340791Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7c06cdf9650cc783.xml (deflated 37%) 2024-04-03T18:32:48.9342947Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-41ddbfd0dbefe7ae.xml (deflated 37%) 2024-04-03T18:32:48.9345120Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-828c6f704a82b009.xml (deflated 37%) 2024-04-03T18:32:48.9347282Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-09ce7ec7c3654920.xml (deflated 37%) 2024-04-03T18:32:48.9349455Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89f679ba089eef9e.xml (deflated 37%) 2024-04-03T18:32:48.9351600Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a111fdb89a552e4.xml (deflated 39%) 2024-04-03T18:32:48.9353772Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89a6a50f98e5b868.xml (deflated 37%) 2024-04-03T18:32:48.9355951Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c3cc6def10ed4b46.xml (deflated 43%) 2024-04-03T18:32:48.9358124Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1db5f69c9eb42ab1.xml (deflated 37%) 2024-04-03T18:32:48.9360299Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-582ef79ceb9508d9.xml (deflated 37%) 2024-04-03T18:32:48.9362471Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-92e6c41ee51592cc.xml (deflated 49%) 2024-04-03T18:32:48.9364641Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7249995d0d35fcb1.xml (deflated 48%) 2024-04-03T18:32:48.9366799Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24f7be8e3384609f.xml (deflated 37%) 2024-04-03T18:32:48.9369276Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-21a7bee3f703486a.xml (deflated 37%) 2024-04-03T18:32:48.9371497Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-12f3d8b10c0aabe9.xml (deflated 37%) 2024-04-03T18:32:48.9373812Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d53a8a519ef732a4.xml (deflated 38%) 2024-04-03T18:32:48.9376022Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6c434aca0aeaeede.xml (deflated 43%) 2024-04-03T18:32:48.9378173Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b694554b49d8ed97.xml (deflated 44%) 2024-04-03T18:32:48.9380341Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2273205b087fc4ad.xml (deflated 45%) 2024-04-03T18:32:48.9382511Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7b7e14d097bcc633.xml (deflated 50%) 2024-04-03T18:32:48.9384660Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b0f6c5485736f9cc.xml (deflated 37%) 2024-04-03T18:32:48.9386826Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ff70eb6de30b48bf.xml (deflated 42%) 2024-04-03T18:32:48.9389012Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ed1b1bf5ce033b79.xml (deflated 42%) 2024-04-03T18:32:48.9391190Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f97a4b8ccf477a54.xml (deflated 57%) 2024-04-03T18:32:48.9393346Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e678cb9550604fed.xml (deflated 38%) 2024-04-03T18:32:48.9395519Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-502afe23d14b06de.xml (deflated 37%) 2024-04-03T18:32:48.9397686Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30fdb1c426414108.xml (deflated 37%) 2024-04-03T18:32:48.9399861Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8c4e3d0c305c29fd.xml (deflated 37%) 2024-04-03T18:32:48.9402006Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c7e495ad11ac3521.xml (deflated 44%) 2024-04-03T18:32:48.9404170Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-09cde4152d42426f.xml (deflated 38%) 2024-04-03T18:32:48.9406333Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-80afe208c83bfe49.xml (deflated 37%) 2024-04-03T18:32:48.9408521Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bbbf487f7379d215.xml (deflated 38%) 2024-04-03T18:32:48.9410881Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9c9c7a44c7edc44b.xml (deflated 37%) 2024-04-03T18:32:48.9413060Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0105af9083840ee5.xml (deflated 37%) 2024-04-03T18:32:48.9415250Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ba2f34082ffff536.xml (deflated 37%) 2024-04-03T18:32:48.9417416Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-773f0424f7a093d5.xml (deflated 37%) 2024-04-03T18:32:48.9419560Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a8e8142bf70da5a8.xml (deflated 38%) 2024-04-03T18:32:48.9421883Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-10fa4f69e0859650.xml (deflated 38%) 2024-04-03T18:32:48.9424052Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e9e8bda46992bcb.xml (deflated 38%) 2024-04-03T18:32:48.9426333Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca2c8cd5c81ac68f.xml (deflated 47%) 2024-04-03T18:32:48.9428517Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2219a618d93fe776.xml (deflated 37%) 2024-04-03T18:32:48.9430693Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ae18a544b4e0a36f.xml (deflated 38%) 2024-04-03T18:32:48.9432859Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e2404878ff1ef053.xml (deflated 38%) 2024-04-03T18:32:48.9435077Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-027c10e2ed93d36e.xml (deflated 38%) 2024-04-03T18:32:48.9437237Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ec1cf24de24f1dcf.xml (deflated 37%) 2024-04-03T18:32:48.9439428Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ab3e1a1e6e92ee40.xml (deflated 38%) 2024-04-03T18:32:48.9441612Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-73a2a2de9305bf1b.xml (deflated 37%) 2024-04-03T18:32:48.9443787Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ed5ee09b9f93d56.xml (deflated 42%) 2024-04-03T18:32:48.9445960Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b723ad5a6263f33b.xml (deflated 38%) 2024-04-03T18:32:48.9448152Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-44fc989176234240.xml (deflated 38%) 2024-04-03T18:32:48.9450499Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-566c85a3165b8eae.xml (deflated 37%) 2024-04-03T18:32:48.9452681Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4d298fc983d94374.xml (deflated 37%) 2024-04-03T18:32:48.9454846Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b2e89b83e2cdefa.xml (deflated 37%) 2024-04-03T18:32:48.9457034Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f8f274ecc6fd5b1b.xml (deflated 39%) 2024-04-03T18:32:48.9459202Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32205f5013ee9e61.xml (deflated 37%) 2024-04-03T18:32:48.9461385Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-372fbc6a5560b88b.xml (deflated 43%) 2024-04-03T18:32:48.9463534Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-09488d756256edf9.xml (deflated 37%) 2024-04-03T18:32:48.9465712Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fdd9e9ff5dc4fc54.xml (deflated 37%) 2024-04-03T18:32:48.9467907Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8c3cb317713c02b9.xml (deflated 48%) 2024-04-03T18:32:48.9470085Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-63b36fa7359ee258.xml (deflated 48%) 2024-04-03T18:32:48.9472228Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-df77625b6d534c2c.xml (deflated 38%) 2024-04-03T18:32:48.9474564Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-19e3f25f37cf0e7d.xml (deflated 37%) 2024-04-03T18:32:48.9476740Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-caa208ac9acfe137.xml (deflated 37%) 2024-04-03T18:32:48.9478996Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ab9a92979a9303b.xml (deflated 38%) 2024-04-03T18:32:48.9481167Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5a5c455cfa475fa8.xml (deflated 43%) 2024-04-03T18:32:48.9483339Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ebc23dacb79e0a81.xml (deflated 44%) 2024-04-03T18:32:48.9485517Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ade0912dca77bb72.xml (deflated 45%) 2024-04-03T18:32:48.9487698Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2ef3827c47796612.xml (deflated 38%) 2024-04-03T18:32:48.9490011Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fec2c49ebd9f709e.xml (deflated 37%) 2024-04-03T18:32:48.9492204Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ea93354086b8fdf0.xml (deflated 42%) 2024-04-03T18:32:48.9494377Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-957e9ebe7ce4b464.xml (deflated 42%) 2024-04-03T18:32:48.9496556Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1ffe555c7ba5f99.xml (deflated 57%) 2024-04-03T18:32:48.9498699Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a44aff3314a6ed82.xml (deflated 37%) 2024-04-03T18:32:48.9500924Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c7ae8f58e1fc1994.xml (deflated 37%) 2024-04-03T18:32:48.9503101Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e6c6c2c4a2481bcf.xml (deflated 38%) 2024-04-03T18:32:48.9505284Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6560c7850d0ed61f.xml (deflated 37%) 2024-04-03T18:32:48.9507429Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2aa976df09c9c99.xml (deflated 44%) 2024-04-03T18:32:48.9509592Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e539306ab3123bfb.xml (deflated 38%) 2024-04-03T18:32:48.9511769Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-370bff9f7c0eecf3.xml (deflated 37%) 2024-04-03T18:32:48.9513950Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a8a614cc55e2eccf.xml (deflated 37%) 2024-04-03T18:32:48.9516112Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-973aa63de5ce4139.xml (deflated 38%) 2024-04-03T18:32:48.9518279Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0ed670fc774618f1.xml (deflated 37%) 2024-04-03T18:32:48.9520445Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f203e8d72375d6e8.xml (deflated 37%) 2024-04-03T18:32:48.9522612Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cddf6af0155c6e6f.xml (deflated 37%) 2024-04-03T18:32:48.9524918Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-46e3d1650a7d6cad.xml (deflated 37%) 2024-04-03T18:32:48.9527093Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-df4bf90afc0a390e.xml (deflated 37%) 2024-04-03T18:32:48.9529540Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8c49ee4e8ca55cbe.xml (deflated 37%) 2024-04-03T18:32:48.9531749Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-270b5949663c7a7f.xml (deflated 47%) 2024-04-03T18:32:48.9533931Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8c20c702fa9531c0.xml (deflated 37%) 2024-04-03T18:32:48.9536094Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-436c49b20979adcd.xml (deflated 37%) 2024-04-03T18:32:48.9538278Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b0f81a8a650e7111.xml (deflated 38%) 2024-04-03T18:32:48.9540448Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2b73740d19bcb1ec.xml (deflated 37%) 2024-04-03T18:32:48.9542608Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-180b115cb4af8b2d.xml (deflated 37%) 2024-04-03T18:32:48.9544781Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1c249e336608fef.xml (deflated 38%) 2024-04-03T18:32:48.9546962Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17bbe1b019bf42a4.xml (deflated 37%) 2024-04-03T18:32:48.9549120Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1148c5ed3074385.xml (deflated 42%) 2024-04-03T18:32:48.9551282Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-19caa146d34a41b2.xml (deflated 38%) 2024-04-03T18:32:48.9553455Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d8f5f84b7c7d05d9.xml (deflated 38%) 2024-04-03T18:32:48.9555625Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-247c71d55229b520.xml (deflated 37%) 2024-04-03T18:32:48.9557796Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ebd6bb440c2705fb.xml (deflated 37%) 2024-04-03T18:32:48.9559946Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1f65453d4a99a78.xml (deflated 38%) 2024-04-03T18:32:48.9562101Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0df3cea5edf60c8e.xml (deflated 38%) 2024-04-03T18:32:48.9564271Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af79b50d56b57fd6.xml (deflated 38%) 2024-04-03T18:32:48.9566436Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-172a8174e0ed68d3.xml (deflated 42%) 2024-04-03T18:32:48.9568574Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a254d5779a885ec.xml (deflated 44%) 2024-04-03T18:32:48.9570971Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0131da04d05ba4c6.xml (deflated 44%) 2024-04-03T18:32:48.9573136Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3b0d28a62c0f9970.xml (deflated 44%) 2024-04-03T18:32:48.9575286Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7faee32560b7fc35.xml (deflated 37%) 2024-04-03T18:32:48.9577583Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-18751466983a8249.xml (deflated 49%) 2024-04-03T18:32:48.9579734Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e1c8b14da95eca6.xml (deflated 49%) 2024-04-03T18:32:48.9581984Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e67f723b0428b1f9.xml (deflated 38%) 2024-04-03T18:32:48.9584152Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-16877d04d1644b03.xml (deflated 37%) 2024-04-03T18:32:48.9586278Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-755996fe517da53d.xml (deflated 38%) 2024-04-03T18:32:48.9588428Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e46359965c4dced9.xml (deflated 44%) 2024-04-03T18:32:48.9590600Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-04884ef83aa100ad.xml (deflated 44%) 2024-04-03T18:32:48.9592745Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-82ae537d5ed295ff.xml (deflated 44%) 2024-04-03T18:32:48.9594911Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-06067c0c972c7990.xml (deflated 38%) 2024-04-03T18:32:48.9597058Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5af931d81000ee2.xml (deflated 44%) 2024-04-03T18:32:48.9599212Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-83a9cf5769403bf6.xml (deflated 42%) 2024-04-03T18:32:48.9601332Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-05f68895252e11ba.xml (deflated 38%) 2024-04-03T18:32:48.9603472Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8ecec3431130a649.xml (deflated 37%) 2024-04-03T18:32:48.9605619Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8eb06b5a4a1884d1.xml (deflated 37%) 2024-04-03T18:32:48.9607783Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dbff435d50017055.xml (deflated 37%) 2024-04-03T18:32:48.9610096Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-18074590b94f81e7.xml (deflated 37%) 2024-04-03T18:32:48.9612267Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c55b635797d04bc7.xml (deflated 37%) 2024-04-03T18:32:48.9614424Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d7d6e34ae84db063.xml (deflated 38%) 2024-04-03T18:32:48.9616603Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d9c75f24419c0d11.xml (deflated 43%) 2024-04-03T18:32:48.9618735Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-20662c810adb6657.xml (deflated 37%) 2024-04-03T18:32:48.9620904Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bad4c68e5dd99f27.xml (deflated 37%) 2024-04-03T18:32:48.9623072Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-38b5125781a03b99.xml (deflated 37%) 2024-04-03T18:32:48.9625235Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e675eb25ffbca945.xml (deflated 42%) 2024-04-03T18:32:48.9627378Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5914ffd10d0b1188.xml (deflated 43%) 2024-04-03T18:32:48.9629684Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b3ae56a1121ea30.xml (deflated 42%) 2024-04-03T18:32:48.9631841Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-23659b446fff7d8b.xml (deflated 43%) 2024-04-03T18:32:48.9634151Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f93b63d3dae2b092.xml (deflated 44%) 2024-04-03T18:32:48.9636320Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a896e32a3a09dbcc.xml (deflated 45%) 2024-04-03T18:32:48.9638471Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-02a618c11f8b8578.xml (deflated 45%) 2024-04-03T18:32:48.9640621Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-58f4bf69d0238aa5.xml (deflated 38%) 2024-04-03T18:32:48.9642793Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-429c71b0bf9b9c17.xml (deflated 37%) 2024-04-03T18:32:48.9644916Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-053401526ca649ec.xml (deflated 37%) 2024-04-03T18:32:48.9647088Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c0ff05a5ef29b7f.xml (deflated 37%) 2024-04-03T18:32:48.9649413Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2ec2638720c37f4c.xml (deflated 38%) 2024-04-03T18:32:48.9651595Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d08d8f7fed50fa58.xml (deflated 38%) 2024-04-03T18:32:48.9653733Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a68c231e4b1a432f.xml (deflated 38%) 2024-04-03T18:32:48.9655911Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c35f77d0663114b1.xml (deflated 42%) 2024-04-03T18:32:48.9658054Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-21292af272fa3281.xml (deflated 44%) 2024-04-03T18:32:48.9660214Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af7164f02574604b.xml (deflated 44%) 2024-04-03T18:32:48.9662367Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a84cd5f088e4bab8.xml (deflated 44%) 2024-04-03T18:32:48.9664545Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3694ba92b48a68cd.xml (deflated 37%) 2024-04-03T18:32:48.9666695Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1dbdf107d89de4db.xml (deflated 49%) 2024-04-03T18:32:48.9668873Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b44ad00f9ffc7326.xml (deflated 48%) 2024-04-03T18:32:48.9671013Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3d1e35e5847b8fd8.xml (deflated 37%) 2024-04-03T18:32:48.9673188Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e85b14c66dab3d38.xml (deflated 38%) 2024-04-03T18:32:48.9675442Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-201300aa71c9d836.xml (deflated 38%) 2024-04-03T18:32:48.9677492Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7d6561ffe2ec72b6.xml (deflated 44%) 2024-04-03T18:32:48.9679634Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4fcfd2e9604dfc6e.xml (deflated 44%) 2024-04-03T18:32:48.9681942Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-84097dd9125c29c1.xml (deflated 44%) 2024-04-03T18:32:48.9684090Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-37acf402ac35449d.xml (deflated 37%) 2024-04-03T18:32:48.9686354Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e6e1475d68eb6ebd.xml (deflated 44%) 2024-04-03T18:32:48.9688510Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-41e8142b9208dcb0.xml (deflated 42%) 2024-04-03T18:32:48.9690860Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1dbfa7197bca261.xml (deflated 38%) 2024-04-03T18:32:48.9693029Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-de9d0d220d1618e9.xml (deflated 37%) 2024-04-03T18:32:48.9695199Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-85898c2d881be8f0.xml (deflated 38%) 2024-04-03T18:32:48.9697331Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f5d44e3c9b13884.xml (deflated 37%) 2024-04-03T18:32:48.9699496Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c9574457a6ac0410.xml (deflated 38%) 2024-04-03T18:32:48.9701659Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d7126352276db286.xml (deflated 37%) 2024-04-03T18:32:48.9703824Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e208fbd843c628f.xml (deflated 37%) 2024-04-03T18:32:48.9705952Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c5137a0beedac6e4.xml (deflated 43%) 2024-04-03T18:32:48.9708130Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15cb30e9ab1ce2ea.xml (deflated 37%) 2024-04-03T18:32:48.9710281Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-624944c1c7e88d47.xml (deflated 37%) 2024-04-03T18:32:48.9712431Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-76b4a27476b563ad.xml (deflated 37%) 2024-04-03T18:32:48.9714563Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f9d8a39108415878.xml (deflated 43%) 2024-04-03T18:32:48.9716715Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1efb246dc943de4.xml (deflated 43%) 2024-04-03T18:32:48.9718888Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6f80bf58811c64ec.xml (deflated 42%) 2024-04-03T18:32:48.9721043Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2b34adad45866c02.xml (deflated 43%) 2024-04-03T18:32:48.9723188Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-be2aa4a8edf8464c.xml (deflated 44%) 2024-04-03T18:32:48.9725343Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e9c4b4d93354ff58.xml (deflated 45%) 2024-04-03T18:32:48.9727501Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-003b53e312d29343.xml (deflated 45%) 2024-04-03T18:32:48.9729806Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9267b2d5f53f371a.xml (deflated 37%) 2024-04-03T18:32:48.9731957Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cada5d8de66f033b.xml (deflated 37%) 2024-04-03T18:32:48.9734296Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-de98cd1386233166.xml (deflated 37%) 2024-04-03T18:32:48.9736449Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-302cd661fa37ec90.xml (deflated 37%) 2024-04-03T18:32:48.9738710Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2975a42202923f1a.xml (deflated 37%) 2024-04-03T18:32:48.9740868Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-38051711c4c188bb.xml (deflated 39%) 2024-04-03T18:32:48.9743015Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ce6553028de36f7.xml (deflated 37%) 2024-04-03T18:32:48.9745191Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-28edd86d30a75273.xml (deflated 43%) 2024-04-03T18:32:48.9747352Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-595e7ae1e934e666.xml (deflated 37%) 2024-04-03T18:32:48.9749482Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-59a0132f6193ae38.xml (deflated 44%) 2024-04-03T18:32:48.9751639Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-906a921ae6f0acc8.xml (deflated 44%) 2024-04-03T18:32:48.9753855Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-889f5d3e285aa2c4.xml (deflated 44%) 2024-04-03T18:32:48.9756004Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-75534054259ad79b.xml (deflated 44%) 2024-04-03T18:32:48.9758176Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b25bccb6fdb20fff.xml (deflated 37%) 2024-04-03T18:32:48.9760325Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c888e8a994f08f59.xml (deflated 38%) 2024-04-03T18:32:48.9762488Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e8597205879a0006.xml (deflated 45%) 2024-04-03T18:32:48.9764644Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0ab272ebb9510a06.xml (deflated 43%) 2024-04-03T18:32:48.9766790Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a31ccc655c23936a.xml (deflated 44%) 2024-04-03T18:32:48.9768939Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e0a7f4782869fb6.xml (deflated 45%) 2024-04-03T18:32:48.9771287Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-814dcebcbca8179d.xml (deflated 45%) 2024-04-03T18:32:48.9773442Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ade611d4c3ae59f0.xml (deflated 41%) 2024-04-03T18:32:48.9775607Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-23765e64d9fa292c.xml (deflated 42%) 2024-04-03T18:32:48.9777741Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6e1017bc9190e3fb.xml (deflated 42%) 2024-04-03T18:32:48.9779890Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e59e8fb05d6ec5cc.xml (deflated 57%) 2024-04-03T18:32:48.9782042Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-169bac8d875dc3ba.xml (deflated 38%) 2024-04-03T18:32:48.9784355Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-61e35c63c4eb8a8e.xml (deflated 37%) 2024-04-03T18:32:48.9786493Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ef20e38d0e54b456.xml (deflated 37%) 2024-04-03T18:32:48.9788755Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-edf2c6eb27fb41ff.xml (deflated 37%) 2024-04-03T18:32:48.9790940Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e15f18113b07b424.xml (deflated 44%) 2024-04-03T18:32:48.9793089Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7bd56d59a23bb39c.xml (deflated 38%) 2024-04-03T18:32:48.9795223Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c20dc9148c773596.xml (deflated 37%) 2024-04-03T18:32:48.9797401Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc8f809d4074dc40.xml (deflated 38%) 2024-04-03T18:32:48.9799606Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-04f8c7707fc80858.xml (deflated 37%) 2024-04-03T18:32:48.9801767Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5afc2f7c7465ffb6.xml (deflated 38%) 2024-04-03T18:32:48.9803924Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d81fa8726867c7b8.xml (deflated 43%) 2024-04-03T18:32:48.9806077Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ecc82bda17904351.xml (deflated 43%) 2024-04-03T18:32:48.9808245Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c9740b4c00dec8a.xml (deflated 45%) 2024-04-03T18:32:48.9810594Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-884e604def514d74.xml (deflated 37%) 2024-04-03T18:32:48.9812728Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-de4f8adb6a70730e.xml (deflated 38%) 2024-04-03T18:32:48.9814899Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-092d1a8c5927d312.xml (deflated 47%) 2024-04-03T18:32:48.9817069Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-52293368de00b5ba.xml (deflated 37%) 2024-04-03T18:32:48.9819231Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3b136d773bc35d1f.xml (deflated 38%) 2024-04-03T18:32:48.9821366Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b6be42469318ec2.xml (deflated 38%) 2024-04-03T18:32:48.9823536Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd5a72e667fb7bcf.xml (deflated 38%) 2024-04-03T18:32:48.9825684Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-156905a113bbcd64.xml (deflated 43%) 2024-04-03T18:32:48.9827823Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-de4acba819ff8834.xml (deflated 38%) 2024-04-03T18:32:48.9829990Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d9d9c981d632af4c.xml (deflated 45%) 2024-04-03T18:32:48.9832156Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1f6ba3db80d64de9.xml (deflated 42%) 2024-04-03T18:32:48.9834359Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-740000575d7444f7.xml (deflated 45%) 2024-04-03T18:32:48.9836629Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-58239f80d8082ef6.xml (deflated 38%) 2024-04-03T18:32:48.9838778Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-40cd4456e4a02c06.xml (deflated 37%) 2024-04-03T18:32:48.9841019Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5cba4bd4be1960b8.xml (deflated 37%) 2024-04-03T18:32:48.9843200Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-92880db2b0c2eb98.xml (deflated 37%) 2024-04-03T18:32:48.9845326Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7214b6544ef667f2.xml (deflated 39%) 2024-04-03T18:32:48.9847472Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2d7368dd588457eb.xml (deflated 37%) 2024-04-03T18:32:48.9849804Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f68ed2dacbe9f487.xml (deflated 43%) 2024-04-03T18:32:48.9851979Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-193e2a6eebed53f7.xml (deflated 37%) 2024-04-03T18:32:48.9854123Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7219292b57bf8625.xml (deflated 44%) 2024-04-03T18:32:48.9856285Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c32482ab9a544a9.xml (deflated 44%) 2024-04-03T18:32:48.9858428Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-640558a8b4f060ae.xml (deflated 44%) 2024-04-03T18:32:48.9860576Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e60d5f17eb280812.xml (deflated 44%) 2024-04-03T18:32:48.9862722Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1d9ae8f3e1c838a2.xml (deflated 37%) 2024-04-03T18:32:48.9864876Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-56883534c87bc451.xml (deflated 37%) 2024-04-03T18:32:48.9867023Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-49a4df69bc4046a7.xml (deflated 44%) 2024-04-03T18:32:48.9869202Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bfc84bca1ff5a8af.xml (deflated 43%) 2024-04-03T18:32:48.9871339Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6704d68a5bc078c4.xml (deflated 44%) 2024-04-03T18:32:48.9873494Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-00aee7fb7dfec551.xml (deflated 45%) 2024-04-03T18:32:48.9875658Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-93fc0428777d6cf4.xml (deflated 44%) 2024-04-03T18:32:48.9877800Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a239148e7098127c.xml (deflated 41%) 2024-04-03T18:32:48.9879932Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9803f539c9803afe.xml (deflated 42%) 2024-04-03T18:32:48.9882070Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ced0c6086cdb6dc.xml (deflated 42%) 2024-04-03T18:32:48.9884222Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1dbc1fcae9d26d34.xml (deflated 57%) 2024-04-03T18:32:48.9886378Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9268dce328ccbab.xml (deflated 37%) 2024-04-03T18:32:48.9888710Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b86435ebda9dbf69.xml (deflated 37%) 2024-04-03T18:32:48.9891052Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-caa3048bfd056c06.xml (deflated 37%) 2024-04-03T18:32:48.9893318Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c51b5e6b1bbfd63d.xml (deflated 37%) 2024-04-03T18:32:48.9895506Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4d74bcc93c05fec6.xml (deflated 44%) 2024-04-03T18:32:48.9897640Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-221dae6b3e1e9d42.xml (deflated 37%) 2024-04-03T18:32:48.9899807Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ffdab3aa52dfa33.xml (deflated 37%) 2024-04-03T18:32:48.9901979Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-02a074a7d5a38f47.xml (deflated 37%) 2024-04-03T18:32:48.9904130Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3aff1a8555ca4014.xml (deflated 37%) 2024-04-03T18:32:48.9906264Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-18045e4b2ad62748.xml (deflated 37%) 2024-04-03T18:32:48.9908434Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-370476ad58b5303d.xml (deflated 43%) 2024-04-03T18:32:48.9910578Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7c475ac51ed39a5f.xml (deflated 43%) 2024-04-03T18:32:48.9912728Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a25c76973e6bf527.xml (deflated 44%) 2024-04-03T18:32:48.9914862Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-49bb85b31714a524.xml (deflated 38%) 2024-04-03T18:32:48.9917011Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a90462d2ecaaa31e.xml (deflated 37%) 2024-04-03T18:32:48.9919179Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-594619d0f0861fff.xml (deflated 47%) 2024-04-03T18:32:48.9921335Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2737fbc4fb6e76c.xml (deflated 37%) 2024-04-03T18:32:48.9923466Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6fc658de7771f2d2.xml (deflated 38%) 2024-04-03T18:32:48.9925606Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15c30746a7af7c1a.xml (deflated 38%) 2024-04-03T18:32:48.9927751Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f3a893755458f42.xml (deflated 38%) 2024-04-03T18:32:48.9930073Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb269b1b6c8e2d3d.xml (deflated 43%) 2024-04-03T18:32:48.9932227Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e634b399179bc4e.xml (deflated 38%) 2024-04-03T18:32:48.9934415Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4b6592e1de72fd5f.xml (deflated 45%) 2024-04-03T18:32:48.9936554Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b79b83fcc6b84bfa.xml (deflated 42%) 2024-04-03T18:32:48.9938706Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bb78b8bffd8a1009.xml (deflated 45%) 2024-04-03T18:32:48.9940987Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-14b9910993260706.xml (deflated 38%) 2024-04-03T18:32:48.9943124Z adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/distributed.test_distributed_spawn-62d2d1cbd7a47379.xml (deflated 37%) 2024-04-03T18:32:48.9998042Z ##[group]Run # Remove any previous test reports if they exist 2024-04-03T18:32:48.9998897Z # Remove any previous test reports if they exist 2024-04-03T18:32:48.9999525Z rm -f logs-*.zip 2024-04-03T18:32:49.0000369Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-04-03T18:32:49.0001361Z # so check to see if the file exists first 2024-04-03T18:32:49.0001976Z if [ -f 'usage_log.txt' ]; then 2024-04-03T18:32:49.0002639Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-04-03T18:32:49.0003263Z fi 2024-04-03T18:32:49.0003696Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-04-03T18:32:49.0004416Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-04-03T18:32:49.0005038Z fi 2024-04-03T18:32:49.0013659Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:32:49.0014273Z env: 2024-04-03T18:32:49.0014612Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:32:49.0015188Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:32:49.0016122Z DOCKER_CONTAINER_ID: b758c44c85badb76c4c382a8895cd5e1bfdf37df294e5f1babfcbd8fbafbdd0d 2024-04-03T18:32:49.0017226Z FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_23402705639 2024-04-03T18:32:49.0017985Z ##[endgroup] 2024-04-03T18:32:49.0615306Z adding: usage_log.txt (deflated 96%) 2024-04-03T18:32:49.0920560Z adding: test/test-reports/distributed.test_distributed_spawn_5.7_856cb86d1ee1daf9_.log (deflated 94%) 2024-04-03T18:32:49.0922134Z adding: test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_1a50b1bcadad1c56_.log (deflated 54%) 2024-04-03T18:32:49.0923596Z adding: test/test-reports/distributed._spmd.test_tracing_1.1_d5f30c847a02f129_.log (stored 0%) 2024-04-03T18:32:49.0924991Z adding: test/test-reports/distributed._spmd.test_graph_utils_1.1_0271ed49474f7cc1_.log (stored 0%) 2024-04-03T18:32:49.0932223Z adding: test/test-reports/distributed.test_distributed_spawn_5.7_b32bbab13b3eec33_.log (deflated 94%) 2024-04-03T18:32:49.0933739Z adding: test/test-reports/distributed._composable.test_contract_1.1_cbebec6dfee90cd3_.log (deflated 64%) 2024-04-03T18:32:49.0944563Z adding: test/test-reports/distributed.test_distributed_spawn_5.7_d4c0c027f37bbc54_.log (deflated 93%) 2024-04-03T18:32:49.0946059Z adding: test/test-reports/distributed.checkpoint.test_traverse_1.1_24daab968d1b6f89_.log (deflated 71%) 2024-04-03T18:32:49.0947540Z adding: test/test-reports/distributed.fsdp.test_shard_utils_1.1_269e2c9fdad190d0_.log (deflated 59%) 2024-04-03T18:32:49.0948997Z adding: test/test-reports/distributed.fsdp.test_fsdp_fx_1.1_9ea34eeace00f196_.log (deflated 53%) 2024-04-03T18:32:49.0957043Z adding: test/test-reports/distributed.test_distributed_spawn_5.7_915ce8a80924b639_.log (deflated 93%) 2024-04-03T18:32:49.0958595Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_bb6c00d2dd342206_.log (deflated 71%) 2024-04-03T18:32:49.0969692Z adding: test/test-reports/distributed.test_distributed_spawn_5.7_8ff465ec967bfa97_.log (deflated 93%) 2024-04-03T18:32:49.0971267Z adding: test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_0363f49263bc932a_.log (deflated 52%) 2024-04-03T18:32:49.0972835Z adding: test/test-reports/distributed.checkpoint.test_tp_checkpoint_1.1_87ebbf6392b481be_.log (deflated 77%) 2024-04-03T18:32:49.0974415Z adding: test/test-reports/distributed.checkpoint.test_compatibility_1.1_c8765490a36f84fb_.log (deflated 59%) 2024-04-03T18:32:49.0976190Z adding: test/test-reports/distributed._tensor.test_tensor_ops_1.1_a64503e4a4f6c248_.log (deflated 80%) 2024-04-03T18:32:49.0977749Z adding: test/test-reports/distributed.checkpoint.fsdp.test_fsdp_dsd_1.1_d2f8c7f0e958b7a5_.log (deflated 62%) 2024-04-03T18:32:49.0979349Z adding: test/test-reports/distributed.checkpoint.test_format_utils_1.1_1488c45e86ed8da0_.log (deflated 71%) 2024-04-03T18:32:49.0980966Z adding: test/test-reports/distributed._tensor.test_common_rules_1.1_ee667e9ed72dbc56_.log (deflated 77%) 2024-04-03T18:32:49.0982517Z adding: test/test-reports/distributed.test_dynamo_distributed_1.1_be94fbcf6ae657f3_.log (deflated 82%) 2024-04-03T18:32:49.0983965Z adding: test/test-reports/distributed.test_c10d_functional_native_1.1_8091fbb27d8a0715_.log (deflated 86%) 2024-04-03T18:32:49.0985405Z adding: test/test-reports/distributed.test_device_mesh_1.1_8d033f1f4ead02be_.log (deflated 90%) 2024-04-03T18:32:49.0986830Z adding: test/test-reports/distributed._tensor.test_dtensor_ops_1.1_d778597415f28c45_.log (deflated 52%) 2024-04-03T18:32:49.0988263Z adding: test/test-reports/distributed.fsdp.test_wrap_1.1_989e6a50fe1d14e9_.log (deflated 92%) 2024-04-03T18:32:49.0989705Z adding: test/test-reports/distributed.test_distributed_spawn_2.7_300eb5a82d2641f4_.log (deflated 12%) 2024-04-03T18:32:49.0991144Z adding: test/test-reports/distributed.fsdp.test_fsdp_meta_1.1_a454565346ba189b_.log (deflated 86%) 2024-04-03T18:32:49.0992554Z adding: test/test-reports/distributed.test_distributed_spawn_2.7_5d2ab21067d12ed5_.log (deflated 12%) 2024-04-03T18:32:49.0994010Z adding: test/test-reports/distributed.fsdp.test_fsdp_comm_1.1_09165abdbe44cf1c_.log (deflated 88%) 2024-04-03T18:32:49.0998560Z adding: test/test-reports/distributed.test_distributed_spawn_2.7_4cd359dbf8fd866e_.log (deflated 95%) 2024-04-03T18:32:49.1001204Z adding: test/test-reports/distributed.fsdp.test_fsdp_misc_1.1_4452d7a2a205935b_.log (deflated 86%) 2024-04-03T18:32:49.1008061Z adding: test/test-reports/distributed.test_distributed_spawn_2.7_fb4e089d400a7867_.log (deflated 95%) 2024-04-03T18:32:49.1009974Z adding: test/test-reports/distributed.test_c10d_spawn_gloo_1.1_b518259205201e28_.log (deflated 93%) 2024-04-03T18:32:49.1018894Z adding: test/test-reports/distributed.test_distributed_spawn_2.7_d05238268882bd15_.log (deflated 95%) 2024-04-03T18:32:49.1033421Z adding: test/test-reports/distributed.test_c10d_ucc_1.1_5752da3391f6adfd_.log (deflated 95%) 2024-04-03T18:32:49.1041985Z adding: test/test-reports/distributed.test_distributed_spawn_2.7_96c97bb3235c70e2_.log (deflated 95%) 2024-04-03T18:32:49.1043352Z adding: test/test-reports/distributed.test_launcher_1.1_b45e3f774af04074_.log (deflated 58%) 2024-04-03T18:32:49.1052148Z adding: test/test-reports/distributed.test_distributed_spawn_2.7_091797c278c884aa_.log (deflated 94%) 2024-04-03T18:32:49.1079463Z adding: test/test-reports/distributed.test_c10d_nccl_1.2_f89881d3fdec9ff8_.log (deflated 94%) 2024-04-03T18:32:49.1088731Z adding: test/test-reports/distributed.test_distributed_spawn_2.7_fc5e2274bae7b13e_.log (deflated 94%) 2024-04-03T18:32:49.1090418Z adding: test/test-reports/distributed.elastic.timer.api_test_1.1_c5873c44dc3fc7d9_.log (stored 0%) 2024-04-03T18:32:49.1091848Z adding: test/test-reports/distributed.test_distributed_spawn_5.7_a7aae86745f3466c_.log (deflated 12%) 2024-04-03T18:32:49.1093192Z adding: test/test-reports/cpp.FileStoreTest_1.1_c488c8fb6ae9e202_.log (deflated 71%) 2024-04-03T18:32:49.1094562Z adding: test/test-reports/distributed.test_distributed_spawn_5.7_715d09f971c889c2_.log (deflated 12%) 2024-04-03T18:32:49.1095903Z adding: test/test-reports/cpp.HashStoreTest_1.1_a437887af7a7d024_.log (deflated 71%) 2024-04-03T18:32:49.1101196Z adding: test/test-reports/distributed.test_distributed_spawn_5.7_62e08a1a18ddadaf_.log (deflated 93%) 2024-04-03T18:32:49.1102741Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_init_1.1_0a3062f88c4293eb_.log (deflated 84%) 2024-04-03T18:32:49.1104341Z adding: test/test-reports/distributed._tools.test_memory_tracker_1.1_3275b5adcd8c41c3_.log (deflated 54%) 2024-04-03T18:32:49.1105995Z adding: test/test-reports/distributed._tensor.test_xla_integration_1.1_19b4f89c81ed5ddb_.log (deflated 66%) 2024-04-03T18:32:49.1107365Z adding: test/test-reports/cpp.TCPStoreTest_1.1_75e357ac54058d35_.log (deflated 79%) 2024-04-03T18:32:49.1110486Z adding: test/test-reports/distributed.optim.test_zero_redundancy_optimizer_1.1_6e5f0d7c773bd51a_.log (deflated 93%) 2024-04-03T18:32:49.1112223Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_overlap_1.1_fd6e6144e195c7b7_.log (deflated 57%) 2024-04-03T18:32:49.1113786Z adding: test/test-reports/distributed._tensor.test_pointwise_ops_1.1_d548ab8fbd32d761_.log (deflated 69%) 2024-04-03T18:32:49.1115383Z adding: test/test-reports/distributed.tensor.parallel.test_tp_style_1.1_2d21dafd9d43b722_.log (deflated 74%) 2024-04-03T18:32:49.1117034Z adding: test/test-reports/distributed.checkpoint.test_dtensor_checkpoint_1.1_6a5b91b09f5dbe12_.log (deflated 67%) 2024-04-03T18:32:49.1118610Z adding: test/test-reports/distributed._tensor.test_op_strategy_1.1_a1be2495db9726d7_.log (deflated 78%) 2024-04-03T18:32:49.1119980Z adding: test/test-reports/cpp.ProcessGroupGlooTest_1.1_54857242c1147efe_.log (deflated 85%) 2024-04-03T18:32:49.1121558Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_memory_1.1_f1a19e0da63a0ab3_.log (deflated 56%) 2024-04-03T18:32:49.1123210Z adding: test/test-reports/distributed.checkpoint.e2e.test_pipeline_1.1_472b6598bd660329_.log (deflated 53%) 2024-04-03T18:32:49.1124716Z adding: test/test-reports/distributed._tensor.test_embedding_ops_1.1_40395176b4c17540_.log (deflated 63%) 2024-04-03T18:32:49.1126295Z adding: test/test-reports/distributed.checkpoint.test_hsdp_checkpoint_1.1_3b8bd579070614c7_.log (deflated 70%) 2024-04-03T18:32:49.1127873Z adding: test/test-reports/distributed.checkpoint.e2e.test_fine_tuning_1.1_e6e7dbe809df9c08_.log (deflated 54%) 2024-04-03T18:32:49.1129714Z adding: test/test-reports/distributed._tensor.test_matrix_ops_1.1_76dfade85d5f6b25_.log (deflated 76%) 2024-04-03T18:32:49.1131318Z adding: test/test-reports/distributed._shard.sharded_optim.test_sharded_optim_1.1_0eb7e660423b38d1_.log (deflated 61%) 2024-04-03T18:32:49.1133093Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_mixed_precision_1.1_f6e2fd1841674e45_.log (deflated 66%) 2024-04-03T18:32:49.1134990Z adding: test/test-reports/distributed.pipeline.sync.skip.test_inspect_skip_layout_1.1_9e1d6c75a7047875_.log (deflated 67%) 2024-04-03T18:32:49.1136803Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_compile_1.1_b093cef8e7e7c1c3_.log (deflated 57%) 2024-04-03T18:32:49.1138509Z adding: test/test-reports/distributed.checkpoint.test_fsdp_model_state_1.1_8e94ffe223e2e77a_.log (deflated 71%) 2024-04-03T18:32:49.1140033Z adding: test/test-reports/cpp.ProcessGroupNCCLTest_1.1_f74d0a8259aa79d9_.log (deflated 82%) 2024-04-03T18:32:49.1295681Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_1fabdb19266e44bd_.log (deflated 95%) 2024-04-03T18:32:49.1297355Z adding: test/test-reports/distributed.fsdp.test_fsdp_pure_fp16_1.1_4aa4b9e70a8d30a4_.log (deflated 74%) 2024-04-03T18:32:49.1306925Z adding: test/test-reports/distributed.fsdp.test_fsdp_optim_state_1.1_33ed4a25d596538d_.log (deflated 96%) 2024-04-03T18:32:49.1308481Z adding: test/test-reports/distributed.fsdp.test_checkpoint_wrapper_1.1_e901cad03dbf70f8_.log (deflated 71%) 2024-04-03T18:32:49.1312989Z adding: test/test-reports/distributed.fsdp.test_fsdp_grad_acc_1.1_c6645c6ae19e08c7_.log (deflated 95%) 2024-04-03T18:32:49.1316104Z adding: test/test-reports/distributed.fsdp.test_fsdp_sharded_grad_scaler_1.1_2fa04a26791b357d_.log (deflated 93%) 2024-04-03T18:32:49.1325606Z adding: test/test-reports/distributed.fsdp.test_fsdp_state_dict_2.2_dad4170aff65df97_.log (deflated 95%) 2024-04-03T18:32:49.1327110Z adding: test/test-reports/cpp.ProcessGroupNCCLErrorsTest_1.1_78f69ca470e83b20_.log (deflated 78%) 2024-04-03T18:32:49.1328926Z adding: test/test-reports/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks_1.1_512b80199acde4d7_.log (deflated 74%) 2024-04-03T18:32:49.1330751Z adding: test/test-reports/cpp.test_cpp_rpc_1.1_6ed2ad04e9d1d6f0_.log (deflated 79%) 2024-04-03T18:32:49.1332384Z adding: test/test-reports/distributed.checkpoint.test_file_system_checkpoint_cpu_1.1_7254c26b5ab6c16b_.log (deflated 93%) 2024-04-03T18:32:49.1334079Z adding: test/test-reports/distributed.fsdp.test_fsdp_comm_hooks_1.1_17b16091b18d084c_.log (deflated 92%) 2024-04-03T18:32:49.1339294Z adding: test/test-reports/distributed.fsdp.test_fsdp_use_orig_params_1.1_7c24337a26eef6dc_.log (deflated 93%) 2024-04-03T18:32:49.1340893Z adding: test/test-reports/distributed.fsdp.test_fsdp_multiple_forward_1.1_71962f4b9c85ba99_.log (deflated 63%) 2024-04-03T18:32:49.1342490Z adding: test/test-reports/distributed.pipeline.sync.test_microbatch_1.1_8d1843845297fa95_.log (deflated 71%) 2024-04-03T18:32:49.1344110Z adding: test/test-reports/distributed.pipeline.sync.test_checkpoint_1.1_63ca0b1fc50a3584_.log (deflated 70%) 2024-04-03T18:32:49.1345713Z adding: test/test-reports/distributed.pipeline.sync.skip.test_portal_1.1_cc20ba5afe23ded7_.log (deflated 72%) 2024-04-03T18:32:49.1347370Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_init_1.1_8a5aed4219eff876_.log (deflated 66%) 2024-04-03T18:32:49.1348956Z adding: test/test-reports/distributed.rpc.test_tensorpipe_agent_1.1_3341a2dc731117d2_.log (deflated 36%) 2024-04-03T18:32:49.1350462Z adding: test/test-reports/distributed.pipeline.sync.test_copy_1.1_9ca68a953fcc5127_.log (deflated 62%) 2024-04-03T18:32:49.1358384Z adding: test/test-reports/distributed.fsdp.test_fsdp_mixed_precision_1.1_6f32697801a011a3_.log (deflated 95%) 2024-04-03T18:32:49.1360075Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_binary_cmp_1.1_602bea0828bd0340_.log (deflated 69%) 2024-04-03T18:32:49.1361768Z adding: test/test-reports/distributed.pipeline.sync.skip.test_tracker_1.1_c76e5079f4873f80_.log (deflated 66%) 2024-04-03T18:32:49.1363382Z adding: test/test-reports/distributed.pipeline.sync.skip.test_leak_1.1_7a5939bbe6d2696a_.log (deflated 70%) 2024-04-03T18:32:49.1365025Z adding: test/test-reports/distributed.pipeline.sync.skip.test_stash_pop_1.1_0d607cdf81cc8845_.log (deflated 67%) 2024-04-03T18:32:49.1434184Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-04-03T18:32:49.1434724Z with: 2024-04-03T18:32:49.1435067Z s3-bucket: gha-artifacts 2024-04-03T18:32:49.1435624Z s3-prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T18:32:49.1436178Z retention-days: 14 2024-04-03T18:32:49.1436622Z if-no-files-found: warn 2024-04-03T18:32:49.1437073Z path: test-jsons-*.zip 2024-04-03T18:32:49.1437505Z name: artifact 2024-04-03T18:32:49.1437868Z region: us-east-1 2024-04-03T18:32:49.1438241Z env: 2024-04-03T18:32:49.1438576Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:32:49.1439136Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:32:49.1440116Z DOCKER_CONTAINER_ID: b758c44c85badb76c4c382a8895cd5e1bfdf37df294e5f1babfcbd8fbafbdd0d 2024-04-03T18:32:49.1440990Z ##[endgroup] 2024-04-03T18:32:49.5948634Z NOTE: s3-prefix specified, ignoring name parameter 2024-04-03T18:32:49.5949638Z With the provided path, there will be 1 file uploaded 2024-04-03T18:32:49.5950451Z Uploading to s3 prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T18:32:49.5998356Z Starting upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_23402705639.zip 2024-04-03T18:32:49.7571503Z Finished upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_23402705639.zip 2024-04-03T18:32:49.7833184Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-04-03T18:32:49.7833787Z with: 2024-04-03T18:32:49.7834137Z s3-bucket: gha-artifacts 2024-04-03T18:32:49.7834673Z s3-prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T18:32:49.7835242Z retention-days: 14 2024-04-03T18:32:49.7835902Z if-no-files-found: error 2024-04-03T18:32:49.7836358Z path: test-reports-*.zip 2024-04-03T18:32:49.7836776Z name: artifact 2024-04-03T18:32:49.7837149Z region: us-east-1 2024-04-03T18:32:49.7837517Z env: 2024-04-03T18:32:49.7837832Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:32:49.7838395Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:32:49.7839496Z DOCKER_CONTAINER_ID: b758c44c85badb76c4c382a8895cd5e1bfdf37df294e5f1babfcbd8fbafbdd0d 2024-04-03T18:32:49.7840339Z ##[endgroup] 2024-04-03T18:32:50.2306996Z NOTE: s3-prefix specified, ignoring name parameter 2024-04-03T18:32:50.2307939Z With the provided path, there will be 1 file uploaded 2024-04-03T18:32:50.2308662Z Uploading to s3 prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T18:32:50.2356320Z Starting upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_23402705639.zip 2024-04-03T18:32:50.4563682Z Finished upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_23402705639.zip 2024-04-03T18:32:50.4763261Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-04-03T18:32:50.4763810Z with: 2024-04-03T18:32:50.4764162Z s3-bucket: gha-artifacts 2024-04-03T18:32:50.4764698Z s3-prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T18:32:50.4765271Z retention-days: 14 2024-04-03T18:32:50.4765695Z if-no-files-found: ignore 2024-04-03T18:32:50.4766142Z path: logs-*.zip 2024-04-03T18:32:50.4766507Z name: artifact 2024-04-03T18:32:50.4766908Z region: us-east-1 2024-04-03T18:32:50.4767283Z env: 2024-04-03T18:32:50.4767599Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:32:50.4768168Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:32:50.4769462Z DOCKER_CONTAINER_ID: b758c44c85badb76c4c382a8895cd5e1bfdf37df294e5f1babfcbd8fbafbdd0d 2024-04-03T18:32:50.4770386Z ##[endgroup] 2024-04-03T18:32:50.9311844Z NOTE: s3-prefix specified, ignoring name parameter 2024-04-03T18:32:50.9312810Z With the provided path, there will be 1 file uploaded 2024-04-03T18:32:50.9313615Z Uploading to s3 prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T18:32:50.9360908Z Starting upload of logs-test-distributed-1-3-linux.8xlarge.nvidia.gpu_23402705639.zip 2024-04-03T18:32:51.2256416Z Finished upload of logs-test-distributed-1-3-linux.8xlarge.nvidia.gpu_23402705639.zip 2024-04-03T18:32:51.2476026Z ##[group]Run # shellcheck disable=SC2156 2024-04-03T18:32:51.2476780Z # shellcheck disable=SC2156 2024-04-03T18:32:51.2477811Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-04-03T18:32:51.2486778Z shell: /usr/bin/bash -e {0} 2024-04-03T18:32:51.2487209Z env: 2024-04-03T18:32:51.2487558Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:32:51.2488130Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:32:51.2489394Z DOCKER_CONTAINER_ID: b758c44c85badb76c4c382a8895cd5e1bfdf37df294e5f1babfcbd8fbafbdd0d 2024-04-03T18:32:51.2490235Z ##[endgroup] 2024-04-03T18:32:51.5028050Z GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2 2024-04-03T18:32:51.5028804Z Copyright (C) 2020 Free Software Foundation, Inc. 2024-04-03T18:32:51.5029701Z License GPLv3+: GNU GPL version 3 or later 2024-04-03T18:32:51.5030773Z This is free software: you are free to change and redistribute it. 2024-04-03T18:32:51.5031589Z There is NO WARRANTY, to the extent permitted by law. 2024-04-03T18:32:51.5032316Z Type "show copying" and "show warranty" for details. 2024-04-03T18:32:51.5033072Z This GDB was configured as "x86_64-linux-gnu". 2024-04-03T18:32:51.5033825Z Type "show configuration" for configuration details. 2024-04-03T18:32:51.5034479Z For bug reporting instructions, please see: 2024-04-03T18:32:51.5035090Z . 2024-04-03T18:32:51.5035825Z Find the GDB manual and other documentation resources online at: 2024-04-03T18:32:51.5036618Z . 2024-04-03T18:32:51.5037093Z 2024-04-03T18:32:51.5037585Z For help, type "help". 2024-04-03T18:32:51.5038196Z Type "apropos word" to search for commands related to "word"... 2024-04-03T18:32:51.5802492Z Reading symbols from python... 2024-04-03T18:32:52.2811332Z 2024-04-03T18:32:52.2811832Z warning: core file may not match specified executable file. 2024-04-03T18:32:52.3127439Z [New LWP 126341] 2024-04-03T18:32:52.3127875Z [New LWP 126273] 2024-04-03T18:32:52.3128265Z [New LWP 126276] 2024-04-03T18:32:52.3128632Z [New LWP 126281] 2024-04-03T18:32:52.3129012Z [New LWP 126283] 2024-04-03T18:32:52.3129651Z [New LWP 126284] 2024-04-03T18:32:52.3129995Z [New LWP 126288] 2024-04-03T18:32:52.3130385Z [New LWP 126290] 2024-04-03T18:32:52.3130753Z [New LWP 126297] 2024-04-03T18:32:52.3131122Z [New LWP 126275] 2024-04-03T18:32:52.3131468Z [New LWP 126274] 2024-04-03T18:32:52.3131837Z [New LWP 126278] 2024-04-03T18:32:52.3132190Z [New LWP 126277] 2024-04-03T18:32:52.3132562Z [New LWP 126279] 2024-04-03T18:32:52.3132934Z [New LWP 126280] 2024-04-03T18:32:52.3133329Z [New LWP 126282] 2024-04-03T18:32:52.3133728Z [New LWP 126303] 2024-04-03T18:32:52.3134101Z [New LWP 126271] 2024-04-03T18:32:52.3134448Z [New LWP 126287] 2024-04-03T18:32:52.3134814Z [New LWP 126285] 2024-04-03T18:32:52.3135182Z [New LWP 126289] 2024-04-03T18:32:52.3135530Z [New LWP 126286] 2024-04-03T18:32:52.3135906Z [New LWP 126291] 2024-04-03T18:32:52.3136275Z [New LWP 126292] 2024-04-03T18:32:52.3136622Z [New LWP 126293] 2024-04-03T18:32:52.3136987Z [New LWP 126295] 2024-04-03T18:32:52.3137352Z [New LWP 126294] 2024-04-03T18:32:52.3137702Z [New LWP 126296] 2024-04-03T18:32:52.3138066Z [New LWP 126298] 2024-04-03T18:32:52.3138431Z [New LWP 126299] 2024-04-03T18:32:52.3138775Z [New LWP 126300] 2024-04-03T18:32:52.3139141Z [New LWP 126301] 2024-04-03T18:32:52.3139509Z [New LWP 126302] 2024-04-03T18:32:52.3139868Z [New LWP 126342] 2024-04-03T18:32:52.3140233Z [New LWP 126350] 2024-04-03T18:32:52.3140601Z [New LWP 126339] 2024-04-03T18:32:52.3140949Z [New LWP 126336] 2024-04-03T18:32:52.3141313Z [New LWP 126340] 2024-04-03T18:32:52.3141681Z [New LWP 126354] 2024-04-03T18:32:52.3142048Z [New LWP 126352] 2024-04-03T18:32:52.3172790Z [Thread debugging using libthread_db enabled] 2024-04-03T18:32:52.3173839Z Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1". 2024-04-03T18:33:08.1610478Z 50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. 2024-04-03T18:33:08.1612119Z Core was generated by `/opt/conda/envs/py_3.10/bin/python -c from multiprocessing.spawn import spawn_m'. 2024-04-03T18:33:08.1614062Z warning: File "/var/lib/jenkins/workspace/.gdbinit" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load". 2024-04-03T18:33:08.1615712Z Program terminated with signal SIGABRT, Aborted. 2024-04-03T18:33:08.1616506Z #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 2024-04-03T18:33:08.1617357Z [Current thread is 1 (Thread 0x7f8d16745700 (LWP 126341))] 2024-04-03T18:33:08.1706664Z To enable execution of this file add 2024-04-03T18:33:08.1707528Z add-auto-load-safe-path /var/lib/jenkins/workspace/.gdbinit 2024-04-03T18:33:08.1708352Z line to your configuration file "/var/lib/jenkins/.gdbinit". 2024-04-03T18:33:08.1709115Z To completely disable this security protection add 2024-04-03T18:33:08.1709788Z set auto-load safe-path / 2024-04-03T18:33:08.1710391Z line to your configuration file "/var/lib/jenkins/.gdbinit". 2024-04-03T18:33:08.1711209Z For more information about this security protection see the 2024-04-03T18:33:08.1712226Z "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: 2024-04-03T18:33:08.1713084Z info "(gdb)Auto-loading safe path" 2024-04-03T18:33:08.1713787Z #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 2024-04-03T18:33:08.1714591Z #1 0x00007f8e19d54859 in __GI_abort () at abort.c:79 2024-04-03T18:33:08.2824732Z #2 0x00007f8df43ab5e9 in c10::MessageLogger::~MessageLogger() [clone .cold] 2024-04-03T18:33:08.2825477Z () 2024-04-03T18:33:08.2826313Z from /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so 2024-04-03T18:33:08.2838679Z #3 0x00007f8df56f8874 in c10d::ProcessGroupNCCL::terminateProcess(std::__cxx11::basic_string, std::allocator >) () 2024-04-03T18:33:08.2840348Z from /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so 2024-04-03T18:33:08.2853631Z #4 0x00007f8df56feb13 in c10d::ProcessGroupNCCL::heartbeatMonitor() () 2024-04-03T18:33:08.2855993Z from /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so 2024-04-03T18:33:08.2946975Z #5 0x00007f8e118c3aeb in std::execute_native_thread_routine ( 2024-04-03T18:33:08.2948951Z __p=) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 2024-04-03T18:33:08.2950540Z #6 0x00007f8e1a086609 in start_thread (arg=) 2024-04-03T18:33:08.2951721Z at pthread_create.c:477 2024-04-03T18:33:08.2952518Z #7 0x00007f8e19e51353 in clone () 2024-04-03T18:33:08.2953509Z at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 2024-04-03T18:33:08.4969773Z GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2 2024-04-03T18:33:08.4970536Z Copyright (C) 2020 Free Software Foundation, Inc. 2024-04-03T18:33:08.4971492Z License GPLv3+: GNU GPL version 3 or later 2024-04-03T18:33:08.4972492Z This is free software: you are free to change and redistribute it. 2024-04-03T18:33:08.4973311Z There is NO WARRANTY, to the extent permitted by law. 2024-04-03T18:33:08.4974047Z Type "show copying" and "show warranty" for details. 2024-04-03T18:33:08.4974805Z This GDB was configured as "x86_64-linux-gnu". 2024-04-03T18:33:08.4975487Z Type "show configuration" for configuration details. 2024-04-03T18:33:08.4976147Z For bug reporting instructions, please see: 2024-04-03T18:33:08.4976765Z . 2024-04-03T18:33:08.4977545Z Find the GDB manual and other documentation resources online at: 2024-04-03T18:33:08.4978334Z . 2024-04-03T18:33:08.4978834Z 2024-04-03T18:33:08.4978997Z For help, type "help". 2024-04-03T18:33:08.4979608Z Type "apropos word" to search for commands related to "word"... 2024-04-03T18:33:08.5731371Z Reading symbols from python... 2024-04-03T18:33:09.2588742Z 2024-04-03T18:33:09.2590079Z warning: core file may not match specified executable file. 2024-04-03T18:33:09.2893660Z [New LWP 126465] 2024-04-03T18:33:09.2894041Z [New LWP 126441] 2024-04-03T18:33:09.2894467Z [New LWP 126402] 2024-04-03T18:33:09.2894796Z [New LWP 126410] 2024-04-03T18:33:09.2895127Z [New LWP 126418] 2024-04-03T18:33:09.2895566Z [New LWP 126424] 2024-04-03T18:33:09.2895902Z [New LWP 126429] 2024-04-03T18:33:09.2896244Z [New LWP 126433] 2024-04-03T18:33:09.2896567Z [New LWP 126397] 2024-04-03T18:33:09.2896932Z [New LWP 126399] 2024-04-03T18:33:09.2897281Z [New LWP 126400] 2024-04-03T18:33:09.2897647Z [New LWP 126405] 2024-04-03T18:33:09.2898016Z [New LWP 126403] 2024-04-03T18:33:09.2898381Z [New LWP 126407] 2024-04-03T18:33:09.2898770Z [New LWP 126414] 2024-04-03T18:33:09.2899129Z [New LWP 126412] 2024-04-03T18:33:09.2899472Z [New LWP 126416] 2024-04-03T18:33:09.2899835Z [New LWP 126394] 2024-04-03T18:33:09.2900201Z [New LWP 126422] 2024-04-03T18:33:09.2900548Z [New LWP 126421] 2024-04-03T18:33:09.2900919Z [New LWP 126425] 2024-04-03T18:33:09.2901270Z [New LWP 126427] 2024-04-03T18:33:09.2901632Z [New LWP 126431] 2024-04-03T18:33:09.2901993Z [New LWP 126437] 2024-04-03T18:33:09.2902336Z [New LWP 126435] 2024-04-03T18:33:09.2902699Z [New LWP 126443] 2024-04-03T18:33:09.2903061Z [New LWP 126439] 2024-04-03T18:33:09.2903406Z [New LWP 126445] 2024-04-03T18:33:09.2903768Z [New LWP 126447] 2024-04-03T18:33:09.2904132Z [New LWP 126449] 2024-04-03T18:33:09.2904474Z [New LWP 126453] 2024-04-03T18:33:09.2904834Z [New LWP 126455] 2024-04-03T18:33:09.2905198Z [New LWP 126462] 2024-04-03T18:33:09.2905545Z [New LWP 126451] 2024-04-03T18:33:09.2905912Z [New LWP 126475] 2024-04-03T18:33:09.2906277Z [New LWP 126473] 2024-04-03T18:33:09.2906862Z [New LWP 126478] 2024-04-03T18:33:09.2907225Z [New LWP 126461] 2024-04-03T18:33:09.2907586Z [New LWP 126468] 2024-04-03T18:33:09.2907933Z [New LWP 126464] 2024-04-03T18:33:09.2937582Z [Thread debugging using libthread_db enabled] 2024-04-03T18:33:09.2938621Z Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1". 2024-04-03T18:33:24.9657941Z 50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. 2024-04-03T18:33:24.9659374Z Core was generated by `/opt/conda/envs/py_3.10/bin/python -c from multiprocessing.spawn import spawn_m'. 2024-04-03T18:33:24.9661300Z warning: File "/var/lib/jenkins/workspace/.gdbinit" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load". 2024-04-03T18:33:24.9662680Z Program terminated with signal SIGABRT, Aborted. 2024-04-03T18:33:24.9663467Z #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 2024-04-03T18:33:24.9664319Z [Current thread is 1 (Thread 0x7fbc6e3b5700 (LWP 126465))] 2024-04-03T18:33:24.9736903Z To enable execution of this file add 2024-04-03T18:33:24.9737696Z add-auto-load-safe-path /var/lib/jenkins/workspace/.gdbinit 2024-04-03T18:33:24.9738485Z line to your configuration file "/var/lib/jenkins/.gdbinit". 2024-04-03T18:33:24.9739251Z To completely disable this security protection add 2024-04-03T18:33:24.9739914Z set auto-load safe-path / 2024-04-03T18:33:24.9740505Z line to your configuration file "/var/lib/jenkins/.gdbinit". 2024-04-03T18:33:24.9741304Z For more information about this security protection see the 2024-04-03T18:33:24.9742315Z "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: 2024-04-03T18:33:24.9743154Z info "(gdb)Auto-loading safe path" 2024-04-03T18:33:24.9743872Z #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 2024-04-03T18:33:24.9744674Z #1 0x00007fbd6112a859 in __GI_abort () at abort.c:79 2024-04-03T18:33:25.0834163Z #2 0x00007fbd3b8815e9 in c10::MessageLogger::~MessageLogger() [clone .cold] 2024-04-03T18:33:25.0834897Z () 2024-04-03T18:33:25.0835681Z from /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so 2024-04-03T18:33:25.0847483Z #3 0x00007fbd3cbce874 in c10d::ProcessGroupNCCL::terminateProcess(std::__cxx11::basic_string, std::allocator >) () 2024-04-03T18:33:25.0849644Z from /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so 2024-04-03T18:33:25.0859574Z #4 0x00007fbd3cbd4b13 in c10d::ProcessGroupNCCL::heartbeatMonitor() () 2024-04-03T18:33:25.0860698Z from /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so 2024-04-03T18:33:25.0950507Z #5 0x00007fbd58d99aeb in std::execute_native_thread_routine ( 2024-04-03T18:33:25.0951431Z __p=) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 2024-04-03T18:33:25.0953032Z #6 0x00007fbd6145c609 in start_thread (arg=) 2024-04-03T18:33:25.0953686Z at pthread_create.c:477 2024-04-03T18:33:25.0954095Z #7 0x00007fbd61227353 in clone () 2024-04-03T18:33:25.0954650Z at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 2024-04-03T18:33:25.3058894Z GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2 2024-04-03T18:33:25.3059608Z Copyright (C) 2020 Free Software Foundation, Inc. 2024-04-03T18:33:25.3060495Z License GPLv3+: GNU GPL version 3 or later 2024-04-03T18:33:25.3061483Z This is free software: you are free to change and redistribute it. 2024-04-03T18:33:25.3062311Z There is NO WARRANTY, to the extent permitted by law. 2024-04-03T18:33:25.3063019Z Type "show copying" and "show warranty" for details. 2024-04-03T18:33:25.3063768Z This GDB was configured as "x86_64-linux-gnu". 2024-04-03T18:33:25.3064447Z Type "show configuration" for configuration details. 2024-04-03T18:33:25.3065099Z For bug reporting instructions, please see: 2024-04-03T18:33:25.3065704Z . 2024-04-03T18:33:25.3066425Z Find the GDB manual and other documentation resources online at: 2024-04-03T18:33:25.3067557Z . 2024-04-03T18:33:25.3068032Z 2024-04-03T18:33:25.3068192Z For help, type "help". 2024-04-03T18:33:25.3068787Z Type "apropos word" to search for commands related to "word"... 2024-04-03T18:33:25.3828771Z Reading symbols from python... 2024-04-03T18:33:26.0941739Z 2024-04-03T18:33:26.0942483Z warning: core file may not match specified executable file. 2024-04-03T18:33:26.1253161Z [New LWP 126587] 2024-04-03T18:33:26.1253591Z [New LWP 126547] 2024-04-03T18:33:26.1253971Z [New LWP 126525] 2024-04-03T18:33:26.1254347Z [New LWP 126545] 2024-04-03T18:33:26.1254716Z [New LWP 126556] 2024-04-03T18:33:26.1255104Z [New LWP 126559] 2024-04-03T18:33:26.1255431Z [New LWP 126519] 2024-04-03T18:33:26.1255805Z [New LWP 126523] 2024-04-03T18:33:26.1256178Z [New LWP 126521] 2024-04-03T18:33:26.1256535Z [New LWP 126527] 2024-04-03T18:33:26.1256910Z [New LWP 126529] 2024-04-03T18:33:26.1257272Z [New LWP 126533] 2024-04-03T18:33:26.1257649Z [New LWP 126538] 2024-04-03T18:33:26.1257934Z [New LWP 126531] 2024-04-03T18:33:26.1258300Z [New LWP 126537] 2024-04-03T18:33:26.1258646Z [New LWP 126535] 2024-04-03T18:33:26.1259128Z [New LWP 126542] 2024-04-03T18:33:26.1259489Z [New LWP 126517] 2024-04-03T18:33:26.1259851Z [New LWP 126544] 2024-04-03T18:33:26.1260222Z [New LWP 126557] 2024-04-03T18:33:26.1260603Z [New LWP 126551] 2024-04-03T18:33:26.1260949Z [New LWP 126558] 2024-04-03T18:33:26.1261314Z [New LWP 126562] 2024-04-03T18:33:26.1261677Z [New LWP 126560] 2024-04-03T18:33:26.1262020Z [New LWP 126564] 2024-04-03T18:33:26.1262381Z [New LWP 126568] 2024-04-03T18:33:26.1262741Z [New LWP 126566] 2024-04-03T18:33:26.1263086Z [New LWP 126549] 2024-04-03T18:33:26.1263446Z [New LWP 126570] 2024-04-03T18:33:26.1263808Z [New LWP 126576] 2024-04-03T18:33:26.1264152Z [New LWP 126572] 2024-04-03T18:33:26.1264514Z [New LWP 126574] 2024-04-03T18:33:26.1264885Z [New LWP 126585] 2024-04-03T18:33:26.1265234Z [New LWP 126586] 2024-04-03T18:33:26.1265606Z [New LWP 126597] 2024-04-03T18:33:26.1265951Z [New LWP 126599] 2024-04-03T18:33:26.1266317Z [New LWP 126584] 2024-04-03T18:33:26.1266679Z [New LWP 126578] 2024-04-03T18:33:26.1267026Z [New LWP 126588] 2024-04-03T18:33:26.1267390Z [New LWP 126601] 2024-04-03T18:33:26.1296246Z [Thread debugging using libthread_db enabled] 2024-04-03T18:33:26.1297325Z Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1". 2024-04-03T18:33:42.0541986Z 50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. 2024-04-03T18:33:42.0543705Z warning: File "/var/lib/jenkins/workspace/.gdbinit" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load". 2024-04-03T18:33:42.0545570Z Core was generated by `/opt/conda/envs/py_3.10/bin/python -c from multiprocessing.spawn import spawn_m'. 2024-04-03T18:33:42.0546639Z Program terminated with signal SIGABRT, Aborted. 2024-04-03T18:33:42.0547426Z #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 2024-04-03T18:33:42.0548326Z [Current thread is 1 (Thread 0x7f758b384700 (LWP 126587))] 2024-04-03T18:33:42.0620932Z To enable execution of this file add 2024-04-03T18:33:42.0621770Z add-auto-load-safe-path /var/lib/jenkins/workspace/.gdbinit 2024-04-03T18:33:42.0622551Z line to your configuration file "/var/lib/jenkins/.gdbinit". 2024-04-03T18:33:42.0623309Z To completely disable this security protection add 2024-04-03T18:33:42.0623955Z set auto-load safe-path / 2024-04-03T18:33:42.0624562Z line to your configuration file "/var/lib/jenkins/.gdbinit". 2024-04-03T18:33:42.0625364Z For more information about this security protection see the 2024-04-03T18:33:42.0626378Z "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: 2024-04-03T18:33:42.0627218Z info "(gdb)Auto-loading safe path" 2024-04-03T18:33:42.0627937Z #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 2024-04-03T18:33:42.0628741Z #1 0x00007f7693312859 in __GI_abort () at abort.c:79 2024-04-03T18:33:42.1742132Z #2 0x00007f766da695e9 in c10::MessageLogger::~MessageLogger() [clone .cold] 2024-04-03T18:33:42.1742828Z () 2024-04-03T18:33:42.1743644Z from /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so 2024-04-03T18:33:42.1756050Z #3 0x00007f766edb6874 in c10d::ProcessGroupNCCL::terminateProcess(std::__cxx11::basic_string, std::allocator >) () 2024-04-03T18:33:42.1757633Z from /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so 2024-04-03T18:33:42.1769324Z #4 0x00007f766edbcb13 in c10d::ProcessGroupNCCL::heartbeatMonitor() () 2024-04-03T18:33:42.1770933Z from /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so 2024-04-03T18:33:42.1861476Z #5 0x00007f768af81aeb in std::execute_native_thread_routine ( 2024-04-03T18:33:42.1862471Z __p=) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 2024-04-03T18:33:42.1863572Z #6 0x00007f7693644609 in start_thread (arg=) 2024-04-03T18:33:42.1864265Z at pthread_create.c:477 2024-04-03T18:33:42.1864642Z #7 0x00007f769340f353 in clone () 2024-04-03T18:33:42.1865199Z at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 2024-04-03T18:33:42.4235960Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-04-03T18:33:42.4236678Z with: 2024-04-03T18:33:42.4236995Z env: 2024-04-03T18:33:42.4237336Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:33:42.4237891Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:33:42.4238848Z DOCKER_CONTAINER_ID: b758c44c85badb76c4c382a8895cd5e1bfdf37df294e5f1babfcbd8fbafbdd0d 2024-04-03T18:33:42.4239691Z ##[endgroup] 2024-04-03T18:33:42.4265519Z ##[group]Run set -eou pipefail 2024-04-03T18:33:42.4266031Z set -eou pipefail 2024-04-03T18:33:42.4266468Z  2024-04-03T18:33:42.4267116Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-04-03T18:33:42.4267908Z for _ in $(seq 1440); do 2024-04-03T18:33:42.4268519Z  # Break if no ssh session exists anymore 2024-04-03T18:33:42.4269144Z  if [ "$(who)" = "" ]; then 2024-04-03T18:33:42.4269633Z  break 2024-04-03T18:33:42.4270077Z  fi 2024-04-03T18:33:42.4270435Z  echo "." 2024-04-03T18:33:42.4270839Z  sleep 5 2024-04-03T18:33:42.4271224Z done 2024-04-03T18:33:42.4279369Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:33:42.4280001Z env: 2024-04-03T18:33:42.4280339Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:33:42.4280889Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:33:42.4281843Z DOCKER_CONTAINER_ID: b758c44c85badb76c4c382a8895cd5e1bfdf37df294e5f1babfcbd8fbafbdd0d 2024-04-03T18:33:42.4282688Z ##[endgroup] 2024-04-03T18:33:42.4308935Z Holding runner for 2 hours until all ssh sessions have logged out 2024-04-03T18:33:42.4361686Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T18:33:42.4362671Z # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T18:33:42.4363420Z # shellcheck disable=SC2046 2024-04-03T18:33:42.4364007Z docker stop $(docker ps -q) || true 2024-04-03T18:33:42.4364593Z # Prune all of the docker images 2024-04-03T18:33:42.4365154Z docker system prune -af 2024-04-03T18:33:42.4373360Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T18:33:42.4373989Z env: 2024-04-03T18:33:42.4374315Z GIT_DEFAULT_BRANCH: main 2024-04-03T18:33:42.4374882Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-03T18:33:42.4375840Z DOCKER_CONTAINER_ID: b758c44c85badb76c4c382a8895cd5e1bfdf37df294e5f1babfcbd8fbafbdd0d 2024-04-03T18:33:42.4376666Z ##[endgroup] 2024-04-03T18:33:42.8246354Z b758c44c85ba 2024-04-03T18:33:43.3829566Z Deleted Containers: 2024-04-03T18:33:43.3830276Z b758c44c85badb76c4c382a8895cd5e1bfdf37df294e5f1babfcbd8fbafbdd0d 2024-04-03T18:33:43.3831145Z 2024-04-03T18:33:50.1048626Z Deleted Images: 2024-04-03T18:33:50.1050618Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T18:33:50.1053189Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9@sha256:68e0ca2e38253fb3bd10c84d380560df0602da1b16bccdfd7d4dd7ac22529759 2024-04-03T18:33:50.1055016Z deleted: sha256:f0aa5113e491b8f8ef3b51fc066bce798422ecc2bdeb4be54a8a4efa6914052e 2024-04-03T18:33:50.1056132Z deleted: sha256:c454db9d5e4e4efa0edf7c73a701618617c23c0f03a17d5e9c5f49ccbbfa0e73 2024-04-03T18:33:50.1057222Z deleted: sha256:b2dd02706fb0cdd8a4420189b04471a7739f76c16bb6c9e36747a62daaf28ad9 2024-04-03T18:33:50.1058301Z deleted: sha256:3e52b95dec2563c40922e1bfd58e5ec9813906f6958f9e481f3dad044b00c734 2024-04-03T18:33:50.1059364Z deleted: sha256:eeeb44278fe18f54afa2e7db8fd0210ec20683601770a2353cb53c27f7a1b5bc 2024-04-03T18:33:50.1060463Z deleted: sha256:3de3ee3c1c2cccb5001455ea6adcb75600111cc3fd52f8f80c11c495d6ee4e57 2024-04-03T18:33:50.1061597Z deleted: sha256:d6bf411cddfbc13bf76f9472a8b37a35ba76ffdccd5a2e21b14829368e18d3f7 2024-04-03T18:33:50.1062958Z deleted: sha256:8613c4bc44191f5e1cc622290ed8b516510d1ab876c25ccd377c36f51c2773f5 2024-04-03T18:33:50.1064082Z deleted: sha256:c3b2fbe550b9b298db4f89162757d368522e9cfede30c563e85dbf4440beb582 2024-04-03T18:33:50.1065170Z deleted: sha256:7a2ba3e1861525f15dcbdea7d5f2b39e6d14cc6a1a4d3a4669496c7a5c3a2d75 2024-04-03T18:33:50.1067056Z deleted: sha256:09e9c5c1b4f18382df5cddefe5c9b0dc27e5679be209c1e8622765d97df05bd2 2024-04-03T18:33:50.1069213Z deleted: sha256:3afff7cf2c94a194113068301442afd3f5a1b5c921841f28093ed5947ff17bf9 2024-04-03T18:33:50.1070340Z deleted: sha256:01421ca8d6d1e66f01d50067086730b4100f9a1966a2266332540cfd2fa67288 2024-04-03T18:33:50.1071397Z deleted: sha256:9d8429c72401fd7217175b98df195cdbe1dd4ff878348104c6a0be2c8fe87f07 2024-04-03T18:33:50.1072481Z deleted: sha256:b0ab639903fffd49ad6bda7fdf2fcd209dcbfce6f84ef28b9dac3dae321bb025 2024-04-03T18:33:50.1073614Z deleted: sha256:e1e2ba6eb5c7694469bb6fbd6820df749e354c265ab0b261960dec6a7a3a28be 2024-04-03T18:33:50.1074699Z deleted: sha256:9722a85cb366e43d512471e05004f0bba6c5a2c2dd592ba038ca03d96e348d92 2024-04-03T18:33:50.1075763Z deleted: sha256:59105bb9683ece65c0c62952242c8f582486ef48e62af263bdadbc6f61608fad 2024-04-03T18:33:50.1076805Z deleted: sha256:053162ab6c481eb96b5f661978e688dcc6bee984e7ec68e2d6ce6296bc891738 2024-04-03T18:33:50.1077876Z deleted: sha256:aa95ee7dea81036a66f9398d7a92020ea53c0de98fea9690a848c4887242c204 2024-04-03T18:33:50.1078954Z deleted: sha256:b00f90cd5628b07e6c16d09e2bdd195897085b1fd7c583800717b13fdb2bddba 2024-04-03T18:33:50.1080003Z deleted: sha256:d8764b2eab188f5b3c2c6573294c0e7d25cabe9e2206b51a1089bee5fdc9a099 2024-04-03T18:33:50.1081078Z deleted: sha256:2f02a119b01d7d4f2da3ae04e62eb82c6862630ad151910586b7d5fd54e3dd20 2024-04-03T18:33:50.1082154Z deleted: sha256:bb8ac4a8069a057d8ce651918a3f3a37f69cf070660ae23904dc9cd9675da101 2024-04-03T18:33:50.1083227Z deleted: sha256:20129be101ad623cea8a076f51abe873d22e6a1b9182dcffa89aefda79b6a7ec 2024-04-03T18:33:50.1084318Z deleted: sha256:ddc263e2d5206bdf8e3face17284fd07e61bedefc7820f3d15e43754688f56f6 2024-04-03T18:33:50.1085400Z deleted: sha256:331c5b9483ad593f5aa15565c0c1f483b7827a1ef011a7c0d9d40658f7e43a11 2024-04-03T18:33:50.1086473Z deleted: sha256:e2746ff342dbe1a36a1388a0bb33bc12e69b1627af1a7a13ffa6231db1645bd3 2024-04-03T18:33:50.1087547Z deleted: sha256:d2af2292b7bf3bd83e33811bdfceb98e128b901af3f13071e17ca8bb9a852a6f 2024-04-03T18:33:50.1089422Z deleted: sha256:a2ec7109ed947976900ebfb586b3d4e2552f6af6ce04fa04adda236eee7550ef 2024-04-03T18:33:50.1091392Z deleted: sha256:6b40bbee3bc6087e5c14ba5d43abde7796d62653a46181c6f87aa635f9ade9ac 2024-04-03T18:33:50.1092474Z deleted: sha256:377d27777d5b7ccd2c4a239e28d1d2ced864e35163e3ffc9131aee6b50ebfd44 2024-04-03T18:33:50.1093547Z deleted: sha256:139d166e407995d29a3d1c469c6af0eccb96895ec95721e668d7d1a96f1601e6 2024-04-03T18:33:50.1094938Z deleted: sha256:9e29d5432f1d387203719074a0d079463fc945d166c6fe9168342acd94d25565 2024-04-03T18:33:50.1096046Z deleted: sha256:7c66ee17d9810d8b654318ddadf226dc224d7ca4adbb64287aa4e1d378369d40 2024-04-03T18:33:50.1097225Z deleted: sha256:ca79b52f3eae23be700ee8aedcb485040d1918aacb1dcac027ff07a5e30d49e5 2024-04-03T18:33:50.1098385Z deleted: sha256:7fd608199fde74f258f5ff0cdb5708bbbb269c40c238f0941357570ceb3c0f97 2024-04-03T18:33:50.1099522Z deleted: sha256:953c2071c9b50052113d1dcbf64eaf51cd11e8dfa56e581b412bc075a784fb30 2024-04-03T18:33:50.1100631Z deleted: sha256:e397941ce9440cfbb931d0384718d8689325a0bca0c603e02412c5d28dc91891 2024-04-03T18:33:50.1101748Z deleted: sha256:a2f0692c36460bc69d0253b25333b5b17eb59f9f0cbc5325329b3823ac81025d 2024-04-03T18:33:50.1102860Z deleted: sha256:53a2bb00ca64b3972528438023e17333f36c3b85db48605df0f1111c08b6a91c 2024-04-03T18:33:50.1103939Z deleted: sha256:350185224c8f26322214dfdf6f1f72776da8088d3168d6f83723c98445e43f10 2024-04-03T18:33:50.1105044Z deleted: sha256:76385dca1ebd668f08aad474882b95d21679544054579c76ad79776a75b83e22 2024-04-03T18:33:50.1106185Z deleted: sha256:b11bcb556f21ebd78f10a9125d7f534ea59f86e81a006d815962a83602b0e662 2024-04-03T18:33:50.1107469Z deleted: sha256:7c6e21fb86aabd7cccb7531f8e7f4fa39607bf4136679b174638238ba2036339 2024-04-03T18:33:50.1108654Z deleted: sha256:49b69334db1af3decf2111f3474f9a7a76aeadafd34b647465bf35baf4496632 2024-04-03T18:33:50.1109793Z deleted: sha256:d6830305dc4ef0d8627ddc0fe0f1e49f5304333b7151a30b2aed904945e6a68f 2024-04-03T18:33:50.1110945Z deleted: sha256:21e3145f7deb8e4c8e4c359f495779f0b2a6ffd8a54ad879ec0bb6bcdbed9001 2024-04-03T18:33:50.1112057Z deleted: sha256:1c978ef9d890a352bfc23641fa000d332f042527032227c9715f297ce15a9d3a 2024-04-03T18:33:50.1113171Z deleted: sha256:5e2571ab32a3976828a3cf45d33bf651d60949687f8a81f3970c51c90a5a4a1d 2024-04-03T18:33:50.1114275Z deleted: sha256:547e1709b8e6365cc024934a2d46fe0b4029f4195b372a5ca63522747b09f9f5 2024-04-03T18:33:50.1115365Z deleted: sha256:d6f91263e804bd96920276a41b0538e01c33bf03b3ab32c4175588b5a9b5c217 2024-04-03T18:33:50.1116484Z deleted: sha256:ac317a820242b9e6c255f8065538f07519c66bbb5b40f313434cccb99ed34e56 2024-04-03T18:33:50.1117632Z deleted: sha256:6787bc9a8b82b4efce75e9e6d725ce95ff1e03a4f53768f96d420c5c0d5904d4 2024-04-03T18:33:50.1118765Z deleted: sha256:6a19981c4f76113029dbbcc0d87cdf2d324ec8d370b70ffb9d64623c804cfc3e 2024-04-03T18:33:50.1119908Z deleted: sha256:90ee54386e1cfb202860b4f78bfa7bb0db9722c0b8cc055c9f578130f09363d3 2024-04-03T18:33:50.1121036Z deleted: sha256:e039079492e83156c78b02baa6b4a38218c1c1529f7aec4dece116017320584d 2024-04-03T18:33:50.1122162Z deleted: sha256:488705e52ca1ed434cf0c3ee439eca1262115a69eb7492de82b7610234e2f6c5 2024-04-03T18:33:50.1123272Z deleted: sha256:5c567c50a43b1644e3211ef0e28dc73aa22a25d12bb1346308746375c9efbcfd 2024-04-03T18:33:50.1124393Z deleted: sha256:7c59d78b7f784e56d846633c9da755128c52d841ac70c649169b80efc9861e94 2024-04-03T18:33:50.1125504Z deleted: sha256:0702e7a9df3e7459b15b0f3d87c1029524539618e4676908b5934b923ed199ea 2024-04-03T18:33:50.1126602Z deleted: sha256:57a6f7d803492e9f768503b021ed8248e4d78d575185dd1938e57d18245667ad 2024-04-03T18:33:50.1127712Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-04-03T18:33:50.1128865Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-04-03T18:33:50.1130407Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-04-03T18:33:50.1131489Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-04-03T18:33:50.1132621Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-04-03T18:33:50.1133724Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-04-03T18:33:50.1134796Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-04-03T18:33:50.1135857Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-04-03T18:33:50.1136933Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-04-03T18:33:50.1138174Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-04-03T18:33:50.1139241Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-04-03T18:33:50.1139889Z 2024-04-03T18:33:50.1140083Z Total reclaimed space: 25.63GB 2024-04-03T18:33:50.1241208Z Post job cleanup. 2024-04-03T18:33:50.1308764Z Post job cleanup. 2024-04-03T18:33:50.2491623Z [command]/usr/bin/git version 2024-04-03T18:33:50.2545371Z git version 2.40.1 2024-04-03T18:33:50.2608386Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/36af015d-edbf-4098-96dd-0b68abe8bc3f' before making global git config changes 2024-04-03T18:33:50.2610293Z Adding repository directory to the temporary git global config as a safe directory 2024-04-03T18:33:50.2616349Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T18:33:50.2658624Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-04-03T18:33:50.2696133Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-04-03T18:33:50.3020853Z Entering 'android/libs/fbjni' 2024-04-03T18:33:50.3073458Z Entering 'third_party/FP16' 2024-04-03T18:33:50.3124832Z Entering 'third_party/FXdiv' 2024-04-03T18:33:50.3177266Z Entering 'third_party/NNPACK' 2024-04-03T18:33:50.3229707Z Entering 'third_party/QNNPACK' 2024-04-03T18:33:50.3280880Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T18:33:50.3332713Z Entering 'third_party/XNNPACK' 2024-04-03T18:33:50.3407086Z Entering 'third_party/benchmark' 2024-04-03T18:33:50.3457871Z Entering 'third_party/cpuinfo' 2024-04-03T18:33:50.3510957Z Entering 'third_party/cub' 2024-04-03T18:33:50.3562706Z Entering 'third_party/cudnn_frontend' 2024-04-03T18:33:50.3615587Z Entering 'third_party/cutlass' 2024-04-03T18:33:50.3677635Z Entering 'third_party/eigen' 2024-04-03T18:33:50.3734010Z Entering 'third_party/fbgemm' 2024-04-03T18:33:50.3785155Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T18:33:50.3836074Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T18:33:50.3887486Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T18:33:50.3945840Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T18:33:50.3995860Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T18:33:50.4047692Z Entering 'third_party/flatbuffers' 2024-04-03T18:33:50.4103871Z Entering 'third_party/fmt' 2024-04-03T18:33:50.4157577Z Entering 'third_party/foxi' 2024-04-03T18:33:50.4209808Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T18:33:50.4262240Z Entering 'third_party/gloo' 2024-04-03T18:33:50.4313754Z Entering 'third_party/googletest' 2024-04-03T18:33:50.4366054Z Entering 'third_party/ideep' 2024-04-03T18:33:50.4416512Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T18:33:50.4476746Z Entering 'third_party/ios-cmake' 2024-04-03T18:33:50.4530293Z Entering 'third_party/ittapi' 2024-04-03T18:33:50.4580693Z Entering 'third_party/kineto' 2024-04-03T18:33:50.4648801Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T18:33:50.4683246Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T18:33:50.4737796Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T18:33:50.4789802Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T18:33:50.4842625Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T18:33:50.4893366Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T18:33:50.4946690Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T18:33:50.4997790Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T18:33:50.5050485Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T18:33:50.5103152Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T18:33:50.5157044Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T18:33:50.5207877Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T18:33:50.5260852Z Entering 'third_party/mimalloc' 2024-04-03T18:33:50.5314761Z Entering 'third_party/nccl/nccl' 2024-04-03T18:33:50.5368252Z Entering 'third_party/neon2sse' 2024-04-03T18:33:50.5419971Z Entering 'third_party/nlohmann' 2024-04-03T18:33:50.5472994Z Entering 'third_party/onnx' 2024-04-03T18:33:50.5543097Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T18:33:50.5594217Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T18:33:50.5649121Z Entering 'third_party/onnx-tensorrt' 2024-04-03T18:33:50.5702123Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T18:33:50.5757451Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T18:33:50.5809504Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T18:33:50.5860369Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T18:33:50.5915968Z Entering 'third_party/pocketfft' 2024-04-03T18:33:50.5968468Z Entering 'third_party/protobuf' 2024-04-03T18:33:50.6023550Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T18:33:50.6074469Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T18:33:50.6126620Z Entering 'third_party/psimd' 2024-04-03T18:33:50.6178764Z Entering 'third_party/pthreadpool' 2024-04-03T18:33:50.6229706Z Entering 'third_party/pybind11' 2024-04-03T18:33:50.6281682Z Entering 'third_party/python-peachpy' 2024-04-03T18:33:50.6333927Z Entering 'third_party/sleef' 2024-04-03T18:33:50.6384760Z Entering 'third_party/tbb' 2024-04-03T18:33:50.6438610Z Entering 'third_party/tensorpipe' 2024-04-03T18:33:50.6489778Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T18:33:50.6540521Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T18:33:50.6590432Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T18:33:50.6640629Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T18:33:50.6690790Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T18:33:50.6744078Z Entering 'third_party/zstd' 2024-04-03T18:33:50.6815825Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-04-03T18:33:50.6843111Z http.https://github.com/.extraheader 2024-04-03T18:33:50.6855698Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-04-03T18:33:50.6893180Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-04-03T18:33:50.7213341Z Entering 'android/libs/fbjni' 2024-04-03T18:33:50.7246672Z http.https://github.com/.extraheader 2024-04-03T18:33:50.7282313Z Entering 'third_party/FP16' 2024-04-03T18:33:50.7315964Z http.https://github.com/.extraheader 2024-04-03T18:33:50.7351277Z Entering 'third_party/FXdiv' 2024-04-03T18:33:50.7385284Z http.https://github.com/.extraheader 2024-04-03T18:33:50.7419340Z Entering 'third_party/NNPACK' 2024-04-03T18:33:50.7452168Z http.https://github.com/.extraheader 2024-04-03T18:33:50.7488498Z Entering 'third_party/QNNPACK' 2024-04-03T18:33:50.7521610Z http.https://github.com/.extraheader 2024-04-03T18:33:50.7557326Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T18:33:50.7590234Z http.https://github.com/.extraheader 2024-04-03T18:33:50.7624830Z Entering 'third_party/XNNPACK' 2024-04-03T18:33:50.7658026Z http.https://github.com/.extraheader 2024-04-03T18:33:50.7712288Z Entering 'third_party/benchmark' 2024-04-03T18:33:50.7745284Z http.https://github.com/.extraheader 2024-04-03T18:33:50.7781288Z Entering 'third_party/cpuinfo' 2024-04-03T18:33:50.7814133Z http.https://github.com/.extraheader 2024-04-03T18:33:50.7848841Z Entering 'third_party/cub' 2024-04-03T18:33:50.7881237Z http.https://github.com/.extraheader 2024-04-03T18:33:50.7916994Z Entering 'third_party/cudnn_frontend' 2024-04-03T18:33:50.7951633Z http.https://github.com/.extraheader 2024-04-03T18:33:50.7985285Z Entering 'third_party/cutlass' 2024-04-03T18:33:50.8018681Z http.https://github.com/.extraheader 2024-04-03T18:33:50.8062796Z Entering 'third_party/eigen' 2024-04-03T18:33:50.8095872Z http.https://github.com/.extraheader 2024-04-03T18:33:50.8134894Z Entering 'third_party/fbgemm' 2024-04-03T18:33:50.8168675Z http.https://github.com/.extraheader 2024-04-03T18:33:50.8203773Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T18:33:50.8236305Z http.https://github.com/.extraheader 2024-04-03T18:33:50.8271034Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T18:33:50.8303514Z http.https://github.com/.extraheader 2024-04-03T18:33:50.8338744Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T18:33:50.8370083Z http.https://github.com/.extraheader 2024-04-03T18:33:50.8412894Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T18:33:50.8444648Z http.https://github.com/.extraheader 2024-04-03T18:33:50.8479482Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T18:33:50.8511602Z http.https://github.com/.extraheader 2024-04-03T18:33:50.8546744Z Entering 'third_party/flatbuffers' 2024-04-03T18:33:50.8579786Z http.https://github.com/.extraheader 2024-04-03T18:33:50.8617610Z Entering 'third_party/fmt' 2024-04-03T18:33:50.8649492Z http.https://github.com/.extraheader 2024-04-03T18:33:50.8683946Z Entering 'third_party/foxi' 2024-04-03T18:33:50.8717000Z http.https://github.com/.extraheader 2024-04-03T18:33:50.8750758Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T18:33:50.8784933Z http.https://github.com/.extraheader 2024-04-03T18:33:50.8819494Z Entering 'third_party/gloo' 2024-04-03T18:33:50.8852250Z http.https://github.com/.extraheader 2024-04-03T18:33:50.8886421Z Entering 'third_party/googletest' 2024-04-03T18:33:50.8920014Z http.https://github.com/.extraheader 2024-04-03T18:33:50.8954940Z Entering 'third_party/ideep' 2024-04-03T18:33:50.8988317Z http.https://github.com/.extraheader 2024-04-03T18:33:50.9021035Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T18:33:50.9054146Z http.https://github.com/.extraheader 2024-04-03T18:33:50.9096220Z Entering 'third_party/ios-cmake' 2024-04-03T18:33:50.9129342Z http.https://github.com/.extraheader 2024-04-03T18:33:50.9163294Z Entering 'third_party/ittapi' 2024-04-03T18:33:50.9197525Z http.https://github.com/.extraheader 2024-04-03T18:33:50.9232861Z Entering 'third_party/kineto' 2024-04-03T18:33:50.9266412Z http.https://github.com/.extraheader 2024-04-03T18:33:50.9300599Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T18:33:50.9333583Z http.https://github.com/.extraheader 2024-04-03T18:33:50.9369223Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T18:33:50.9401564Z http.https://github.com/.extraheader 2024-04-03T18:33:50.9437876Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T18:33:50.9470633Z http.https://github.com/.extraheader 2024-04-03T18:33:50.9505348Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T18:33:50.9537111Z http.https://github.com/.extraheader 2024-04-03T18:33:50.9571533Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T18:33:50.9602522Z http.https://github.com/.extraheader 2024-04-03T18:33:50.9636180Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T18:33:50.9669268Z http.https://github.com/.extraheader 2024-04-03T18:33:50.9705688Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T18:33:50.9739049Z http.https://github.com/.extraheader 2024-04-03T18:33:50.9773958Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T18:33:50.9806090Z http.https://github.com/.extraheader 2024-04-03T18:33:50.9841042Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T18:33:50.9873732Z http.https://github.com/.extraheader 2024-04-03T18:33:50.9910503Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T18:33:50.9942017Z http.https://github.com/.extraheader 2024-04-03T18:33:50.9978185Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T18:33:51.0010471Z http.https://github.com/.extraheader 2024-04-03T18:33:51.0046338Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T18:33:51.0078831Z http.https://github.com/.extraheader 2024-04-03T18:33:51.0115464Z Entering 'third_party/mimalloc' 2024-04-03T18:33:51.0148809Z http.https://github.com/.extraheader 2024-04-03T18:33:51.0184490Z Entering 'third_party/nccl/nccl' 2024-04-03T18:33:51.0218458Z http.https://github.com/.extraheader 2024-04-03T18:33:51.0253906Z Entering 'third_party/neon2sse' 2024-04-03T18:33:51.0287032Z http.https://github.com/.extraheader 2024-04-03T18:33:51.0321858Z Entering 'third_party/nlohmann' 2024-04-03T18:33:51.0355147Z http.https://github.com/.extraheader 2024-04-03T18:33:51.0391137Z Entering 'third_party/onnx' 2024-04-03T18:33:51.0423680Z http.https://github.com/.extraheader 2024-04-03T18:33:51.0476599Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T18:33:51.0509560Z http.https://github.com/.extraheader 2024-04-03T18:33:51.0543481Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T18:33:51.0576027Z http.https://github.com/.extraheader 2024-04-03T18:33:51.0612314Z Entering 'third_party/onnx-tensorrt' 2024-04-03T18:33:51.0645397Z http.https://github.com/.extraheader 2024-04-03T18:33:51.0679191Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T18:33:51.0712131Z http.https://github.com/.extraheader 2024-04-03T18:33:51.0750827Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T18:33:51.0783670Z http.https://github.com/.extraheader 2024-04-03T18:33:51.0818357Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T18:33:51.0851116Z http.https://github.com/.extraheader 2024-04-03T18:33:51.0884195Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T18:33:51.0916667Z http.https://github.com/.extraheader 2024-04-03T18:33:51.0956439Z Entering 'third_party/pocketfft' 2024-04-03T18:33:51.0989445Z http.https://github.com/.extraheader 2024-04-03T18:33:51.1023684Z Entering 'third_party/protobuf' 2024-04-03T18:33:51.1056722Z http.https://github.com/.extraheader 2024-04-03T18:33:51.1095563Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T18:33:51.1127659Z http.https://github.com/.extraheader 2024-04-03T18:33:51.1162722Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T18:33:51.1195101Z http.https://github.com/.extraheader 2024-04-03T18:33:51.1231249Z Entering 'third_party/psimd' 2024-04-03T18:33:51.1265328Z http.https://github.com/.extraheader 2024-04-03T18:33:51.1299512Z Entering 'third_party/pthreadpool' 2024-04-03T18:33:51.1332992Z http.https://github.com/.extraheader 2024-04-03T18:33:51.1366890Z Entering 'third_party/pybind11' 2024-04-03T18:33:51.1399884Z http.https://github.com/.extraheader 2024-04-03T18:33:51.1434632Z Entering 'third_party/python-peachpy' 2024-04-03T18:33:51.1467733Z http.https://github.com/.extraheader 2024-04-03T18:33:51.1503148Z Entering 'third_party/sleef' 2024-04-03T18:33:51.1535976Z http.https://github.com/.extraheader 2024-04-03T18:33:51.1570240Z Entering 'third_party/tbb' 2024-04-03T18:33:51.1602629Z http.https://github.com/.extraheader 2024-04-03T18:33:51.1640129Z Entering 'third_party/tensorpipe' 2024-04-03T18:33:51.1672913Z http.https://github.com/.extraheader 2024-04-03T18:33:51.1707070Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T18:33:51.1739541Z http.https://github.com/.extraheader 2024-04-03T18:33:51.1773802Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T18:33:51.1805445Z http.https://github.com/.extraheader 2024-04-03T18:33:51.1839860Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T18:33:51.1872200Z http.https://github.com/.extraheader 2024-04-03T18:33:51.1906664Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T18:33:51.1938471Z http.https://github.com/.extraheader 2024-04-03T18:33:51.1971663Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T18:33:51.2003091Z http.https://github.com/.extraheader 2024-04-03T18:33:51.2041254Z Entering 'third_party/zstd' 2024-04-03T18:33:51.2074288Z http.https://github.com/.extraheader 2024-04-03T18:33:51.2486702Z Cleaning up orphan processes