2024-12-17T23:58:02.8746208Z Current runner version: '2.321.0' 2024-12-17T23:58:02.8754110Z Runner name: 'i-0d4c5ae1b54bc839d' 2024-12-17T23:58:02.8755213Z Runner group name: 'Default' 2024-12-17T23:58:02.8756241Z Machine name: 'ip-10-0-16-113' 2024-12-17T23:58:02.8761570Z ##[group]GITHUB_TOKEN Permissions 2024-12-17T23:58:02.8764267Z Actions: read 2024-12-17T23:58:02.8764899Z Attestations: read 2024-12-17T23:58:02.8765561Z Checks: read 2024-12-17T23:58:02.8766232Z Contents: read 2024-12-17T23:58:02.8766868Z Deployments: read 2024-12-17T23:58:02.8767504Z Discussions: read 2024-12-17T23:58:02.8768167Z Issues: read 2024-12-17T23:58:02.8768790Z Metadata: read 2024-12-17T23:58:02.8769407Z Packages: read 2024-12-17T23:58:02.8770114Z Pages: read 2024-12-17T23:58:02.8770755Z PullRequests: read 2024-12-17T23:58:02.8771361Z RepositoryProjects: read 2024-12-17T23:58:02.8772172Z SecurityEvents: read 2024-12-17T23:58:02.8772851Z Statuses: read 2024-12-17T23:58:02.8773505Z ##[endgroup] 2024-12-17T23:58:02.8776991Z Secret source: Actions 2024-12-17T23:58:02.8778165Z Prepare workflow directory 2024-12-17T23:58:02.9306830Z Prepare all required actions 2024-12-17T23:58:02.9349097Z Getting action download info 2024-12-17T23:58:03.3305602Z Download action repository 'pytorch/test-infra@release/2.6' (SHA:eb0adf5a84668865394af69e26428b32c8105c1c) 2024-12-17T23:58:04.7901808Z Download action repository 'pytorch/pytorch@release/2.6' (SHA:0cdf8b1d09254cfda66191d1bd01e3041c3c76f7) 2024-12-17T23:58:17.6381234Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-12-17T23:58:17.8516772Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-12-17T23:58:18.1454310Z Getting action download info 2024-12-17T23:58:18.2696446Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-12-17T23:58:18.5724528Z Getting action download info 2024-12-17T23:58:18.7139462Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2024-12-17T23:58:18.9025810Z Getting action download info 2024-12-17T23:58:19.0434856Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-12-17T23:58:19.2120290Z Getting action download info 2024-12-17T23:58:19.3392394Z Download action repository 'pytorch/test-infra@main' (SHA:a07505a74641a4ff5123d635defac481ef28ef1e) 2024-12-17T23:58:20.7616057Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/release/2.6 (0cdf8b1d09254cfda66191d1bd01e3041c3c76f7) 2024-12-17T23:58:20.7618282Z ##[group] Inputs 2024-12-17T23:58:20.7618710Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-12-17T23:58:20.7620550Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]} 2024-12-17T23:58:20.7622594Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-17T23:58:20.7623531Z sync-tag: 2024-12-17T23:58:20.7624452Z timeout-minutes: 360 2024-12-17T23:58:20.7624789Z use-gha: 2024-12-17T23:58:20.7625073Z dashboard-tag: 2024-12-17T23:58:20.7625381Z s3-bucket: gha-artifacts 2024-12-17T23:58:20.7625714Z aws-role-to-assume: 2024-12-17T23:58:20.7626387Z disable-monitor: false 2024-12-17T23:58:20.7626739Z ##[endgroup] 2024-12-17T23:58:20.7627354Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-17T23:58:20.8322478Z A job started hook has been configured by the self-hosted runner administrator 2024-12-17T23:58:20.8433494Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-12-17T23:58:20.8443870Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-17T23:58:20.8444712Z ##[endgroup] 2024-12-17T23:58:22.0217889Z Runner Type: linux.g4dn.12xlarge.nvidia.gpu 2024-12-17T23:58:22.0218422Z Instance Type: g4dn.12xlarge 2024-12-17T23:58:22.0218873Z AMI Name: unknown 2024-12-17T23:58:22.0247938Z AMI ID: ami-0fff1b9a61dec8a5f 2024-12-17T23:58:27.9096748Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@release/2.6 2024-12-17T23:58:27.9097262Z with: 2024-12-17T23:58:27.9097978Z github-secret: *** 2024-12-17T23:58:27.9098834Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-12-17T23:58:27.9099901Z activate-with-label: false 2024-12-17T23:58:27.9100253Z label: with-ssh 2024-12-17T23:58:27.9100565Z remove-existing-keys: true 2024-12-17T23:58:27.9100911Z fail-silently: true 2024-12-17T23:58:27.9101200Z env: 2024-12-17T23:58:27.9101466Z GIT_DEFAULT_BRANCH: main 2024-12-17T23:58:27.9101825Z ##[endgroup] 2024-12-17T23:58:28.0339816Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-12-17T23:58:28.0341498Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2024-12-17T23:58:28.0472596Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@release/2.6 2024-12-17T23:58:28.0473110Z with: 2024-12-17T23:58:28.0473339Z no-sudo: true 2024-12-17T23:58:28.0473615Z submodules: recursive 2024-12-17T23:58:28.0473900Z fetch-depth: 0 2024-12-17T23:58:28.0474153Z env: 2024-12-17T23:58:28.0474375Z GIT_DEFAULT_BRANCH: main 2024-12-17T23:58:28.0474670Z ##[endgroup] 2024-12-17T23:58:28.0556630Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-17T23:58:28.0557707Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-17T23:58:28.0567226Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-17T23:58:28.0567649Z env: 2024-12-17T23:58:28.0567913Z GIT_DEFAULT_BRANCH: main 2024-12-17T23:58:28.0568252Z ##[endgroup] 2024-12-17T23:58:28.0665457Z ##[group]Run retry () { 2024-12-17T23:58:28.0665825Z retry () { 2024-12-17T23:58:28.0666230Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-12-17T23:58:28.0666707Z } 2024-12-17T23:58:28.0667011Z echo "${GITHUB_WORKSPACE}" 2024-12-17T23:58:28.0667415Z if [ -z "${NO_SUDO}" ]; then 2024-12-17T23:58:28.0667919Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-12-17T23:58:28.0668354Z else 2024-12-17T23:58:28.0668663Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-12-17T23:58:28.0669060Z fi 2024-12-17T23:58:28.0669359Z mkdir "${GITHUB_WORKSPACE}" 2024-12-17T23:58:28.0675388Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-17T23:58:28.0675821Z env: 2024-12-17T23:58:28.0676056Z GIT_DEFAULT_BRANCH: main 2024-12-17T23:58:28.0676351Z NO_SUDO: true 2024-12-17T23:58:28.0676610Z ##[endgroup] 2024-12-17T23:58:28.0700486Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-17T23:58:28.0790072Z ##[group]Run malfet/checkout@silent-checkout 2024-12-17T23:58:28.0790445Z with: 2024-12-17T23:58:28.0790723Z ref: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-17T23:58:28.0791090Z fetch-depth: 0 2024-12-17T23:58:28.0791362Z submodules: recursive 2024-12-17T23:58:28.0791635Z quiet-checkout: true 2024-12-17T23:58:28.0791929Z repository: pytorch/pytorch 2024-12-17T23:58:28.0792353Z token: *** 2024-12-17T23:58:28.0792607Z ssh-strict: true 2024-12-17T23:58:28.0792886Z persist-credentials: true 2024-12-17T23:58:28.0793190Z clean: true 2024-12-17T23:58:28.0793452Z sparse-checkout-cone-mode: true 2024-12-17T23:58:28.0793773Z lfs: false 2024-12-17T23:58:28.0794031Z set-safe-directory: true 2024-12-17T23:58:28.0794504Z env: 2024-12-17T23:58:28.0794733Z GIT_DEFAULT_BRANCH: main 2024-12-17T23:58:28.0795024Z ##[endgroup] 2024-12-17T23:58:28.1681176Z Syncing repository: pytorch/pytorch 2024-12-17T23:58:28.1682843Z ##[group]Getting Git version info 2024-12-17T23:58:28.1683397Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-12-17T23:58:28.1684112Z [command]/usr/bin/git version 2024-12-17T23:58:28.1684431Z git version 2.40.1 2024-12-17T23:58:28.1696685Z ##[endgroup] 2024-12-17T23:58:28.1709800Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/da1358c3-b076-4e65-bbac-b5e22cb4d8b1' before making global git config changes 2024-12-17T23:58:28.1711069Z Adding repository directory to the temporary git global config as a safe directory 2024-12-17T23:58:28.1713805Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-17T23:58:28.1742429Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-12-17T23:58:28.1745359Z ##[group]Initializing the repository 2024-12-17T23:58:28.1748103Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-17T23:58:28.1773291Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-12-17T23:58:28.1774022Z hint: is subject to change. To configure the initial branch name to use in all 2024-12-17T23:58:28.1774694Z hint: of your new repositories, which will suppress this warning, call: 2024-12-17T23:58:28.1775179Z hint: 2024-12-17T23:58:28.1775536Z hint: git config --global init.defaultBranch 2024-12-17T23:58:28.1775955Z hint: 2024-12-17T23:58:28.1776354Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-12-17T23:58:28.1777015Z hint: 'development'. The just-created branch can be renamed via this command: 2024-12-17T23:58:28.1777519Z hint: 2024-12-17T23:58:28.1777787Z hint: git branch -m 2024-12-17T23:58:28.1778388Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-12-17T23:58:28.1783257Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-12-17T23:58:28.1810046Z ##[endgroup] 2024-12-17T23:58:28.1810602Z ##[group]Disabling automatic garbage collection 2024-12-17T23:58:28.1811115Z [command]/usr/bin/git config --local gc.auto 0 2024-12-17T23:58:28.1835240Z ##[endgroup] 2024-12-17T23:58:28.1835713Z ##[group]Setting up auth 2024-12-17T23:58:28.1840206Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-12-17T23:58:28.1863801Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-12-17T23:58:28.2169739Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-12-17T23:58:28.2192855Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-12-17T23:58:28.2475663Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-12-17T23:58:28.2521143Z ##[endgroup] 2024-12-17T23:58:28.2525091Z ##[group]Fetching the repository 2024-12-17T23:58:28.2526068Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-12-17T23:58:31.1465678Z remote: Enumerating objects: 1056622 2024-12-17T23:58:31.1467010Z remote: Enumerating objects: 1057261, done. 2024-12-17T23:58:31.1467474Z remote: Counting objects: 0% (1/639) 2024-12-17T23:58:31.1467909Z remote: Counting objects: 1% (7/639) 2024-12-17T23:58:31.1468336Z remote: Counting objects: 2% (13/639) 2024-12-17T23:58:31.1468752Z remote: Counting objects: 3% (20/639) 2024-12-17T23:58:31.1469440Z remote: Counting objects: 4% (26/639) 2024-12-17T23:58:31.1469866Z remote: Counting objects: 5% (32/639) 2024-12-17T23:58:31.1470293Z remote: Counting objects: 6% (39/639) 2024-12-17T23:58:31.1470732Z remote: Counting objects: 7% (45/639) 2024-12-17T23:58:31.1471143Z remote: Counting objects: 8% (52/639) 2024-12-17T23:58:31.1471566Z remote: Counting objects: 9% (58/639) 2024-12-17T23:58:31.1471996Z remote: Counting objects: 10% (64/639) 2024-12-17T23:58:31.1472419Z remote: Counting objects: 11% (71/639) 2024-12-17T23:58:31.1472845Z remote: Counting objects: 12% (77/639) 2024-12-17T23:58:31.1473256Z remote: Counting objects: 13% (84/639) 2024-12-17T23:58:31.1473679Z remote: Counting objects: 14% (90/639) 2024-12-17T23:58:31.1474105Z remote: Counting objects: 15% (96/639) 2024-12-17T23:58:31.1474535Z remote: Counting objects: 16% (103/639) 2024-12-17T23:58:31.1474973Z remote: Counting objects: 17% (109/639) 2024-12-17T23:58:31.1475389Z remote: Counting objects: 18% (116/639) 2024-12-17T23:58:31.1475817Z remote: Counting objects: 19% (122/639) 2024-12-17T23:58:31.1476372Z remote: Counting objects: 20% (128/639) 2024-12-17T23:58:31.1476807Z remote: Counting objects: 21% (135/639) 2024-12-17T23:58:31.1477295Z remote: Counting objects: 22% (141/639) 2024-12-17T23:58:31.1477729Z remote: Counting objects: 23% (147/639) 2024-12-17T23:58:31.1478156Z remote: Counting objects: 24% (154/639) 2024-12-17T23:58:31.1478567Z remote: Counting objects: 25% (160/639) 2024-12-17T23:58:31.1479001Z remote: Counting objects: 26% (167/639) 2024-12-17T23:58:31.1479501Z remote: Counting objects: 27% (173/639) 2024-12-17T23:58:31.1479929Z remote: Counting objects: 28% (179/639) 2024-12-17T23:58:31.1480364Z remote: Counting objects: 29% (186/639) 2024-12-17T23:58:31.1480801Z remote: Counting objects: 30% (192/639) 2024-12-17T23:58:31.1481215Z remote: Counting objects: 31% (199/639) 2024-12-17T23:58:31.1481645Z remote: Counting objects: 32% (205/639) 2024-12-17T23:58:31.1518520Z remote: Counting objects: 33% (211/639) 2024-12-17T23:58:31.1519239Z remote: Counting objects: 34% (218/639) 2024-12-17T23:58:31.1519678Z remote: Counting objects: 35% (224/639) 2024-12-17T23:58:31.1520103Z remote: Counting objects: 36% (231/639) 2024-12-17T23:58:31.1520531Z remote: Counting objects: 37% (237/639) 2024-12-17T23:58:31.1520936Z remote: Counting objects: 38% (243/639) 2024-12-17T23:58:31.1521355Z remote: Counting objects: 39% (250/639) 2024-12-17T23:58:31.1521773Z remote: Counting objects: 40% (256/639) 2024-12-17T23:58:31.1522193Z remote: Counting objects: 41% (262/639) 2024-12-17T23:58:31.1522610Z remote: Counting objects: 42% (269/639) 2024-12-17T23:58:31.1523036Z remote: Counting objects: 43% (275/639) 2024-12-17T23:58:31.1523439Z remote: Counting objects: 44% (282/639) 2024-12-17T23:58:31.1523858Z remote: Counting objects: 45% (288/639) 2024-12-17T23:58:31.1524279Z remote: Counting objects: 46% (294/639) 2024-12-17T23:58:31.1524696Z remote: Counting objects: 47% (301/639) 2024-12-17T23:58:31.1525114Z remote: Counting objects: 48% (307/639) 2024-12-17T23:58:31.1525519Z remote: Counting objects: 49% (314/639) 2024-12-17T23:58:31.1525935Z remote: Counting objects: 50% (320/639) 2024-12-17T23:58:31.1526353Z remote: Counting objects: 51% (326/639) 2024-12-17T23:58:31.1526772Z remote: Counting objects: 52% (333/639) 2024-12-17T23:58:31.1527190Z remote: Counting objects: 53% (339/639) 2024-12-17T23:58:31.1527594Z remote: Counting objects: 54% (346/639) 2024-12-17T23:58:31.1528011Z remote: Counting objects: 55% (352/639) 2024-12-17T23:58:31.1528573Z remote: Counting objects: 56% (358/639) 2024-12-17T23:58:31.1528990Z remote: Counting objects: 57% (365/639) 2024-12-17T23:58:31.1529437Z remote: Counting objects: 58% (371/639) 2024-12-17T23:58:31.1529902Z remote: Counting objects: 59% (378/639) 2024-12-17T23:58:31.1530322Z remote: Counting objects: 60% (384/639) 2024-12-17T23:58:31.1530740Z remote: Counting objects: 61% (390/639) 2024-12-17T23:58:31.1531154Z remote: Counting objects: 62% (397/639) 2024-12-17T23:58:31.1531954Z remote: Counting objects: 63% (403/639) 2024-12-17T23:58:31.1532512Z remote: Counting objects: 64% (409/639) 2024-12-17T23:58:31.1533004Z remote: Counting objects: 65% (416/639) 2024-12-17T23:58:31.1533419Z remote: Counting objects: 66% (422/639) 2024-12-17T23:58:31.1533848Z remote: Counting objects: 67% (429/639) 2024-12-17T23:58:31.1534276Z remote: Counting objects: 68% (435/639) 2024-12-17T23:58:31.1534711Z remote: Counting objects: 69% (441/639) 2024-12-17T23:58:31.1535139Z remote: Counting objects: 70% (448/639) 2024-12-17T23:58:31.1535571Z remote: Counting objects: 71% (454/639) 2024-12-17T23:58:31.1535986Z remote: Counting objects: 72% (461/639) 2024-12-17T23:58:31.1536545Z remote: Counting objects: 73% (467/639) 2024-12-17T23:58:31.1536984Z remote: Counting objects: 74% (473/639) 2024-12-17T23:58:31.1537415Z remote: Counting objects: 75% (480/639) 2024-12-17T23:58:31.1537847Z remote: Counting objects: 76% (486/639) 2024-12-17T23:58:31.1538264Z remote: Counting objects: 77% (493/639) 2024-12-17T23:58:31.1538776Z remote: Counting objects: 78% (499/639) 2024-12-17T23:58:31.1539209Z remote: Counting objects: 79% (505/639) 2024-12-17T23:58:31.1539901Z remote: Counting objects: 80% (512/639) 2024-12-17T23:58:31.1540346Z remote: Counting objects: 81% (518/639) 2024-12-17T23:58:31.1540776Z remote: Counting objects: 82% (524/639) 2024-12-17T23:58:31.1541230Z remote: Counting objects: 83% (531/639) 2024-12-17T23:58:31.1541670Z remote: Counting objects: 84% (537/639) 2024-12-17T23:58:31.1542112Z remote: Counting objects: 85% (544/639) 2024-12-17T23:58:31.1542564Z remote: Counting objects: 86% (550/639) 2024-12-17T23:58:31.1542989Z remote: Counting objects: 87% (556/639) 2024-12-17T23:58:31.1543430Z remote: Counting objects: 88% (563/639) 2024-12-17T23:58:31.1543873Z remote: Counting objects: 89% (569/639) 2024-12-17T23:58:31.1544313Z remote: Counting objects: 90% (576/639) 2024-12-17T23:58:31.1544755Z remote: Counting objects: 91% (582/639) 2024-12-17T23:58:31.1545197Z remote: Counting objects: 92% (588/639) 2024-12-17T23:58:31.1545625Z remote: Counting objects: 93% (595/639) 2024-12-17T23:58:31.1546069Z remote: Counting objects: 94% (601/639) 2024-12-17T23:58:31.1546514Z remote: Counting objects: 95% (608/639) 2024-12-17T23:58:31.1546965Z remote: Counting objects: 96% (614/639) 2024-12-17T23:58:31.1547408Z remote: Counting objects: 97% (620/639) 2024-12-17T23:58:31.1547836Z remote: Counting objects: 98% (627/639) 2024-12-17T23:58:31.1548282Z remote: Counting objects: 99% (633/639) 2024-12-17T23:58:31.1548724Z remote: Counting objects: 100% (639/639) 2024-12-17T23:58:31.1549193Z remote: Counting objects: 100% (639/639), done. 2024-12-17T23:58:31.1549679Z remote: Compressing objects: 0% (1/297) 2024-12-17T23:58:31.1561437Z remote: Compressing objects: 1% (3/297) 2024-12-17T23:58:31.1692338Z remote: Compressing objects: 2% (6/297) 2024-12-17T23:58:31.1826266Z remote: Compressing objects: 3% (9/297) 2024-12-17T23:58:31.1858217Z remote: Compressing objects: 4% (12/297) 2024-12-17T23:58:31.1937675Z remote: Compressing objects: 5% (15/297) 2024-12-17T23:58:31.1991637Z remote: Compressing objects: 6% (18/297) 2024-12-17T23:58:31.2119075Z remote: Compressing objects: 7% (21/297) 2024-12-17T23:58:31.2393509Z remote: Compressing objects: 8% (24/297) 2024-12-17T23:58:31.2781136Z remote: Compressing objects: 9% (27/297) 2024-12-17T23:58:31.3491250Z remote: Compressing objects: 10% (30/297) 2024-12-17T23:58:31.3944814Z remote: Compressing objects: 11% (33/297) 2024-12-17T23:58:31.4139043Z remote: Compressing objects: 12% (36/297) 2024-12-17T23:58:31.4176450Z remote: Compressing objects: 13% (39/297) 2024-12-17T23:58:31.4177098Z remote: Compressing objects: 14% (42/297) 2024-12-17T23:58:31.4177608Z remote: Compressing objects: 15% (45/297) 2024-12-17T23:58:31.4180532Z remote: Compressing objects: 16% (48/297) 2024-12-17T23:58:31.4183067Z remote: Compressing objects: 17% (51/297) 2024-12-17T23:58:31.4185982Z remote: Compressing objects: 18% (54/297) 2024-12-17T23:58:31.4188511Z remote: Compressing objects: 19% (57/297) 2024-12-17T23:58:31.4191263Z remote: Compressing objects: 20% (60/297) 2024-12-17T23:58:31.4193728Z remote: Compressing objects: 21% (63/297) 2024-12-17T23:58:31.4194175Z remote: Compressing objects: 22% (66/297) 2024-12-17T23:58:31.4196412Z remote: Compressing objects: 23% (69/297) 2024-12-17T23:58:31.4198914Z remote: Compressing objects: 24% (72/297) 2024-12-17T23:58:31.4202958Z remote: Compressing objects: 25% (75/297) 2024-12-17T23:58:31.4205338Z remote: Compressing objects: 26% (78/297) 2024-12-17T23:58:31.4208181Z remote: Compressing objects: 27% (81/297) 2024-12-17T23:58:31.4208635Z remote: Compressing objects: 28% (84/297) 2024-12-17T23:58:31.4210497Z remote: Compressing objects: 29% (87/297) 2024-12-17T23:58:31.4213014Z remote: Compressing objects: 30% (90/297) 2024-12-17T23:58:31.4219990Z remote: Compressing objects: 31% (93/297) 2024-12-17T23:58:31.4229538Z remote: Compressing objects: 32% (96/297) 2024-12-17T23:58:31.4230135Z remote: Compressing objects: 33% (99/297) 2024-12-17T23:58:31.4232292Z remote: Compressing objects: 34% (101/297) 2024-12-17T23:58:31.4234500Z remote: Compressing objects: 35% (104/297) 2024-12-17T23:58:31.4240256Z remote: Compressing objects: 36% (107/297) 2024-12-17T23:58:31.4242872Z remote: Compressing objects: 37% (110/297) 2024-12-17T23:58:31.4246524Z remote: Compressing objects: 38% (113/297) 2024-12-17T23:58:31.4252234Z remote: Compressing objects: 39% (116/297) 2024-12-17T23:58:31.4257819Z remote: Compressing objects: 40% (119/297) 2024-12-17T23:58:31.4260282Z remote: Compressing objects: 41% (122/297) 2024-12-17T23:58:31.4262912Z remote: Compressing objects: 42% (125/297) 2024-12-17T23:58:31.4265568Z remote: Compressing objects: 43% (128/297) 2024-12-17T23:58:31.4268077Z remote: Compressing objects: 44% (131/297) 2024-12-17T23:58:31.4272926Z remote: Compressing objects: 45% (134/297) 2024-12-17T23:58:31.4275951Z remote: Compressing objects: 46% (137/297) 2024-12-17T23:58:31.4278478Z remote: Compressing objects: 47% (140/297) 2024-12-17T23:58:31.4282631Z remote: Compressing objects: 48% (143/297) 2024-12-17T23:58:31.4285738Z remote: Compressing objects: 49% (146/297) 2024-12-17T23:58:31.4288278Z remote: Compressing objects: 50% (149/297) 2024-12-17T23:58:31.4291552Z remote: Compressing objects: 51% (152/297) 2024-12-17T23:58:31.4293998Z remote: Compressing objects: 52% (155/297) 2024-12-17T23:58:31.4296580Z remote: Compressing objects: 53% (158/297) 2024-12-17T23:58:31.4299241Z remote: Compressing objects: 54% (161/297) 2024-12-17T23:58:31.4301737Z remote: Compressing objects: 55% (164/297) 2024-12-17T23:58:31.4302202Z remote: Compressing objects: 56% (167/297) 2024-12-17T23:58:31.4304291Z remote: Compressing objects: 57% (170/297) 2024-12-17T23:58:31.4304943Z remote: Compressing objects: 58% (173/297) 2024-12-17T23:58:31.4306822Z remote: Compressing objects: 59% (176/297) 2024-12-17T23:58:31.4309470Z remote: Compressing objects: 60% (179/297) 2024-12-17T23:58:31.4309938Z remote: Compressing objects: 61% (182/297) 2024-12-17T23:58:31.4311952Z remote: Compressing objects: 62% (185/297) 2024-12-17T23:58:31.4314460Z remote: Compressing objects: 63% (188/297) 2024-12-17T23:58:31.4317065Z remote: Compressing objects: 64% (191/297) 2024-12-17T23:58:31.4319856Z remote: Compressing objects: 65% (194/297) 2024-12-17T23:58:31.4320307Z remote: Compressing objects: 66% (197/297) 2024-12-17T23:58:31.4322361Z remote: Compressing objects: 67% (199/297) 2024-12-17T23:58:31.4325082Z remote: Compressing objects: 68% (202/297) 2024-12-17T23:58:31.4327870Z remote: Compressing objects: 69% (205/297) 2024-12-17T23:58:31.4330674Z remote: Compressing objects: 70% (208/297) 2024-12-17T23:58:31.4333332Z remote: Compressing objects: 71% (211/297) 2024-12-17T23:58:31.4335909Z remote: Compressing objects: 72% (214/297) 2024-12-17T23:58:31.4336488Z remote: Compressing objects: 73% (217/297) 2024-12-17T23:58:31.4337091Z remote: Compressing objects: 74% (220/297) 2024-12-17T23:58:31.4338551Z remote: Compressing objects: 75% (223/297) 2024-12-17T23:58:31.4339259Z remote: Compressing objects: 76% (226/297) 2024-12-17T23:58:31.4339966Z remote: Compressing objects: 77% (229/297) 2024-12-17T23:58:31.4340864Z remote: Compressing objects: 78% (232/297) 2024-12-17T23:58:31.4342356Z remote: Compressing objects: 79% (235/297) 2024-12-17T23:58:31.4343687Z remote: Compressing objects: 80% (238/297) 2024-12-17T23:58:31.4345709Z remote: Compressing objects: 81% (241/297) 2024-12-17T23:58:31.4347568Z remote: Compressing objects: 82% (244/297) 2024-12-17T23:58:31.4351214Z remote: Compressing objects: 83% (247/297) 2024-12-17T23:58:31.4351644Z remote: Compressing objects: 84% (250/297) 2024-12-17T23:58:31.4355466Z remote: Compressing objects: 85% (253/297) 2024-12-17T23:58:31.4356303Z remote: Compressing objects: 86% (256/297) 2024-12-17T23:58:31.4357123Z remote: Compressing objects: 87% (259/297) 2024-12-17T23:58:31.4357570Z remote: Compressing objects: 88% (262/297) 2024-12-17T23:58:31.4358322Z remote: Compressing objects: 89% (265/297) 2024-12-17T23:58:31.4358856Z remote: Compressing objects: 90% (268/297) 2024-12-17T23:58:31.4359579Z remote: Compressing objects: 91% (271/297) 2024-12-17T23:58:31.4360217Z remote: Compressing objects: 92% (274/297) 2024-12-17T23:58:31.4361043Z remote: Compressing objects: 93% (277/297) 2024-12-17T23:58:31.4361507Z remote: Compressing objects: 94% (280/297) 2024-12-17T23:58:31.4362310Z remote: Compressing objects: 95% (283/297) 2024-12-17T23:58:31.4362822Z remote: Compressing objects: 96% (286/297) 2024-12-17T23:58:31.4363621Z remote: Compressing objects: 97% (289/297) 2024-12-17T23:58:31.4364142Z remote: Compressing objects: 98% (292/297) 2024-12-17T23:58:31.4364591Z remote: Compressing objects: 99% (295/297) 2024-12-17T23:58:31.4365019Z remote: Compressing objects: 100% (297/297) 2024-12-17T23:58:31.4365499Z remote: Compressing objects: 100% (297/297), done. 2024-12-17T23:58:54.3383308Z remote: Total 1057261 (delta 491), reused 366 (delta 342), pack-reused 1056622 (from 5) 2024-12-17T23:59:11.1142073Z [command]/usr/bin/git rev-parse --verify --quiet 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7^{object} 2024-12-17T23:59:11.1163928Z 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-17T23:59:11.1166616Z ##[endgroup] 2024-12-17T23:59:11.1167132Z ##[group]Determining the checkout info 2024-12-17T23:59:11.1167945Z ##[endgroup] 2024-12-17T23:59:11.1168735Z ##[group]Checking out the ref 2024-12-17T23:59:11.1170626Z [command]/usr/bin/git checkout --quiet --force 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-17T23:59:12.6289312Z ##[endgroup] 2024-12-17T23:59:12.6289885Z ##[group]Setting up auth for fetching submodules 2024-12-17T23:59:12.6291561Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-12-17T23:59:12.6340013Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-12-17T23:59:12.6362664Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-12-17T23:59:12.6387391Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-12-17T23:59:12.6413152Z ##[endgroup] 2024-12-17T23:59:12.6413641Z ##[group]Fetching submodules 2024-12-17T23:59:12.6414053Z [command]/usr/bin/git submodule sync --recursive 2024-12-17T23:59:12.6725172Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-12-17T23:59:12.7025865Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-12-17T23:59:12.7027826Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-12-17T23:59:12.7029602Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-12-17T23:59:12.7032483Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-12-17T23:59:12.7035304Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2024-12-17T23:59:12.7038510Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-12-17T23:59:12.7041064Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-12-17T23:59:12.7044213Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-12-17T23:59:12.7047388Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2024-12-17T23:59:12.7050479Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-12-17T23:59:12.7053645Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-12-17T23:59:12.7057146Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-12-17T23:59:12.7060861Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-12-17T23:59:12.7064542Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-12-17T23:59:12.7068305Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-12-17T23:59:12.7073847Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-12-17T23:59:12.7079456Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-12-17T23:59:12.7083757Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-12-17T23:59:12.7087745Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-12-17T23:59:12.7092171Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-12-17T23:59:12.7096393Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-12-17T23:59:12.7101269Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-12-17T23:59:12.7105929Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-12-17T23:59:12.7110769Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-12-17T23:59:12.7115525Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-12-17T23:59:12.7120362Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-12-17T23:59:12.7125253Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-12-17T23:59:12.7130555Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-12-17T23:59:12.7136045Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-12-17T23:59:12.7141788Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-12-17T23:59:12.7147143Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-12-17T23:59:12.7152869Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-12-17T23:59:12.7158277Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-12-17T23:59:12.7165508Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-12-17T23:59:12.7171268Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-12-17T23:59:12.7177251Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-12-17T23:59:12.7208873Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-12-17T23:59:13.0342186Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-12-17T23:59:13.2333341Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-12-17T23:59:13.4714751Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-12-17T23:59:13.7334363Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2024-12-17T23:59:14.1096922Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-12-17T23:59:15.8532207Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-12-17T23:59:25.0769301Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-12-17T23:59:25.5384539Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2024-12-17T23:59:27.6099934Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-12-17T23:59:28.0822884Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-12-17T23:59:28.6807063Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-12-17T23:59:29.8618239Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-12-17T23:59:31.9216332Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-12-17T23:59:37.4010370Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-12-17T23:59:38.8188221Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-12-17T23:59:39.9222020Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-12-17T23:59:40.9929823Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-12-17T23:59:41.4123962Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-12-17T23:59:41.7462131Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-12-17T23:59:42.7300258Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-12-17T23:59:43.1188609Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-12-17T23:59:43.4288542Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-12-17T23:59:45.2020191Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-12-17T23:59:46.1312081Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-12-17T23:59:46.4947182Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-12-17T23:59:52.9897264Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-12-17T23:59:54.8283108Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-12-17T23:59:58.3739176Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-12-17T23:59:58.6173962Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-12-18T00:00:06.8490270Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-12-18T00:00:07.0535611Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-12-18T00:00:07.5308934Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-12-18T00:00:08.5290332Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-12-18T00:00:08.8530630Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-12-18T00:00:09.5583671Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-12-18T00:00:09.9691805Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-12-18T00:00:09.9811864Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-12-18T00:00:09.9910055Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-12-18T00:00:10.0160176Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-12-18T00:00:10.0508593Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1' 2024-12-18T00:00:10.0884230Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-12-18T00:00:10.8188324Z Submodule path 'third_party/XNNPACK': checked out '4ea82e595b36106653175dcb04b2aa532660d0d8' 2024-12-18T00:00:10.8429167Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-12-18T00:00:11.0781515Z Submodule path 'third_party/composable_kernel': checked out '50ee4267e27b875d149e642f4cebd47be1dc3b57' 2024-12-18T00:00:11.1271877Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-12-18T00:00:11.2279882Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99' 2024-12-18T00:00:11.2615728Z Submodule path 'third_party/cudnn_frontend': checked out '936021bfed8c91dc416af1588b2c4eca631a9e45' 2024-12-18T00:00:11.7768849Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-12-18T00:00:12.0379765Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-12-18T00:00:12.1353918Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-12-18T00:00:12.1372481Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:00:12.1375248Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:00:12.1376511Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:00:12.1377813Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:00:12.1381254Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:00:12.1410488Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-12-18T00:00:12.9924656Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-12-18T00:00:13.6454834Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-12-18T00:00:15.6955809Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-12-18T00:00:16.6285989Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-12-18T00:00:17.0080597Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-12-18T00:00:17.1087641Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-12-18T00:00:17.5219542Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-12-18T00:00:17.5861552Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-12-18T00:00:17.5993713Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-12-18T00:00:17.7327992Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-12-18T00:00:17.7731909Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-12-18T00:00:17.8130696Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-12-18T00:00:17.8391983Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-12-18T00:00:17.8838745Z Submodule path 'third_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59' 2024-12-18T00:00:17.8971421Z Submodule path 'third_party/ideep': checked out 'c7ccd5bdbe5434ba156f4e856dcef0601637334b' 2024-12-18T00:00:17.8987851Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-12-18T00:00:17.9015792Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-12-18T00:00:30.7889355Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af' 2024-12-18T00:00:30.8079529Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-12-18T00:00:30.8977689Z Submodule path 'third_party/kineto': checked out '338140f58a28d599da3434ced4fd2d75dd1a213d' 2024-12-18T00:00:30.8996597Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:00:30.8998335Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:00:30.8999751Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:00:30.9030435Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-12-18T00:00:31.6913535Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-12-18T00:00:32.7894789Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-12-18T00:00:33.8037919Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-12-18T00:00:33.8055200Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:00:33.8056892Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:00:33.8058278Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:00:33.8061160Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:00:33.8064011Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:00:33.8067162Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:00:33.8070318Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:00:33.8073630Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:00:33.8102198Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-12-18T00:00:34.7511244Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-12-18T00:00:35.1892362Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-12-18T00:00:36.2558021Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-12-18T00:00:36.5996295Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-12-18T00:00:37.0849865Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-12-18T00:00:37.9938767Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-12-18T00:00:45.3831437Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-12-18T00:00:45.8083994Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-12-18T00:00:45.8275286Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-12-18T00:00:45.8657008Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-12-18T00:00:45.8803603Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-12-18T00:00:45.8821222Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:00:45.8848696Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-12-18T00:00:46.2001846Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-12-18T00:00:46.2195232Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-12-18T00:00:46.2606430Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-12-18T00:00:46.3646331Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-12-18T00:00:46.3815588Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-12-18T00:00:46.4214511Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-12-18T00:00:46.4804837Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-12-18T00:00:46.5183867Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-12-18T00:00:46.5480972Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-12-18T00:00:46.6621595Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-12-18T00:00:47.0282712Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c' 2024-12-18T00:00:47.0318652Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-12-18T00:00:47.0345460Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-12-18T00:00:48.0480482Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-12-18T00:00:48.1176027Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-12-18T00:00:48.1195951Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:00:48.1197414Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:00:48.1198852Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:00:48.1201677Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:00:48.1204590Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:00:48.1207394Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:00:48.1210673Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:00:48.1213679Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:00:48.1240755Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-12-18T00:00:48.5554204Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-12-18T00:00:49.5009727Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-12-18T00:00:49.8208426Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-12-18T00:00:56.2829865Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-12-18T00:00:56.5968531Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-12-18T00:00:56.9168222Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-12-18T00:00:57.2582027Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-12-18T00:01:03.6374305Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-12-18T00:01:03.6797572Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-12-18T00:01:03.6974516Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-12-18T00:01:03.8082559Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-12-18T00:01:03.8215673Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-12-18T00:01:03.8365494Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-12-18T00:01:03.8520652Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-12-18T00:01:03.8535070Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:01:03.8537502Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:01:03.8566352Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-12-18T00:01:05.5486725Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-12-18T00:01:06.7166273Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-12-18T00:01:06.7648894Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-12-18T00:01:07.2306763Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-12-18T00:01:07.2424652Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-12-18T00:01:07.5281658Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-12-18T00:01:07.5304009Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:01:07.5305832Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-12-18T00:01:07.5334765Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-12-18T00:01:07.9869159Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-12-18T00:01:08.9223162Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-12-18T00:01:08.9960598Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-12-18T00:01:09.0059226Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-12-18T00:01:09.0186419Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-12-18T00:01:09.0555711Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2024-12-18T00:01:09.0851600Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-12-18T00:01:09.1294842Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-12-18T00:01:09.1562859Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-12-18T00:01:09.1581202Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:01:09.1583004Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:01:09.1584326Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:01:09.1586744Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:01:09.1620339Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-12-18T00:01:10.0982261Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-12-18T00:01:10.3675271Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-12-18T00:01:12.5549506Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-12-18T00:01:13.6316790Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-12-18T00:01:13.6470400Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-12-18T00:01:13.7096469Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-12-18T00:01:13.7385479Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-12-18T00:01:13.7403030Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:01:13.7435890Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-12-18T00:01:13.9582311Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-12-18T00:01:13.9622626Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-12-18T00:01:13.9930543Z Entering 'android/libs/fbjni' 2024-12-18T00:01:13.9975271Z Entering 'third_party/FP16' 2024-12-18T00:01:14.0018066Z Entering 'third_party/FXdiv' 2024-12-18T00:01:14.0061520Z Entering 'third_party/NNPACK' 2024-12-18T00:01:14.0108446Z Entering 'third_party/NVTX' 2024-12-18T00:01:14.0153083Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:01:14.0201468Z Entering 'third_party/XNNPACK' 2024-12-18T00:01:14.0261055Z Entering 'third_party/benchmark' 2024-12-18T00:01:14.0302915Z Entering 'third_party/composable_kernel' 2024-12-18T00:01:14.0355959Z Entering 'third_party/cpp-httplib' 2024-12-18T00:01:14.0397682Z Entering 'third_party/cpuinfo' 2024-12-18T00:01:14.0439403Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:01:14.0482995Z Entering 'third_party/cutlass' 2024-12-18T00:01:14.0531751Z Entering 'third_party/eigen' 2024-12-18T00:01:14.0578500Z Entering 'third_party/fbgemm' 2024-12-18T00:01:14.0621833Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:01:14.0667824Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:01:14.0709443Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:01:14.0759971Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:01:14.0801558Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:01:14.0846866Z Entering 'third_party/flatbuffers' 2024-12-18T00:01:14.0893350Z Entering 'third_party/fmt' 2024-12-18T00:01:14.0936027Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:01:14.0980194Z Entering 'third_party/gloo' 2024-12-18T00:01:14.1021037Z Entering 'third_party/googletest' 2024-12-18T00:01:14.1064356Z Entering 'third_party/ideep' 2024-12-18T00:01:14.1106924Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:01:14.1160550Z Entering 'third_party/ittapi' 2024-12-18T00:01:14.1204891Z Entering 'third_party/kineto' 2024-12-18T00:01:14.1247367Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:01:14.1290673Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:01:14.1333389Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:01:14.1378623Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:01:14.1420582Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:01:14.1462516Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:01:14.1510906Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:01:14.1554607Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:01:14.1601710Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:01:14.1643543Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:01:14.1689064Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:01:14.1731172Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:01:14.1777174Z Entering 'third_party/mimalloc' 2024-12-18T00:01:14.1820131Z Entering 'third_party/nccl/nccl' 2024-12-18T00:01:14.1863701Z Entering 'third_party/nlohmann' 2024-12-18T00:01:14.1906718Z Entering 'third_party/onnx' 2024-12-18T00:01:14.1966220Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:01:14.2015346Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:01:14.2059842Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:01:14.2104893Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:01:14.2147382Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:01:14.2190221Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:01:14.2234604Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:01:14.2277403Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:01:14.2321672Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:01:14.2369598Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:01:14.2415644Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:01:14.2460542Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:01:14.2522343Z Entering 'third_party/pocketfft' 2024-12-18T00:01:14.2565531Z Entering 'third_party/protobuf' 2024-12-18T00:01:14.2613792Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:01:14.2656275Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:01:14.2701938Z Entering 'third_party/psimd' 2024-12-18T00:01:14.2743847Z Entering 'third_party/pthreadpool' 2024-12-18T00:01:14.2785105Z Entering 'third_party/pybind11' 2024-12-18T00:01:14.2827765Z Entering 'third_party/python-peachpy' 2024-12-18T00:01:14.2870731Z Entering 'third_party/sleef' 2024-12-18T00:01:14.2915031Z Entering 'third_party/tensorpipe' 2024-12-18T00:01:14.2959094Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:01:14.3001452Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:01:14.3043579Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:01:14.3084065Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:01:14.3124506Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:01:14.3180940Z ##[endgroup] 2024-12-18T00:01:14.3183110Z ##[group]Persisting credentials for submodules 2024-12-18T00:01:14.3185724Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-12-18T00:01:14.3489988Z Entering 'android/libs/fbjni' 2024-12-18T00:01:14.3549366Z Entering 'third_party/FP16' 2024-12-18T00:01:14.3605244Z Entering 'third_party/FXdiv' 2024-12-18T00:01:14.3660467Z Entering 'third_party/NNPACK' 2024-12-18T00:01:14.3722692Z Entering 'third_party/NVTX' 2024-12-18T00:01:14.3779099Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:01:14.3833477Z Entering 'third_party/XNNPACK' 2024-12-18T00:01:14.3902804Z Entering 'third_party/benchmark' 2024-12-18T00:01:14.3959022Z Entering 'third_party/composable_kernel' 2024-12-18T00:01:14.4020453Z Entering 'third_party/cpp-httplib' 2024-12-18T00:01:14.4078529Z Entering 'third_party/cpuinfo' 2024-12-18T00:01:14.4133586Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:01:14.4188871Z Entering 'third_party/cutlass' 2024-12-18T00:01:14.4253552Z Entering 'third_party/eigen' 2024-12-18T00:01:14.4317473Z Entering 'third_party/fbgemm' 2024-12-18T00:01:14.4375925Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:01:14.4428893Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:01:14.4485269Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:01:14.4545578Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:01:14.4600050Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:01:14.4656796Z Entering 'third_party/flatbuffers' 2024-12-18T00:01:14.4718080Z Entering 'third_party/fmt' 2024-12-18T00:01:14.4776667Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:01:14.4831178Z Entering 'third_party/gloo' 2024-12-18T00:01:14.4892855Z Entering 'third_party/googletest' 2024-12-18T00:01:14.4949632Z Entering 'third_party/ideep' 2024-12-18T00:01:14.5002945Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:01:14.5062471Z Entering 'third_party/ittapi' 2024-12-18T00:01:14.5117508Z Entering 'third_party/kineto' 2024-12-18T00:01:14.5176749Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:01:14.5239337Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:01:14.5302382Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:01:14.5359669Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:01:14.5418667Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:01:14.5480040Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:01:14.5542950Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:01:14.5598830Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:01:14.5655459Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:01:14.5716609Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:01:14.5775186Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:01:14.5832199Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:01:14.5893810Z Entering 'third_party/mimalloc' 2024-12-18T00:01:14.5949109Z Entering 'third_party/nccl/nccl' 2024-12-18T00:01:14.6004558Z Entering 'third_party/nlohmann' 2024-12-18T00:01:14.6060410Z Entering 'third_party/onnx' 2024-12-18T00:01:14.6130304Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:01:14.6189511Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:01:14.6247311Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:01:14.6301004Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:01:14.6358173Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:01:14.6413287Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:01:14.6470573Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:01:14.6525219Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:01:14.6579995Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:01:14.6630798Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:01:14.6692768Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:01:14.6760160Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:01:14.6833457Z Entering 'third_party/pocketfft' 2024-12-18T00:01:14.6889285Z Entering 'third_party/protobuf' 2024-12-18T00:01:14.6956569Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:01:14.7016876Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:01:14.7076778Z Entering 'third_party/psimd' 2024-12-18T00:01:14.7133588Z Entering 'third_party/pthreadpool' 2024-12-18T00:01:14.7188553Z Entering 'third_party/pybind11' 2024-12-18T00:01:14.7243189Z Entering 'third_party/python-peachpy' 2024-12-18T00:01:14.7300675Z Entering 'third_party/sleef' 2024-12-18T00:01:14.7357023Z Entering 'third_party/tensorpipe' 2024-12-18T00:01:14.7411801Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:01:14.7470978Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:01:14.7525211Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:01:14.7581370Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:01:14.7639237Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:01:14.7722459Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-12-18T00:01:14.8031776Z Entering 'android/libs/fbjni' 2024-12-18T00:01:14.8084592Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-12-18T00:01:14.8100813Z Entering 'third_party/FP16' 2024-12-18T00:01:14.8154819Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-12-18T00:01:14.8172129Z Entering 'third_party/FXdiv' 2024-12-18T00:01:14.8221391Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-12-18T00:01:14.8237222Z Entering 'third_party/NNPACK' 2024-12-18T00:01:14.8290121Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-12-18T00:01:14.8306117Z Entering 'third_party/NVTX' 2024-12-18T00:01:14.8359638Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2024-12-18T00:01:14.8377152Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:01:14.8429073Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-12-18T00:01:14.8447394Z Entering 'third_party/XNNPACK' 2024-12-18T00:01:14.8499563Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-12-18T00:01:14.8532926Z Entering 'third_party/benchmark' 2024-12-18T00:01:14.8583864Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-12-18T00:01:14.8602786Z Entering 'third_party/composable_kernel' 2024-12-18T00:01:14.8655981Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2024-12-18T00:01:14.8680532Z Entering 'third_party/cpp-httplib' 2024-12-18T00:01:14.8733634Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-12-18T00:01:14.8749127Z Entering 'third_party/cpuinfo' 2024-12-18T00:01:14.8801139Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-12-18T00:01:14.8819083Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:01:14.8871027Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-12-18T00:01:14.8887239Z Entering 'third_party/cutlass' 2024-12-18T00:01:14.8936200Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-12-18T00:01:14.8963366Z Entering 'third_party/eigen' 2024-12-18T00:01:14.9014068Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-12-18T00:01:14.9032257Z Entering 'third_party/fbgemm' 2024-12-18T00:01:14.9085773Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-12-18T00:01:14.9101477Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:01:14.9153892Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-12-18T00:01:14.9170750Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:01:14.9221354Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-12-18T00:01:14.9237081Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:01:14.9290783Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-12-18T00:01:14.9318953Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:01:14.9371491Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:01:14.9386499Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:01:14.9437058Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-12-18T00:01:14.9456593Z Entering 'third_party/flatbuffers' 2024-12-18T00:01:14.9509841Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-12-18T00:01:14.9532386Z Entering 'third_party/fmt' 2024-12-18T00:01:14.9581722Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-12-18T00:01:14.9599734Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:01:14.9650704Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-12-18T00:01:14.9666240Z Entering 'third_party/gloo' 2024-12-18T00:01:14.9718401Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-12-18T00:01:14.9734177Z Entering 'third_party/googletest' 2024-12-18T00:01:14.9784846Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:01:14.9803639Z Entering 'third_party/ideep' 2024-12-18T00:01:14.9857142Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-12-18T00:01:14.9874321Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:01:14.9923748Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-12-18T00:01:14.9947362Z Entering 'third_party/ittapi' 2024-12-18T00:01:14.9998873Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-12-18T00:01:15.0016103Z Entering 'third_party/kineto' 2024-12-18T00:01:15.0066906Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-12-18T00:01:15.0084942Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:01:15.0135213Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-12-18T00:01:15.0152629Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:01:15.0205557Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-12-18T00:01:15.0222255Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:01:15.0276964Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-12-18T00:01:15.0294630Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:01:15.0344834Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-12-18T00:01:15.0363094Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:01:15.0414444Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-12-18T00:01:15.0428845Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:01:15.0482154Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-12-18T00:01:15.0500698Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:01:15.0551566Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-12-18T00:01:15.0571634Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:01:15.0621541Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:01:15.0637144Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:01:15.0690732Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-12-18T00:01:15.0707034Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:01:15.0761670Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-12-18T00:01:15.0780453Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:01:15.0829550Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-12-18T00:01:15.0848226Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:01:15.0898250Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-12-18T00:01:15.0918695Z Entering 'third_party/mimalloc' 2024-12-18T00:01:15.0971772Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-12-18T00:01:15.0986778Z Entering 'third_party/nccl/nccl' 2024-12-18T00:01:15.1037902Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-12-18T00:01:15.1056238Z Entering 'third_party/nlohmann' 2024-12-18T00:01:15.1106254Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-12-18T00:01:15.1126394Z Entering 'third_party/onnx' 2024-12-18T00:01:15.1179350Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-12-18T00:01:15.1211410Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:01:15.1262172Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-12-18T00:01:15.1282952Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:01:15.1333663Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-12-18T00:01:15.1350218Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:01:15.1403216Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-12-18T00:01:15.1420898Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:01:15.1471830Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:01:15.1498540Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:01:15.1540243Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-12-18T00:01:15.1558120Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:01:15.1608757Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-12-18T00:01:15.1625501Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:01:15.1678282Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-12-18T00:01:15.1695657Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:01:15.1745909Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-12-18T00:01:15.1764227Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:01:15.1815058Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-12-18T00:01:15.1829360Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:01:15.1882694Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-12-18T00:01:15.1901150Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:01:15.1954697Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-12-18T00:01:15.1974446Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:01:15.2024445Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-12-18T00:01:15.2067774Z Entering 'third_party/pocketfft' 2024-12-18T00:01:15.2118156Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-12-18T00:01:15.2133677Z Entering 'third_party/protobuf' 2024-12-18T00:01:15.2183472Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-12-18T00:01:15.2205268Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:01:15.2257015Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-12-18T00:01:15.2274384Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:01:15.2324816Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:01:15.2342884Z Entering 'third_party/psimd' 2024-12-18T00:01:15.2394757Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-12-18T00:01:15.2411819Z Entering 'third_party/pthreadpool' 2024-12-18T00:01:15.2462055Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-12-18T00:01:15.2480015Z Entering 'third_party/pybind11' 2024-12-18T00:01:15.2530231Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-12-18T00:01:15.2546027Z Entering 'third_party/python-peachpy' 2024-12-18T00:01:15.2597854Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-12-18T00:01:15.2614897Z Entering 'third_party/sleef' 2024-12-18T00:01:15.2665079Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-12-18T00:01:15.2683560Z Entering 'third_party/tensorpipe' 2024-12-18T00:01:15.2734098Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-12-18T00:01:15.2749079Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:01:15.2800297Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:01:15.2817466Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:01:15.2868334Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-12-18T00:01:15.2885649Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:01:15.2934934Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-12-18T00:01:15.2953296Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:01:15.3004590Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-12-18T00:01:15.3020507Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:01:15.3073531Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-12-18T00:01:15.3863533Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-12-18T00:01:15.4174548Z Entering 'android/libs/fbjni' 2024-12-18T00:01:15.4218419Z Entering 'third_party/FP16' 2024-12-18T00:01:15.4264611Z Entering 'third_party/FXdiv' 2024-12-18T00:01:15.4306118Z Entering 'third_party/NNPACK' 2024-12-18T00:01:15.4348544Z Entering 'third_party/NVTX' 2024-12-18T00:01:15.4390817Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:01:15.4435275Z Entering 'third_party/XNNPACK' 2024-12-18T00:01:15.4493614Z Entering 'third_party/benchmark' 2024-12-18T00:01:15.4536061Z Entering 'third_party/composable_kernel' 2024-12-18T00:01:15.4584588Z Entering 'third_party/cpp-httplib' 2024-12-18T00:01:15.4626035Z Entering 'third_party/cpuinfo' 2024-12-18T00:01:15.4669153Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:01:15.4712569Z Entering 'third_party/cutlass' 2024-12-18T00:01:15.4764372Z Entering 'third_party/eigen' 2024-12-18T00:01:15.4808654Z Entering 'third_party/fbgemm' 2024-12-18T00:01:15.4853389Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:01:15.4900730Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:01:15.4941902Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:01:15.4990200Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:01:15.5032220Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:01:15.5083829Z Entering 'third_party/flatbuffers' 2024-12-18T00:01:15.5130374Z Entering 'third_party/fmt' 2024-12-18T00:01:15.5174351Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:01:15.5217338Z Entering 'third_party/gloo' 2024-12-18T00:01:15.5260462Z Entering 'third_party/googletest' 2024-12-18T00:01:15.5301795Z Entering 'third_party/ideep' 2024-12-18T00:01:15.5344563Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:01:15.5394788Z Entering 'third_party/ittapi' 2024-12-18T00:01:15.5435460Z Entering 'third_party/kineto' 2024-12-18T00:01:15.5477867Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:01:15.5519206Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:01:15.5563528Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:01:15.5606532Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:01:15.5650651Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:01:15.5693210Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:01:15.5743752Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:01:15.5785804Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:01:15.5832021Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:01:15.5878066Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:01:15.5920975Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:01:15.5963517Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:01:15.6007024Z Entering 'third_party/mimalloc' 2024-12-18T00:01:15.6051001Z Entering 'third_party/nccl/nccl' 2024-12-18T00:01:15.6100865Z Entering 'third_party/nlohmann' 2024-12-18T00:01:15.6143745Z Entering 'third_party/onnx' 2024-12-18T00:01:15.6203517Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:01:15.6251815Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:01:15.6297082Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:01:15.6337547Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:01:15.6380293Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:01:15.6420931Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:01:15.6464155Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:01:15.6504392Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:01:15.6544898Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:01:15.6585161Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:01:15.6632758Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:01:15.6682655Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:01:15.6743335Z Entering 'third_party/pocketfft' 2024-12-18T00:01:15.6784713Z Entering 'third_party/protobuf' 2024-12-18T00:01:15.6829606Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:01:15.6874340Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:01:15.6918407Z Entering 'third_party/psimd' 2024-12-18T00:01:15.6962158Z Entering 'third_party/pthreadpool' 2024-12-18T00:01:15.7004988Z Entering 'third_party/pybind11' 2024-12-18T00:01:15.7048144Z Entering 'third_party/python-peachpy' 2024-12-18T00:01:15.7099635Z Entering 'third_party/sleef' 2024-12-18T00:01:15.7142436Z Entering 'third_party/tensorpipe' 2024-12-18T00:01:15.7184237Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:01:15.7231949Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:01:15.7275295Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:01:15.7317249Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:01:15.7360026Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:01:15.7416012Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-12-18T00:01:15.7718021Z Entering 'android/libs/fbjni' 2024-12-18T00:01:15.7760239Z Entering 'third_party/FP16' 2024-12-18T00:01:15.7801504Z Entering 'third_party/FXdiv' 2024-12-18T00:01:15.7845700Z Entering 'third_party/NNPACK' 2024-12-18T00:01:15.7888915Z Entering 'third_party/NVTX' 2024-12-18T00:01:15.7932861Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:01:15.7976904Z Entering 'third_party/XNNPACK' 2024-12-18T00:01:15.8035798Z Entering 'third_party/benchmark' 2024-12-18T00:01:15.8079294Z Entering 'third_party/composable_kernel' 2024-12-18T00:01:15.8129438Z Entering 'third_party/cpp-httplib' 2024-12-18T00:01:15.8174009Z Entering 'third_party/cpuinfo' 2024-12-18T00:01:15.8217727Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:01:15.8264130Z Entering 'third_party/cutlass' 2024-12-18T00:01:15.8316857Z Entering 'third_party/eigen' 2024-12-18T00:01:15.8363467Z Entering 'third_party/fbgemm' 2024-12-18T00:01:15.8406784Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:01:15.8449823Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:01:15.8497518Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:01:15.8549876Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:01:15.8593677Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:01:15.8643932Z Entering 'third_party/flatbuffers' 2024-12-18T00:01:15.8700627Z Entering 'third_party/fmt' 2024-12-18T00:01:15.8742311Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:01:15.8784935Z Entering 'third_party/gloo' 2024-12-18T00:01:15.8827138Z Entering 'third_party/googletest' 2024-12-18T00:01:15.8873660Z Entering 'third_party/ideep' 2024-12-18T00:01:15.8916192Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:01:15.8967355Z Entering 'third_party/ittapi' 2024-12-18T00:01:15.9008894Z Entering 'third_party/kineto' 2024-12-18T00:01:15.9052136Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:01:15.9094656Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:01:15.9137005Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:01:15.9180545Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:01:15.9221124Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:01:15.9269917Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:01:15.9323462Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:01:15.9366273Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:01:15.9408186Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:01:15.9452804Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:01:15.9501656Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:01:15.9543113Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:01:15.9585563Z Entering 'third_party/mimalloc' 2024-12-18T00:01:15.9628087Z Entering 'third_party/nccl/nccl' 2024-12-18T00:01:15.9670719Z Entering 'third_party/nlohmann' 2024-12-18T00:01:15.9715952Z Entering 'third_party/onnx' 2024-12-18T00:01:15.9774713Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:01:15.9819708Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:01:15.9863845Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:01:15.9904538Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:01:15.9946561Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:01:15.9994347Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:01:16.0036115Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:01:16.0078531Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:01:16.0119352Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:01:16.0160845Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:01:16.0205063Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:01:16.0249686Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:01:16.0307356Z Entering 'third_party/pocketfft' 2024-12-18T00:01:16.0350308Z Entering 'third_party/protobuf' 2024-12-18T00:01:16.0398396Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:01:16.0437023Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:01:16.0482239Z Entering 'third_party/psimd' 2024-12-18T00:01:16.0523994Z Entering 'third_party/pthreadpool' 2024-12-18T00:01:16.0567221Z Entering 'third_party/pybind11' 2024-12-18T00:01:16.0610593Z Entering 'third_party/python-peachpy' 2024-12-18T00:01:16.0653361Z Entering 'third_party/sleef' 2024-12-18T00:01:16.0698025Z Entering 'third_party/tensorpipe' 2024-12-18T00:01:16.0739864Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:01:16.0781657Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:01:16.0822613Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:01:16.0866793Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:01:16.0909082Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:01:16.0966827Z ##[endgroup] 2024-12-18T00:01:16.1001594Z [command]/usr/bin/git log -1 --format='%H' 2024-12-18T00:01:16.1023006Z '0cdf8b1d09254cfda66191d1bd01e3041c3c76f7' 2024-12-18T00:01:16.1193221Z Prepare all required actions 2024-12-18T00:01:16.1193797Z Getting action download info 2024-12-18T00:01:16.2682640Z ##[group]Run ./.github/actions/setup-linux 2024-12-18T00:01:16.2682992Z env: 2024-12-18T00:01:16.2683215Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:16.2683503Z ##[endgroup] 2024-12-18T00:01:16.2734079Z ##[group]Run set -euo pipefail 2024-12-18T00:01:16.2734495Z set -euo pipefail 2024-12-18T00:01:16.2734833Z function get_ec2_metadata() { 2024-12-18T00:01:16.2735282Z  # Pulled from instance metadata endpoint for EC2 2024-12-18T00:01:16.2736014Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-12-18T00:01:16.2736663Z  category=$1 2024-12-18T00:01:16.2737093Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-12-18T00:01:16.2737624Z  runner_name_str=i-0d4c5ae1b54bc839d 2024-12-18T00:01:16.2738070Z  if [[ -f /.inarc ]]; then 2024-12-18T00:01:16.2738491Z  echo "ARC Runner, no info on ec2 metadata" 2024-12-18T00:01:16.2739079Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-12-18T00:01:16.2739819Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-12-18T00:01:16.2740341Z  else 2024-12-18T00:01:16.2741364Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-12-18T00:01:16.2742446Z  fi 2024-12-18T00:01:16.2742716Z } 2024-12-18T00:01:16.2743028Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-12-18T00:01:16.2743549Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-12-18T00:01:16.2744123Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-12-18T00:01:16.2744634Z echo "system info $(uname -a)" 2024-12-18T00:01:16.2753615Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:16.2754028Z env: 2024-12-18T00:01:16.2754268Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:16.2754558Z ##[endgroup] 2024-12-18T00:01:16.2901831Z ami-id: ami-0fff1b9a61dec8a5f 2024-12-18T00:01:16.3007763Z instance-id: i-0d4c5ae1b54bc839d 2024-12-18T00:01:16.3114882Z instance-type: g4dn.12xlarge 2024-12-18T00:01:16.3124827Z system info Linux ip-10-0-16-113.ec2.internal 6.1.109-118.189.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 10 08:59:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-12-18T00:01:16.3154808Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:01:16.3155846Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:01:16.3162631Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:16.3163071Z env: 2024-12-18T00:01:16.3163312Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:16.3163616Z ##[endgroup] 2024-12-18T00:01:16.3227687Z ##[group]Run if systemctl is-active --quiet docker; then 2024-12-18T00:01:16.3228211Z if systemctl is-active --quiet docker; then 2024-12-18T00:01:16.3228677Z  echo "Docker daemon is running..."; 2024-12-18T00:01:16.3229059Z else 2024-12-18T00:01:16.3229487Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-12-18T00:01:16.3229993Z fi 2024-12-18T00:01:16.3236119Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:16.3236578Z env: 2024-12-18T00:01:16.3236844Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:16.3237153Z ##[endgroup] 2024-12-18T00:01:16.3315104Z Docker daemon is running... 2024-12-18T00:01:16.3364557Z ##[group]Run nick-fields/retry@v3.0.0 2024-12-18T00:01:16.3365028Z with: 2024-12-18T00:01:16.3365260Z shell: bash 2024-12-18T00:01:16.3365620Z timeout_minutes: 5 2024-12-18T00:01:16.3365892Z max_attempts: 3 2024-12-18T00:01:16.3366154Z retry_wait_seconds: 30 2024-12-18T00:01:16.3368512Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-12-18T00:01:16.3370898Z polling_interval_seconds: 1 2024-12-18T00:01:16.3371215Z warning_on_retry: true 2024-12-18T00:01:16.3371513Z continue_on_error: false 2024-12-18T00:01:16.3371789Z env: 2024-12-18T00:01:16.3372029Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:16.3372325Z AWS_RETRY_MODE: standard 2024-12-18T00:01:16.3372613Z AWS_MAX_ATTEMPTS: 5 2024-12-18T00:01:16.3372894Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T00:01:16.3373179Z ##[endgroup] 2024-12-18T00:01:17.6062358Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:01:17.6063176Z Configure a credential helper to remove this warning. See 2024-12-18T00:01:17.6063872Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:01:17.6064345Z 2024-12-18T00:01:17.6064467Z Login Succeeded 2024-12-18T00:01:18.4774564Z Command completed after 1 attempt(s). 2024-12-18T00:01:18.4878295Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T00:01:18.4878842Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T00:01:18.4879314Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T00:01:18.4886133Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:18.4886524Z env: 2024-12-18T00:01:18.4886742Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:18.4887009Z ##[endgroup] 2024-12-18T00:01:18.4975736Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T00:01:18.4976374Z # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T00:01:18.4976864Z # shellcheck disable=SC2046 2024-12-18T00:01:18.4977236Z docker stop $(docker ps -q) || true 2024-12-18T00:01:18.4977633Z # Prune all of the docker images 2024-12-18T00:01:18.4978005Z docker system prune -af 2024-12-18T00:01:18.4984199Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:18.4984655Z env: 2024-12-18T00:01:18.4984920Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:18.4985228Z ##[endgroup] 2024-12-18T00:01:18.5260360Z "docker stop" requires at least 1 argument. 2024-12-18T00:01:18.5261086Z See 'docker stop --help'. 2024-12-18T00:01:18.5261317Z 2024-12-18T00:01:18.5261531Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-12-18T00:01:18.5261857Z 2024-12-18T00:01:18.5262006Z Stop one or more running containers 2024-12-18T00:01:18.5437776Z Total reclaimed space: 0B 2024-12-18T00:01:18.5483605Z ##[group]Run set +e 2024-12-18T00:01:18.5483929Z set +e 2024-12-18T00:01:18.5484201Z set -x 2024-12-18T00:01:18.5484466Z  2024-12-18T00:01:18.5484741Z PT_DOMAIN=download.pytorch.org 2024-12-18T00:01:18.5485395Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-12-18T00:01:18.5486246Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-12-18T00:01:18.5486843Z # one is returned at random 2024-12-18T00:01:18.5487297Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-12-18T00:01:18.5487865Z  2024-12-18T00:01:18.5488249Z if [ -z "${RESOLVED_IP}" ]; then 2024-12-18T00:01:18.5488751Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-12-18T00:01:18.5489345Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-12-18T00:01:18.5489801Z  2024-12-18T00:01:18.5490178Z  if [ -z "${RESOLVED_IP}" ]; then 2024-12-18T00:01:18.5490599Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-12-18T00:01:18.5490979Z  exit 1 2024-12-18T00:01:18.5491239Z  fi 2024-12-18T00:01:18.5491478Z fi 2024-12-18T00:01:18.5491710Z  2024-12-18T00:01:18.5491993Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-12-18T00:01:18.5492378Z  # Clean up any old records first 2024-12-18T00:01:18.5492768Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-12-18T00:01:18.5493124Z fi 2024-12-18T00:01:18.5493360Z  2024-12-18T00:01:18.5493703Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-12-18T00:01:18.5494117Z cat /etc/hosts 2024-12-18T00:01:18.5499930Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:18.5500384Z env: 2024-12-18T00:01:18.5500649Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:18.5500968Z ##[endgroup] 2024-12-18T00:01:18.5525020Z + PT_DOMAIN=download.pytorch.org 2024-12-18T00:01:18.5531362Z ++ dig -4 +short download.pytorch.org 2024-12-18T00:01:18.5532400Z ++ tail -n1 2024-12-18T00:01:18.5980088Z + RESOLVED_IP=3.167.99.82 2024-12-18T00:01:18.5980790Z + '[' -z 3.167.99.82 ']' 2024-12-18T00:01:18.5981183Z + grep -r download.pytorch.org /etc/hosts 2024-12-18T00:01:18.5997023Z + echo '3.167.99.82 download.pytorch.org' 2024-12-18T00:01:18.5997747Z + sudo tee -a /etc/hosts 2024-12-18T00:01:18.8699873Z 3.167.99.82 download.pytorch.org 2024-12-18T00:01:18.8715493Z + cat /etc/hosts 2024-12-18T00:01:18.8724437Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-12-18T00:01:18.8729333Z ::1 localhost6 localhost6.localdomain6 2024-12-18T00:01:18.8729770Z 3.167.99.82 download.pytorch.org 2024-12-18T00:01:18.8900626Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@release/2.6 2024-12-18T00:01:18.8901201Z with: 2024-12-18T00:01:18.8902062Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:18.8903040Z docker-build-dir: .ci/docker 2024-12-18T00:01:18.8903385Z working-directory: . 2024-12-18T00:01:18.8903809Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:18.8904276Z force-push: false 2024-12-18T00:01:18.8904561Z env: 2024-12-18T00:01:18.8904821Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:18.8905136Z ##[endgroup] 2024-12-18T00:01:18.8934232Z ##[group]Run set -ex 2024-12-18T00:01:18.8934590Z set -ex 2024-12-18T00:01:18.8934876Z  2024-12-18T00:01:18.8935340Z # If the docker build directory or the build script doesn't exist, the action will 2024-12-18T00:01:18.8936159Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-12-18T00:01:18.8936806Z # job could then download the pre-built image as usual 2024-12-18T00:01:18.8937403Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-12-18T00:01:18.8937956Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:18.8938480Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:18.8939069Z  2024-12-18T00:01:18.8939488Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-12-18T00:01:18.8940187Z  exit 0 2024-12-18T00:01:18.8940468Z else 2024-12-18T00:01:18.8940795Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:18.8941319Z fi 2024-12-18T00:01:18.8941579Z  2024-12-18T00:01:18.8941974Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-12-18T00:01:18.8942693Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-12-18T00:01:18.8943332Z  # use it as it is, but first let's extract the tag 2024-12-18T00:01:18.8943909Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-12-18T00:01:18.8944519Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:18.8945107Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:18.8945597Z else 2024-12-18T00:01:18.8945966Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-12-18T00:01:18.8946519Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:18.8947283Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:18.8947953Z fi 2024-12-18T00:01:18.8955074Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:18.8955484Z env: 2024-12-18T00:01:18.8955711Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:18.8956000Z REPO_NAME: pytorch 2024-12-18T00:01:18.8956786Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:18.8957633Z DOCKER_BUILD_DIR: .ci/docker 2024-12-18T00:01:18.8958031Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:18.8958436Z ##[endgroup] 2024-12-18T00:01:18.8983467Z + [[ ! -d .ci/docker ]] 2024-12-18T00:01:18.8983847Z + [[ ! -f .ci/docker/build.sh ]] 2024-12-18T00:01:18.8984199Z + echo skip=false 2024-12-18T00:01:18.8985363Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-12-18T00:01:18.8990627Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:18.8991521Z ++ awk -F '[:,]' '{print $2}' 2024-12-18T00:01:18.9014189Z + DOCKER_TAG=45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:18.9014774Z + echo docker-tag=45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:18.9015836Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:18.9054488Z ##[group]Run set +e 2024-12-18T00:01:18.9054843Z set +e 2024-12-18T00:01:18.9055101Z set -x 2024-12-18T00:01:18.9055348Z  2024-12-18T00:01:18.9055572Z login() { 2024-12-18T00:01:18.9056094Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-12-18T00:01:18.9056645Z } 2024-12-18T00:01:18.9056875Z  2024-12-18T00:01:18.9057110Z retry () { 2024-12-18T00:01:18.9057396Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-12-18T00:01:18.9057738Z } 2024-12-18T00:01:18.9057969Z  2024-12-18T00:01:18.9058228Z retry login "${DOCKER_REGISTRY}" 2024-12-18T00:01:18.9058568Z  2024-12-18T00:01:18.9058800Z START_TIME=$(date +%s) 2024-12-18T00:01:18.9059190Z # Wait up to 90 minutes 2024-12-18T00:01:18.9059795Z while [[ $(( $(date +%s) - 5400 )) -lt $START_TIME ]]; do 2024-12-18T00:01:18.9060383Z  # Check if image already exists, if it does then skip building it 2024-12-18T00:01:18.9060976Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-12-18T00:01:18.9061419Z  exit 0 2024-12-18T00:01:18.9061839Z  fi 2024-12-18T00:01:18.9062105Z  2024-12-18T00:01:18.9062567Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2024-12-18T00:01:18.9063404Z  # use this to differentiate between the Docker build and regular build jobs. For the 2024-12-18T00:01:18.9064398Z  # latter, it will wait for the Docker images to become available before continuing 2024-12-18T00:01:18.9065022Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2024-12-18T00:01:18.9065495Z  # It's a Docker build job, let's build the image 2024-12-18T00:01:18.9065914Z  break 2024-12-18T00:01:18.9066207Z  else 2024-12-18T00:01:18.9066627Z  # It's a regular build job, wait for the image to become available 2024-12-18T00:01:18.9067130Z  sleep 300 2024-12-18T00:01:18.9067424Z  fi 2024-12-18T00:01:18.9067690Z done 2024-12-18T00:01:18.9067955Z  2024-12-18T00:01:18.9068381Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-12-18T00:01:18.9069064Z # be empty. The default action would be to continue rebuild the image 2024-12-18T00:01:18.9069665Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-12-18T00:01:18.9070216Z  # if we're on the base branch then use the parent commit 2024-12-18T00:01:18.9070703Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-12-18T00:01:18.9071087Z else 2024-12-18T00:01:18.9071587Z  # otherwise we're on a PR, so use the most recent base commit 2024-12-18T00:01:18.9072121Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-12-18T00:01:18.9072520Z fi 2024-12-18T00:01:18.9072765Z  2024-12-18T00:01:18.9073049Z if [[ -z "${MERGE_BASE}" ]]; then 2024-12-18T00:01:18.9073444Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:18.9073828Z  2024-12-18T00:01:18.9074361Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-12-18T00:01:18.9074980Z  exit 0 2024-12-18T00:01:18.9075475Z fi 2024-12-18T00:01:18.9075719Z  2024-12-18T00:01:18.9076044Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-12-18T00:01:18.9076771Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-12-18T00:01:18.9077394Z  exit 1 2024-12-18T00:01:18.9077648Z fi 2024-12-18T00:01:18.9077883Z  2024-12-18T00:01:18.9078261Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-12-18T00:01:18.9078966Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-12-18T00:01:18.9079599Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-12-18T00:01:18.9080327Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-12-18T00:01:18.9081145Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-12-18T00:01:18.9081638Z fi 2024-12-18T00:01:18.9081871Z  2024-12-18T00:01:18.9082140Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:18.9087627Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:18.9088036Z env: 2024-12-18T00:01:18.9088272Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:18.9088575Z DOCKER_BUILD_DIR: .ci/docker 2024-12-18T00:01:18.9088935Z BASE_REVISION: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:01:18.9089821Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:18.9090684Z DOCKER_TAG: 45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:18.9091239Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:18.9091656Z DOCKER_PUSH: 2024-12-18T00:01:18.9091911Z ##[endgroup] 2024-12-18T00:01:18.9115004Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:18.9115557Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:18.9118314Z + aws ecr get-login-password --region us-east-1 2024-12-18T00:01:18.9119617Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:19.4661684Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:01:19.4662408Z Configure a credential helper to remove this warning. See 2024-12-18T00:01:19.4663085Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:01:19.4663543Z 2024-12-18T00:01:19.4663677Z Login Succeeded 2024-12-18T00:01:19.4679720Z ++ date +%s 2024-12-18T00:01:19.4689609Z + START_TIME=1734480079 2024-12-18T00:01:19.4693571Z ++ date +%s 2024-12-18T00:01:19.4702789Z + [[ 1734474679 -lt 1734480079 ]] 2024-12-18T00:01:19.4703824Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:19.7543460Z { 2024-12-18T00:01:19.7543822Z "schemaVersion": 2, 2024-12-18T00:01:19.7544327Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-12-18T00:01:19.7544862Z "config": { 2024-12-18T00:01:19.7545254Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-12-18T00:01:19.7545855Z "size": 50450, 2024-12-18T00:01:19.7546527Z "digest": "sha256:166dd14a00627fbd0d4e062a733b69b733b63b1ac12ec3bd5285db8a8befaaf5" 2024-12-18T00:01:19.7547091Z }, 2024-12-18T00:01:19.7547333Z "layers": [ 2024-12-18T00:01:19.7547585Z { 2024-12-18T00:01:19.7547994Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7548622Z "size": 28580681, 2024-12-18T00:01:19.7549109Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-12-18T00:01:19.7549655Z }, 2024-12-18T00:01:19.7549878Z { 2024-12-18T00:01:19.7550468Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7550964Z "size": 7943353, 2024-12-18T00:01:19.7551449Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-12-18T00:01:19.7551992Z }, 2024-12-18T00:01:19.7552209Z { 2024-12-18T00:01:19.7552596Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7553077Z "size": 56226389, 2024-12-18T00:01:19.7553564Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-12-18T00:01:19.7554108Z }, 2024-12-18T00:01:19.7554327Z { 2024-12-18T00:01:19.7554703Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7555184Z "size": 187, 2024-12-18T00:01:19.7555656Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-12-18T00:01:19.7556189Z }, 2024-12-18T00:01:19.7556412Z { 2024-12-18T00:01:19.7556796Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7557505Z "size": 6885, 2024-12-18T00:01:19.7558060Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-12-18T00:01:19.7558607Z }, 2024-12-18T00:01:19.7558837Z { 2024-12-18T00:01:19.7559205Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7559688Z "size": 1383662475, 2024-12-18T00:01:19.7560169Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-12-18T00:01:19.7560697Z }, 2024-12-18T00:01:19.7560930Z { 2024-12-18T00:01:19.7561296Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7561778Z "size": 62282, 2024-12-18T00:01:19.7562249Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-12-18T00:01:19.7562902Z }, 2024-12-18T00:01:19.7563141Z { 2024-12-18T00:01:19.7563516Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7564005Z "size": 1684, 2024-12-18T00:01:19.7564489Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-12-18T00:01:19.7565141Z }, 2024-12-18T00:01:19.7565365Z { 2024-12-18T00:01:19.7565720Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7566288Z "size": 1522, 2024-12-18T00:01:19.7566722Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-12-18T00:01:19.7567216Z }, 2024-12-18T00:01:19.7567426Z { 2024-12-18T00:01:19.7567759Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7568202Z "size": 2445884909, 2024-12-18T00:01:19.7568648Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-12-18T00:01:19.7569139Z }, 2024-12-18T00:01:19.7569350Z { 2024-12-18T00:01:19.7569684Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7570126Z "size": 85882, 2024-12-18T00:01:19.7570562Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-12-18T00:01:19.7571055Z }, 2024-12-18T00:01:19.7571266Z { 2024-12-18T00:01:19.7571601Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7572041Z "size": 1825, 2024-12-18T00:01:19.7572477Z "digest": "sha256:38b3bd834f10691245159178ede4a97949d80aafef88c4da88bc8c59b128c195" 2024-12-18T00:01:19.7572974Z }, 2024-12-18T00:01:19.7573188Z { 2024-12-18T00:01:19.7573523Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7573968Z "size": 655189618, 2024-12-18T00:01:19.7574421Z "digest": "sha256:e45523662aa5eb3b174fdd800641a054a960911c585cea9b0782baf969a4dfa5" 2024-12-18T00:01:19.7574927Z }, 2024-12-18T00:01:19.7575144Z { 2024-12-18T00:01:19.7575478Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7575952Z "size": 704, 2024-12-18T00:01:19.7576461Z "digest": "sha256:5a6d97ee6e47c9a76f0de956b2f0e28bfc3947fdf421470b6202b5ba2702f3a9" 2024-12-18T00:01:19.7576968Z }, 2024-12-18T00:01:19.7577187Z { 2024-12-18T00:01:19.7577544Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7577990Z "size": 1260, 2024-12-18T00:01:19.7578423Z "digest": "sha256:8b9cee92f4081f7ae7b4ac3842ed028a3595044d35c3aabdfdf61fdf414b422c" 2024-12-18T00:01:19.7579010Z }, 2024-12-18T00:01:19.7579394Z { 2024-12-18T00:01:19.7579791Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7580367Z "size": 484, 2024-12-18T00:01:19.7580842Z "digest": "sha256:05a25e0e092fec6d2dd4a72016ea313485058594415703c2be2cc9fb041d483b" 2024-12-18T00:01:19.7581398Z }, 2024-12-18T00:01:19.7581645Z { 2024-12-18T00:01:19.7582039Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7582539Z "size": 91714827, 2024-12-18T00:01:19.7583032Z "digest": "sha256:32f9bd3454b2990eb47fa1b21e333f7e8b560c5c086f531c5ee9389676c218ac" 2024-12-18T00:01:19.7583589Z }, 2024-12-18T00:01:19.7583827Z { 2024-12-18T00:01:19.7584223Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7584717Z "size": 3707, 2024-12-18T00:01:19.7585199Z "digest": "sha256:7cef0be11b192948fbbde9ab3ef1a6d84a3008372f6c8dd6b2bc29b049f76e74" 2024-12-18T00:01:19.7585842Z }, 2024-12-18T00:01:19.7586051Z { 2024-12-18T00:01:19.7586400Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7586841Z "size": 1860, 2024-12-18T00:01:19.7587264Z "digest": "sha256:3a7f27881c57c1035aa6d2109802ac6da11afd93eaa81d48ba8e07f910242248" 2024-12-18T00:01:19.7587758Z }, 2024-12-18T00:01:19.7587971Z { 2024-12-18T00:01:19.7588390Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7588832Z "size": 700, 2024-12-18T00:01:19.7589260Z "digest": "sha256:7feada69620ed9505fc525cf4e6d81f7d25589cb3c26fe8fef7479807372f91a" 2024-12-18T00:01:19.7589761Z }, 2024-12-18T00:01:19.7589977Z { 2024-12-18T00:01:19.7590332Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7590761Z "size": 478, 2024-12-18T00:01:19.7591204Z "digest": "sha256:1f019fb4d944db0c3f6f320a880353d7ebeae6df3456f7e492b7a6c19e7430f2" 2024-12-18T00:01:19.7591706Z }, 2024-12-18T00:01:19.7591916Z { 2024-12-18T00:01:19.7592262Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7592691Z "size": 2953611806, 2024-12-18T00:01:19.7593134Z "digest": "sha256:f900f32845825b74c796aa12c8ca7412b38ae05813a53257f53b49bb2b501bcd" 2024-12-18T00:01:19.7593626Z }, 2024-12-18T00:01:19.7593923Z + exit 0 2024-12-18T00:01:19.7594146Z { 2024-12-18T00:01:19.7594504Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7594939Z "size": 380, 2024-12-18T00:01:19.7595384Z "digest": "sha256:03ef53c6b863b93f161c2bf5eb906c9cae887d4e3cb793fa06212162c09b3584" 2024-12-18T00:01:19.7595890Z }, 2024-12-18T00:01:19.7596107Z { 2024-12-18T00:01:19.7596451Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7596895Z "size": 12909, 2024-12-18T00:01:19.7597337Z "digest": "sha256:8c33d01f9e39d69c625c8a507af96fef2fa8f95ff019b7e060062fbd98da0a3b" 2024-12-18T00:01:19.7597835Z }, 2024-12-18T00:01:19.7598045Z { 2024-12-18T00:01:19.7598382Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7598821Z "size": 863, 2024-12-18T00:01:19.7599255Z "digest": "sha256:49be5867b6ab28064a789b446635d6ec60a1c639deffec9c581499d09a20d31a" 2024-12-18T00:01:19.7599747Z }, 2024-12-18T00:01:19.7599959Z { 2024-12-18T00:01:19.7600295Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7600737Z "size": 106, 2024-12-18T00:01:19.7601169Z "digest": "sha256:2c921c2b926223806ddf443e5a591ae34c19c50f553adea16a2ae986a821b47e" 2024-12-18T00:01:19.7601735Z }, 2024-12-18T00:01:19.7601951Z { 2024-12-18T00:01:19.7602289Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7602733Z "size": 504, 2024-12-18T00:01:19.7603162Z "digest": "sha256:f276569c48f345d048de43d8296869ea278af215142ba5741b765adeac6c238f" 2024-12-18T00:01:19.7603655Z }, 2024-12-18T00:01:19.7603868Z { 2024-12-18T00:01:19.7604200Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7604645Z "size": 121477378, 2024-12-18T00:01:19.7605096Z "digest": "sha256:cac6488770e66c796a9bae0e25b007efd1ff47a360bc5714b7fd2946a39ff59b" 2024-12-18T00:01:19.7605594Z }, 2024-12-18T00:01:19.7605807Z { 2024-12-18T00:01:19.7606144Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7606587Z "size": 109, 2024-12-18T00:01:19.7607016Z "digest": "sha256:9695d05a4951d9c024dd2d0726270c2acd5c81cbb1986ae03f60e167357f43cf" 2024-12-18T00:01:19.7607507Z }, 2024-12-18T00:01:19.7607725Z { 2024-12-18T00:01:19.7608066Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7608510Z "size": 489, 2024-12-18T00:01:19.7608950Z "digest": "sha256:2857ab4d43aab8ad9d8ba6af295f20bfb83a30f5442e8502e1db44c385916241" 2024-12-18T00:01:19.7609451Z }, 2024-12-18T00:01:19.7609662Z { 2024-12-18T00:01:19.7609996Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7652495Z "size": 296, 2024-12-18T00:01:19.7652989Z "digest": "sha256:fcdec1f5e2fde1f1945361f0d3c56b33a33f65d42378ff4ca797060040518bf0" 2024-12-18T00:01:19.7653524Z }, 2024-12-18T00:01:19.7653892Z { 2024-12-18T00:01:19.7654261Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7654735Z "size": 103, 2024-12-18T00:01:19.7655394Z "digest": "sha256:ff1478fbefd979ac80d1b176b2c3f6ddcf4cecf685433398b03bc4f0cbd06110" 2024-12-18T00:01:19.7656034Z }, 2024-12-18T00:01:19.7656233Z { 2024-12-18T00:01:19.7656603Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7657055Z "size": 1476, 2024-12-18T00:01:19.7657512Z "digest": "sha256:f8a6d90f0b3ef982ba39428cb0d40dc4c9e03df7c4011258916d4d5dab1bef27" 2024-12-18T00:01:19.7658034Z }, 2024-12-18T00:01:19.7658239Z { 2024-12-18T00:01:19.7658594Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7659155Z "size": 430258066, 2024-12-18T00:01:19.7659795Z "digest": "sha256:011542937262656639e466e38068de398e4565825a5b1d02acc425bd4da3ed3d" 2024-12-18T00:01:19.7660327Z }, 2024-12-18T00:01:19.7660573Z { 2024-12-18T00:01:19.7660954Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7661440Z "size": 162, 2024-12-18T00:01:19.7661915Z "digest": "sha256:812caf3e433816d6c7bea249be5ff35aa976d1d044d32c02741ac161aade0327" 2024-12-18T00:01:19.7662472Z }, 2024-12-18T00:01:19.7662684Z { 2024-12-18T00:01:19.7663063Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7663546Z "size": 564, 2024-12-18T00:01:19.7664013Z "digest": "sha256:7e2aa240b70cc8e1f87f625b28c79751d71f025760c2402d0c695d6333847c71" 2024-12-18T00:01:19.7664545Z }, 2024-12-18T00:01:19.7664754Z { 2024-12-18T00:01:19.7665124Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7665600Z "size": 35806330, 2024-12-18T00:01:19.7666081Z "digest": "sha256:a0e5fd2630de64c400e6cc5c43094e8ac4550325904f24db8f710842f33cc348" 2024-12-18T00:01:19.7666613Z }, 2024-12-18T00:01:19.7666824Z { 2024-12-18T00:01:19.7667198Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7667680Z "size": 104, 2024-12-18T00:01:19.7668144Z "digest": "sha256:a7253f4f13102b2c05f098066e2c11157d6b24c4cb879b3f3f9a5e29e73aa482" 2024-12-18T00:01:19.7668685Z }, 2024-12-18T00:01:19.7668898Z { 2024-12-18T00:01:19.7669269Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7669870Z "size": 425, 2024-12-18T00:01:19.7670345Z "digest": "sha256:d4bbe33feb5a0018de6d8445ad79aa990043b624c90752a4f264235b00ce9c71" 2024-12-18T00:01:19.7670878Z }, 2024-12-18T00:01:19.7671084Z { 2024-12-18T00:01:19.7671669Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7672097Z "size": 20262180, 2024-12-18T00:01:19.7672545Z "digest": "sha256:c7da3e1fda207aae43390fd25dc953ee6bb0fb29d6f92fe8ea45e25af003040e" 2024-12-18T00:01:19.7673038Z }, 2024-12-18T00:01:19.7673223Z { 2024-12-18T00:01:19.7673552Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7673983Z "size": 643, 2024-12-18T00:01:19.7674406Z "digest": "sha256:a1fa96c8688be0cd5c869e56cb39672774ca1466e3b55b296c07e6ddafc196db" 2024-12-18T00:01:19.7674898Z }, 2024-12-18T00:01:19.7675088Z { 2024-12-18T00:01:19.7675426Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7675854Z "size": 700, 2024-12-18T00:01:19.7676285Z "digest": "sha256:7feada69620ed9505fc525cf4e6d81f7d25589cb3c26fe8fef7479807372f91a" 2024-12-18T00:01:19.7676776Z }, 2024-12-18T00:01:19.7676965Z { 2024-12-18T00:01:19.7677298Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7677727Z "size": 141, 2024-12-18T00:01:19.7678157Z "digest": "sha256:600e7beb6ed65b70903fb3d10a7720f76a0e7ed4225e6a2d974edab4ded9985c" 2024-12-18T00:01:19.7678641Z }, 2024-12-18T00:01:19.7678833Z { 2024-12-18T00:01:19.7679163Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7679588Z "size": 136, 2024-12-18T00:01:19.7680010Z "digest": "sha256:7fee2a426d6e9e46b8fa661021c0cb6cbe7f52186061b61d7f32df2ec7f1dbae" 2024-12-18T00:01:19.7680506Z }, 2024-12-18T00:01:19.7680697Z { 2024-12-18T00:01:19.7681095Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7681525Z "size": 32, 2024-12-18T00:01:19.7681952Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:01:19.7682442Z }, 2024-12-18T00:01:19.7682629Z { 2024-12-18T00:01:19.7682970Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7683392Z "size": 196, 2024-12-18T00:01:19.7683808Z "digest": "sha256:321081f536b035434bb8104a9d93a930eba4415dbeffb4ce926a7c9560a818a9" 2024-12-18T00:01:19.7684276Z }, 2024-12-18T00:01:19.7684476Z { 2024-12-18T00:01:19.7684814Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7685238Z "size": 565, 2024-12-18T00:01:19.7685647Z "digest": "sha256:b2671014f386836740538595eb3d3d98580b1d88d3d63ade2ac59313e53526bd" 2024-12-18T00:01:19.7686111Z }, 2024-12-18T00:01:19.7686311Z { 2024-12-18T00:01:19.7686650Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7687095Z "size": 43163802, 2024-12-18T00:01:19.7687583Z "digest": "sha256:94a3028693ada8037297f5512a3a90c4d307bf06337ef43fcbce578505dbb02d" 2024-12-18T00:01:19.7688069Z }, 2024-12-18T00:01:19.7688271Z { 2024-12-18T00:01:19.7688618Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7689061Z "size": 106, 2024-12-18T00:01:19.7689493Z "digest": "sha256:838f3043a48a3fccad46b59399809bd6a67048404a30c44f8d04272c826634c4" 2024-12-18T00:01:19.7689973Z }, 2024-12-18T00:01:19.7690186Z { 2024-12-18T00:01:19.7690535Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7690976Z "size": 1402, 2024-12-18T00:01:19.7691419Z "digest": "sha256:3bf4f9678a1173c728cf0aa3632bbc5de29b674a2c547de331ecd9d3dd86e8c4" 2024-12-18T00:01:19.7691903Z }, 2024-12-18T00:01:19.7692116Z { 2024-12-18T00:01:19.7692464Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7692911Z "size": 700, 2024-12-18T00:01:19.7693350Z "digest": "sha256:7feada69620ed9505fc525cf4e6d81f7d25589cb3c26fe8fef7479807372f91a" 2024-12-18T00:01:19.7693838Z }, 2024-12-18T00:01:19.7694125Z { 2024-12-18T00:01:19.7694475Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7694916Z "size": 137, 2024-12-18T00:01:19.7695362Z "digest": "sha256:cbd527c6febd14371f782c2752d8947f86adeca1e46e6e1c9dfc0dff6e32df6f" 2024-12-18T00:01:19.7695859Z }, 2024-12-18T00:01:19.7696073Z { 2024-12-18T00:01:19.7696427Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7696867Z "size": 120, 2024-12-18T00:01:19.7697292Z "digest": "sha256:70c75642ca1573a70d7ab208940c68c9ec0518797e55e05b621cef283f348bbd" 2024-12-18T00:01:19.7697772Z }, 2024-12-18T00:01:19.7697982Z { 2024-12-18T00:01:19.7698326Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7698767Z "size": 4886117515, 2024-12-18T00:01:19.7699494Z "digest": "sha256:975ffa249994efd3380d6baae778fb7f9ae31ef5208ceb19da52ab66f447401d" 2024-12-18T00:01:19.7700057Z }, 2024-12-18T00:01:19.7700294Z { 2024-12-18T00:01:19.7700688Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7701185Z "size": 176, 2024-12-18T00:01:19.7701700Z "digest": "sha256:1de45071cffa26dd3b9959d220f37f09e3ebc5282b49c8910ca1b7045f37c763" 2024-12-18T00:01:19.7702254Z }, 2024-12-18T00:01:19.7702492Z { 2024-12-18T00:01:19.7702878Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7703372Z "size": 908, 2024-12-18T00:01:19.7703847Z "digest": "sha256:6f1f1c73799125dbfc36617f78ae22c9507955fb3870b11886da704f63cd79b9" 2024-12-18T00:01:19.7704387Z }, 2024-12-18T00:01:19.7704619Z { 2024-12-18T00:01:19.7705009Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7705502Z "size": 700, 2024-12-18T00:01:19.7706060Z "digest": "sha256:7feada69620ed9505fc525cf4e6d81f7d25589cb3c26fe8fef7479807372f91a" 2024-12-18T00:01:19.7706608Z }, 2024-12-18T00:01:19.7706842Z { 2024-12-18T00:01:19.7707238Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7707739Z "size": 135, 2024-12-18T00:01:19.7708225Z "digest": "sha256:8da387044f457792831b74e2facde5dc612cb058a3fcc22e503a778ad0e9125e" 2024-12-18T00:01:19.7708769Z }, 2024-12-18T00:01:19.7709005Z { 2024-12-18T00:01:19.7709394Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7709890Z "size": 32, 2024-12-18T00:01:19.7710374Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:01:19.7710923Z }, 2024-12-18T00:01:19.7711267Z { 2024-12-18T00:01:19.7711644Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7712123Z "size": 159, 2024-12-18T00:01:19.7712593Z "digest": "sha256:25b1aa58f5caff295c81a716174c70d1207b7efc44a594cb4b4c730d11ca6f58" 2024-12-18T00:01:19.7713123Z }, 2024-12-18T00:01:19.7713459Z { 2024-12-18T00:01:19.7713829Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7714401Z "size": 1931, 2024-12-18T00:01:19.7714840Z "digest": "sha256:f4c8ff23e407e7f06a6b9e83db042522c9cb14a2872e033bf0aec10e0cd68af8" 2024-12-18T00:01:19.7715328Z }, 2024-12-18T00:01:19.7715540Z { 2024-12-18T00:01:19.7715887Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7716329Z "size": 196807254, 2024-12-18T00:01:19.7716787Z "digest": "sha256:d9a1d16ac9fb7fb0a5b04cf9bd8a051fedd0f6ef1467103fc8796790e941e1bb" 2024-12-18T00:01:19.7717278Z }, 2024-12-18T00:01:19.7717490Z { 2024-12-18T00:01:19.7717834Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7718275Z "size": 163, 2024-12-18T00:01:19.7718694Z "digest": "sha256:4c24861203037ee0fc72cda05c5446739232f8fc027622a4982a392434699737" 2024-12-18T00:01:19.7719169Z }, 2024-12-18T00:01:19.7719382Z { 2024-12-18T00:01:19.7719728Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7720171Z "size": 7942, 2024-12-18T00:01:19.7720690Z "digest": "sha256:1c1a1b6b59a8cfdf0a6b0dd0fbf35ea01e8d3d9fbfbf7a1cc3c5129697842953" 2024-12-18T00:01:19.7721191Z }, 2024-12-18T00:01:19.7721405Z { 2024-12-18T00:01:19.7721756Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7722208Z "size": 8071, 2024-12-18T00:01:19.7722741Z "digest": "sha256:65ab75609fc9c1a8878dee49da05ea4c3cae04ec5ecd96bff7a59ba5f6716161" 2024-12-18T00:01:19.7723232Z }, 2024-12-18T00:01:19.7723446Z { 2024-12-18T00:01:19.7723792Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7724232Z "size": 304, 2024-12-18T00:01:19.7724662Z "digest": "sha256:41fe6d82c37a17ff1a0e760821e5ffd58423b2c50138348b2d99214fa8c7f7b9" 2024-12-18T00:01:19.7725143Z }, 2024-12-18T00:01:19.7725361Z { 2024-12-18T00:01:19.7725707Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7726152Z "size": 7632783, 2024-12-18T00:01:19.7726597Z "digest": "sha256:9456ad12e83f1f01e0a64ef29e45b8ce1657f0c45a5f334956fed566782f34d9" 2024-12-18T00:01:19.7727086Z }, 2024-12-18T00:01:19.7727301Z { 2024-12-18T00:01:19.7727659Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7728104Z "size": 108, 2024-12-18T00:01:19.7728547Z "digest": "sha256:7dd812a149578ae7aa4d747b9c0cfdd91a1edc15b63e7ec826ca42a601c7c03f" 2024-12-18T00:01:19.7729039Z }, 2024-12-18T00:01:19.7729256Z { 2024-12-18T00:01:19.7729611Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7730056Z "size": 54145664, 2024-12-18T00:01:19.7730515Z "digest": "sha256:e47d2dd30bb9d7d5f7295bbf4b68683cf7e0a3ec443e3d47696213edb61bb0ad" 2024-12-18T00:01:19.7731009Z }, 2024-12-18T00:01:19.7731225Z { 2024-12-18T00:01:19.7732004Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7732671Z "size": 486, 2024-12-18T00:01:19.7733230Z "digest": "sha256:9a255e925aee3dd6711c86af0e487d1c5ff1b2cde447bd2f64f5cc6aa08a3666" 2024-12-18T00:01:19.7733783Z }, 2024-12-18T00:01:19.7734020Z { 2024-12-18T00:01:19.7734410Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7734907Z "size": 1373229207, 2024-12-18T00:01:19.7735405Z "digest": "sha256:266e2409e545ef9c13ef48d69773c27f5ebe37e17c77808c473870b99ccaceca" 2024-12-18T00:01:19.7735939Z }, 2024-12-18T00:01:19.7736175Z { 2024-12-18T00:01:19.7736564Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7737055Z "size": 106, 2024-12-18T00:01:19.7737519Z "digest": "sha256:100d759b6098f93451cdf0ab691600451c05cbf8de2592d370f32f6be20b6127" 2024-12-18T00:01:19.7738071Z }, 2024-12-18T00:01:19.7738308Z { 2024-12-18T00:01:19.7738695Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7739265Z "size": 568, 2024-12-18T00:01:19.7739738Z "digest": "sha256:a3111847bfa2e9e8629c6c7a6cbdd52dde3359882add44762805cb47464c285f" 2024-12-18T00:01:19.7740306Z }, 2024-12-18T00:01:19.7740544Z { 2024-12-18T00:01:19.7740935Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7741427Z "size": 36532350, 2024-12-18T00:01:19.7741916Z "digest": "sha256:7e6ea32f1b1cfa63f3df243c6bfa8b2659be396b488adcbdf9a9c0d9a1e65193" 2024-12-18T00:01:19.7742482Z }, 2024-12-18T00:01:19.7742715Z { 2024-12-18T00:01:19.7743277Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7743767Z "size": 111, 2024-12-18T00:01:19.7744236Z "digest": "sha256:38bf4ff6a6aca492ad60288413a8a2462e07f715d05252617bb4fa942c0ca565" 2024-12-18T00:01:19.7744790Z }, 2024-12-18T00:01:19.7745025Z { 2024-12-18T00:01:19.7745406Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7745891Z "size": 531, 2024-12-18T00:01:19.7746348Z "digest": "sha256:231d0f8577895b4225081015421c272a6f6c601e9b367fcf49106b9a8e3ff74c" 2024-12-18T00:01:19.7746890Z }, 2024-12-18T00:01:19.7747249Z { 2024-12-18T00:01:19.7747649Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7748138Z "size": 26033, 2024-12-18T00:01:19.7748633Z "digest": "sha256:0fe4c18889ad4b2d23fac8e5c1a1bba4cc65ed4abf30d84ab871f507fd27ee03" 2024-12-18T00:01:19.7749205Z }, 2024-12-18T00:01:19.7749444Z { 2024-12-18T00:01:19.7749839Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7750336Z "size": 106, 2024-12-18T00:01:19.7751015Z "digest": "sha256:c5d910c7fb9c4e239cfeafa9695f9e8b9a2892da466239b362451bb0d9a539df" 2024-12-18T00:01:19.7751632Z }, 2024-12-18T00:01:19.7751833Z { 2024-12-18T00:01:19.7752169Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7752606Z "size": 32, 2024-12-18T00:01:19.7753024Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:01:19.7753526Z }, 2024-12-18T00:01:19.7753740Z { 2024-12-18T00:01:19.7754090Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7754529Z "size": 32, 2024-12-18T00:01:19.7754946Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:01:19.7755446Z }, 2024-12-18T00:01:19.7755655Z { 2024-12-18T00:01:19.7756000Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7756441Z "size": 32, 2024-12-18T00:01:19.7756862Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:01:19.7757360Z }, 2024-12-18T00:01:19.7757568Z { 2024-12-18T00:01:19.7757910Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:19.7758348Z "size": 32, 2024-12-18T00:01:19.7758769Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:01:19.7759337Z } 2024-12-18T00:01:19.7759552Z ] 2024-12-18T00:01:19.7759766Z } 2024-12-18T00:01:19.7795602Z ##[group]Run set -eux 2024-12-18T00:01:19.7795919Z set -eux 2024-12-18T00:01:19.7796883Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin 2024-12-18T00:01:19.7803441Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:19.7803838Z env: 2024-12-18T00:01:19.7804055Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:19.7804334Z ##[endgroup] 2024-12-18T00:01:19.7831868Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2024-12-18T00:01:19.7832468Z + jq --raw-output .SecretString 2024-12-18T00:01:19.7833317Z + jq -r .docker_hub_readonly_token 2024-12-18T00:01:19.7834637Z + docker login --username pytorchbot --password-stdin 2024-12-18T00:01:20.4496588Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:01:20.4497344Z Configure a credential helper to remove this warning. See 2024-12-18T00:01:20.4498182Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:01:20.4498629Z 2024-12-18T00:01:20.4498758Z Login Succeeded 2024-12-18T00:01:20.4594815Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-12-18T00:01:20.4595200Z tag=${ECR_DOCKER_IMAGE##*/} 2024-12-18T00:01:20.4595632Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-12-18T00:01:20.4601423Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:20.4601832Z env: 2024-12-18T00:01:20.4602073Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:20.4602877Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:20.4603702Z ##[endgroup] 2024-12-18T00:01:20.4630451Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9-45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:20.4684009Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@release/2.6 2024-12-18T00:01:20.4684502Z with: 2024-12-18T00:01:20.4685241Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:20.4686153Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:20.4686573Z env: 2024-12-18T00:01:20.4686803Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:20.4687089Z ##[endgroup] 2024-12-18T00:01:20.4715099Z ##[group]Run set -x 2024-12-18T00:01:20.4715414Z set -x 2024-12-18T00:01:20.4715679Z set +e 2024-12-18T00:01:20.4715924Z  2024-12-18T00:01:20.4716179Z login() { 2024-12-18T00:01:20.4716722Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-12-18T00:01:20.4717316Z } 2024-12-18T00:01:20.4717561Z  2024-12-18T00:01:20.4717852Z retry () { 2024-12-18T00:01:20.4718167Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-12-18T00:01:20.4718516Z } 2024-12-18T00:01:20.4718760Z  2024-12-18T00:01:20.4719032Z retry login "${DOCKER_REGISTRY}" 2024-12-18T00:01:20.4719388Z  2024-12-18T00:01:20.4719635Z set -e 2024-12-18T00:01:20.4720014Z # ignore output since only exit code is used for conditional 2024-12-18T00:01:20.4720652Z # only pull docker image if it's not available locally 2024-12-18T00:01:20.4721232Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-12-18T00:01:20.4721771Z  retry docker pull "${DOCKER_IMAGE}" 2024-12-18T00:01:20.4722117Z fi 2024-12-18T00:01:20.4727127Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:20.4727651Z env: 2024-12-18T00:01:20.4727896Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:20.4728691Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:20.4729611Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:20.4730027Z ##[endgroup] 2024-12-18T00:01:20.4753076Z + set +e 2024-12-18T00:01:20.4753473Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:20.4753983Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:20.4756970Z + aws ecr get-login-password --region us-east-1 2024-12-18T00:01:20.4758083Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:21.0430824Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:01:21.0431827Z Configure a credential helper to remove this warning. See 2024-12-18T00:01:21.0432729Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:01:21.0433195Z 2024-12-18T00:01:21.0433343Z Login Succeeded 2024-12-18T00:01:21.0447671Z + set -e 2024-12-18T00:01:21.0448688Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:21.0585629Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:21.0587146Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:21.4169209Z 45e1356b47a284893081276eff3000b7b534f3b1: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9 2024-12-18T00:01:21.4169962Z 7a2c55901189: Pulling fs layer 2024-12-18T00:01:21.4170579Z 09d415c238d7: Pulling fs layer 2024-12-18T00:01:21.4171105Z 9fe6e2e61518: Pulling fs layer 2024-12-18T00:01:21.4171626Z 41f16248e682: Pulling fs layer 2024-12-18T00:01:21.4172332Z 95d7b7817039: Pulling fs layer 2024-12-18T00:01:21.4172674Z 8f6c90485347: Pulling fs layer 2024-12-18T00:01:21.4173005Z ab17245097e4: Pulling fs layer 2024-12-18T00:01:21.4173486Z dfecd7e9912b: Pulling fs layer 2024-12-18T00:01:21.4174073Z 464a8f745445: Pulling fs layer 2024-12-18T00:01:21.4174461Z d67c111fa588: Pulling fs layer 2024-12-18T00:01:21.4174782Z 95d7b7817039: Waiting 2024-12-18T00:01:21.4175079Z 0723abde19c0: Pulling fs layer 2024-12-18T00:01:21.4175446Z 38b3bd834f10: Pulling fs layer 2024-12-18T00:01:21.4175776Z 8f6c90485347: Waiting 2024-12-18T00:01:21.4176069Z e45523662aa5: Pulling fs layer 2024-12-18T00:01:21.4176401Z 5a6d97ee6e47: Pulling fs layer 2024-12-18T00:01:21.4176721Z ab17245097e4: Waiting 2024-12-18T00:01:21.4177018Z 8b9cee92f408: Pulling fs layer 2024-12-18T00:01:21.4177338Z 05a25e0e092f: Pulling fs layer 2024-12-18T00:01:21.4177683Z 32f9bd3454b2: Pulling fs layer 2024-12-18T00:01:21.4178019Z 7cef0be11b19: Pulling fs layer 2024-12-18T00:01:21.4178347Z 464a8f745445: Waiting 2024-12-18T00:01:21.4179105Z 3a7f27881c57: Pulling fs layer 2024-12-18T00:01:21.4179690Z 7feada69620e: Pulling fs layer 2024-12-18T00:01:21.4180149Z 1f019fb4d944: Pulling fs layer 2024-12-18T00:01:21.4180681Z 05a25e0e092f: Waiting 2024-12-18T00:01:21.4181157Z 32f9bd3454b2: Waiting 2024-12-18T00:01:21.4181600Z f900f3284582: Pulling fs layer 2024-12-18T00:01:21.4182076Z 03ef53c6b863: Pulling fs layer 2024-12-18T00:01:21.4182475Z 7feada69620e: Waiting 2024-12-18T00:01:21.4182914Z 7cef0be11b19: Waiting 2024-12-18T00:01:21.4183358Z 8c33d01f9e39: Pulling fs layer 2024-12-18T00:01:21.4183807Z 49be5867b6ab: Pulling fs layer 2024-12-18T00:01:21.4184408Z 1f019fb4d944: Waiting 2024-12-18T00:01:21.4184760Z e45523662aa5: Waiting 2024-12-18T00:01:21.4185159Z 2c921c2b9262: Pulling fs layer 2024-12-18T00:01:21.4185638Z f276569c48f3: Pulling fs layer 2024-12-18T00:01:21.4186242Z dfecd7e9912b: Waiting 2024-12-18T00:01:21.4186626Z cac6488770e6: Pulling fs layer 2024-12-18T00:01:21.4187138Z 9695d05a4951: Pulling fs layer 2024-12-18T00:01:21.4187533Z 2857ab4d43aa: Pulling fs layer 2024-12-18T00:01:21.4187937Z 0723abde19c0: Waiting 2024-12-18T00:01:21.4188422Z fcdec1f5e2fd: Pulling fs layer 2024-12-18T00:01:21.4188851Z 5a6d97ee6e47: Waiting 2024-12-18T00:01:21.4189296Z ff1478fbefd9: Pulling fs layer 2024-12-18T00:01:21.4189702Z 38b3bd834f10: Waiting 2024-12-18T00:01:21.4190109Z f8a6d90f0b3e: Pulling fs layer 2024-12-18T00:01:21.4190593Z 8b9cee92f408: Waiting 2024-12-18T00:01:21.4191001Z f276569c48f3: Waiting 2024-12-18T00:01:21.4191392Z 03ef53c6b863: Waiting 2024-12-18T00:01:21.4191772Z 011542937262: Pulling fs layer 2024-12-18T00:01:21.4192214Z 2c921c2b9262: Waiting 2024-12-18T00:01:21.4192620Z 812caf3e4338: Pulling fs layer 2024-12-18T00:01:21.4193086Z 49be5867b6ab: Waiting 2024-12-18T00:01:21.4193487Z 7e2aa240b70c: Pulling fs layer 2024-12-18T00:01:21.4193899Z a0e5fd2630de: Pulling fs layer 2024-12-18T00:01:21.4194365Z f8a6d90f0b3e: Waiting 2024-12-18T00:01:21.4194753Z f900f3284582: Waiting 2024-12-18T00:01:21.4195146Z a7253f4f1310: Pulling fs layer 2024-12-18T00:01:21.4195649Z d4bbe33feb5a: Pulling fs layer 2024-12-18T00:01:21.4196025Z 011542937262: Waiting 2024-12-18T00:01:21.4196399Z 812caf3e4338: Waiting 2024-12-18T00:01:21.4196865Z 7e2aa240b70c: Waiting 2024-12-18T00:01:21.4197279Z c7da3e1fda20: Pulling fs layer 2024-12-18T00:01:21.4197741Z a1fa96c8688b: Pulling fs layer 2024-12-18T00:01:21.4198149Z 600e7beb6ed6: Pulling fs layer 2024-12-18T00:01:21.4198582Z 8c33d01f9e39: Waiting 2024-12-18T00:01:21.4199003Z 7fee2a426d6e: Pulling fs layer 2024-12-18T00:01:21.4199467Z 4f4fb700ef54: Pulling fs layer 2024-12-18T00:01:21.4199890Z a0e5fd2630de: Waiting 2024-12-18T00:01:21.4200258Z 321081f536b0: Pulling fs layer 2024-12-18T00:01:21.4200713Z c7da3e1fda20: Waiting 2024-12-18T00:01:21.4201113Z b2671014f386: Pulling fs layer 2024-12-18T00:01:21.4201576Z 41f16248e682: Waiting 2024-12-18T00:01:21.4201987Z 94a3028693ad: Pulling fs layer 2024-12-18T00:01:21.4202376Z 838f3043a48a: Pulling fs layer 2024-12-18T00:01:21.4202966Z 600e7beb6ed6: Waiting 2024-12-18T00:01:21.4203372Z 3bf4f9678a11: Pulling fs layer 2024-12-18T00:01:21.4203782Z fcdec1f5e2fd: Waiting 2024-12-18T00:01:21.4204259Z a1fa96c8688b: Waiting 2024-12-18T00:01:21.4204604Z ff1478fbefd9: Waiting 2024-12-18T00:01:21.4204988Z cbd527c6febd: Pulling fs layer 2024-12-18T00:01:21.4205488Z 94a3028693ad: Waiting 2024-12-18T00:01:21.4205934Z 70c75642ca15: Pulling fs layer 2024-12-18T00:01:21.4206381Z 3bf4f9678a11: Waiting 2024-12-18T00:01:21.4206736Z cac6488770e6: Waiting 2024-12-18T00:01:21.4207132Z 975ffa249994: Pulling fs layer 2024-12-18T00:01:21.4207574Z 321081f536b0: Waiting 2024-12-18T00:01:21.4207983Z 1de45071cffa: Pulling fs layer 2024-12-18T00:01:21.4208407Z 2857ab4d43aa: Waiting 2024-12-18T00:01:21.4208769Z 838f3043a48a: Waiting 2024-12-18T00:01:21.4209162Z cbd527c6febd: Waiting 2024-12-18T00:01:21.4209562Z 6f1f1c737991: Pulling fs layer 2024-12-18T00:01:21.4210063Z 8da387044f45: Pulling fs layer 2024-12-18T00:01:21.4210592Z d4bbe33feb5a: Waiting 2024-12-18T00:01:21.4210951Z a7253f4f1310: Waiting 2024-12-18T00:01:21.4211393Z 25b1aa58f5ca: Pulling fs layer 2024-12-18T00:01:21.4211830Z f4c8ff23e407: Pulling fs layer 2024-12-18T00:01:21.4212237Z 25b1aa58f5ca: Waiting 2024-12-18T00:01:21.4212700Z d9a1d16ac9fb: Pulling fs layer 2024-12-18T00:01:21.4213097Z 4c2486120303: Pulling fs layer 2024-12-18T00:01:21.4213494Z b2671014f386: Waiting 2024-12-18T00:01:21.4213939Z 9695d05a4951: Waiting 2024-12-18T00:01:21.4214345Z 1c1a1b6b59a8: Pulling fs layer 2024-12-18T00:01:21.4214825Z 65ab75609fc9: Pulling fs layer 2024-12-18T00:01:21.4215218Z 70c75642ca15: Waiting 2024-12-18T00:01:21.4215688Z 41fe6d82c37a: Pulling fs layer 2024-12-18T00:01:21.4229714Z 4c2486120303: Waiting 2024-12-18T00:01:21.4230116Z 9456ad12e83f: Pulling fs layer 2024-12-18T00:01:21.4230472Z 65ab75609fc9: Waiting 2024-12-18T00:01:21.4230776Z 41fe6d82c37a: Waiting 2024-12-18T00:01:21.4231241Z 7dd812a14957: Pulling fs layer 2024-12-18T00:01:21.4231872Z e47d2dd30bb9: Pulling fs layer 2024-12-18T00:01:21.4232294Z 9a255e925aee: Pulling fs layer 2024-12-18T00:01:21.4232636Z e47d2dd30bb9: Waiting 2024-12-18T00:01:21.4232946Z 266e2409e545: Pulling fs layer 2024-12-18T00:01:21.4233275Z 975ffa249994: Waiting 2024-12-18T00:01:21.4233569Z 100d759b6098: Pulling fs layer 2024-12-18T00:01:21.4233906Z 9a255e925aee: Waiting 2024-12-18T00:01:21.4234210Z a3111847bfa2: Pulling fs layer 2024-12-18T00:01:21.4234538Z 266e2409e545: Waiting 2024-12-18T00:01:21.4234843Z 7e6ea32f1b1c: Pulling fs layer 2024-12-18T00:01:21.4235180Z 38bf4ff6a6ac: Pulling fs layer 2024-12-18T00:01:21.4235513Z 6f1f1c737991: Waiting 2024-12-18T00:01:21.4235810Z 7e6ea32f1b1c: Waiting 2024-12-18T00:01:21.4236114Z 231d0f857789: Pulling fs layer 2024-12-18T00:01:21.4236432Z 38bf4ff6a6ac: Waiting 2024-12-18T00:01:21.4236743Z 0fe4c18889ad: Pulling fs layer 2024-12-18T00:01:21.4237073Z 231d0f857789: Waiting 2024-12-18T00:01:21.4237369Z 8da387044f45: Waiting 2024-12-18T00:01:21.4237672Z c5d910c7fb9c: Pulling fs layer 2024-12-18T00:01:21.4237988Z 0fe4c18889ad: Waiting 2024-12-18T00:01:21.4238288Z c5d910c7fb9c: Waiting 2024-12-18T00:01:21.4238582Z 7fee2a426d6e: Waiting 2024-12-18T00:01:21.4238873Z 100d759b6098: Waiting 2024-12-18T00:01:21.5742681Z 09d415c238d7: Verifying Checksum 2024-12-18T00:01:21.5743095Z 09d415c238d7: Download complete 2024-12-18T00:01:21.7090516Z 41f16248e682: Verifying Checksum 2024-12-18T00:01:21.7091212Z 41f16248e682: Download complete 2024-12-18T00:01:21.7556922Z 7a2c55901189: Verifying Checksum 2024-12-18T00:01:21.7557882Z 7a2c55901189: Download complete 2024-12-18T00:01:21.8058979Z 95d7b7817039: Verifying Checksum 2024-12-18T00:01:21.8059602Z 95d7b7817039: Download complete 2024-12-18T00:01:21.8900321Z ab17245097e4: Download complete 2024-12-18T00:01:21.9789026Z dfecd7e9912b: Verifying Checksum 2024-12-18T00:01:21.9789488Z dfecd7e9912b: Download complete 2024-12-18T00:01:22.0437813Z 9fe6e2e61518: Verifying Checksum 2024-12-18T00:01:22.0438306Z 9fe6e2e61518: Download complete 2024-12-18T00:01:22.0813575Z 464a8f745445: Verifying Checksum 2024-12-18T00:01:22.0814307Z 464a8f745445: Download complete 2024-12-18T00:01:22.1997653Z 0723abde19c0: Download complete 2024-12-18T00:01:22.3020188Z 38b3bd834f10: Verifying Checksum 2024-12-18T00:01:22.3020640Z 38b3bd834f10: Download complete 2024-12-18T00:01:22.4824964Z 7a2c55901189: Pull complete 2024-12-18T00:01:22.6795828Z 09d415c238d7: Pull complete 2024-12-18T00:01:23.4198373Z 9fe6e2e61518: Pull complete 2024-12-18T00:01:23.4337511Z 41f16248e682: Pull complete 2024-12-18T00:01:23.4480828Z 95d7b7817039: Pull complete 2024-12-18T00:01:29.1602552Z e45523662aa5: Verifying Checksum 2024-12-18T00:01:29.1603006Z e45523662aa5: Download complete 2024-12-18T00:01:29.2450187Z 5a6d97ee6e47: Download complete 2024-12-18T00:01:29.3377110Z 8b9cee92f408: Verifying Checksum 2024-12-18T00:01:29.3377933Z 8b9cee92f408: Download complete 2024-12-18T00:01:29.4327138Z 05a25e0e092f: Verifying Checksum 2024-12-18T00:01:29.4328181Z 05a25e0e092f: Download complete 2024-12-18T00:01:30.5663802Z 32f9bd3454b2: Verifying Checksum 2024-12-18T00:01:30.5664292Z 32f9bd3454b2: Download complete 2024-12-18T00:01:30.6549554Z 7cef0be11b19: Download complete 2024-12-18T00:01:30.7857554Z 3a7f27881c57: Verifying Checksum 2024-12-18T00:01:30.7857988Z 3a7f27881c57: Download complete 2024-12-18T00:01:30.8846080Z 7feada69620e: Verifying Checksum 2024-12-18T00:01:30.8846887Z 7feada69620e: Download complete 2024-12-18T00:01:30.9942805Z 1f019fb4d944: Verifying Checksum 2024-12-18T00:01:30.9943231Z 1f019fb4d944: Download complete 2024-12-18T00:01:35.7112064Z 8f6c90485347: Verifying Checksum 2024-12-18T00:01:35.7112494Z 8f6c90485347: Download complete 2024-12-18T00:01:35.7902264Z 03ef53c6b863: Verifying Checksum 2024-12-18T00:01:35.7902700Z 03ef53c6b863: Download complete 2024-12-18T00:01:35.8919536Z 8c33d01f9e39: Download complete 2024-12-18T00:01:35.9840697Z 49be5867b6ab: Verifying Checksum 2024-12-18T00:01:35.9841425Z 49be5867b6ab: Download complete 2024-12-18T00:01:36.1007465Z 2c921c2b9262: Download complete 2024-12-18T00:01:36.2084352Z f276569c48f3: Download complete 2024-12-18T00:01:37.6316556Z cac6488770e6: Verifying Checksum 2024-12-18T00:01:37.6316990Z cac6488770e6: Download complete 2024-12-18T00:01:37.7504274Z 9695d05a4951: Download complete 2024-12-18T00:01:37.8403382Z 2857ab4d43aa: Download complete 2024-12-18T00:01:37.9489950Z fcdec1f5e2fd: Verifying Checksum 2024-12-18T00:01:37.9490414Z fcdec1f5e2fd: Download complete 2024-12-18T00:01:38.1143425Z ff1478fbefd9: Download complete 2024-12-18T00:01:38.1992794Z f8a6d90f0b3e: Verifying Checksum 2024-12-18T00:01:38.1993209Z f8a6d90f0b3e: Download complete 2024-12-18T00:01:43.3794739Z 011542937262: Verifying Checksum 2024-12-18T00:01:43.3795149Z 011542937262: Download complete 2024-12-18T00:01:43.5330198Z 812caf3e4338: Download complete 2024-12-18T00:01:43.6297515Z 7e2aa240b70c: Verifying Checksum 2024-12-18T00:01:43.6298900Z 7e2aa240b70c: Download complete 2024-12-18T00:01:44.0607957Z a0e5fd2630de: Verifying Checksum 2024-12-18T00:01:44.0608837Z a0e5fd2630de: Download complete 2024-12-18T00:01:44.1514997Z a7253f4f1310: Verifying Checksum 2024-12-18T00:01:44.1515401Z a7253f4f1310: Download complete 2024-12-18T00:01:44.2332837Z d4bbe33feb5a: Verifying Checksum 2024-12-18T00:01:44.2333492Z d4bbe33feb5a: Download complete 2024-12-18T00:01:44.5041655Z c7da3e1fda20: Verifying Checksum 2024-12-18T00:01:44.5042107Z c7da3e1fda20: Download complete 2024-12-18T00:01:44.6266572Z a1fa96c8688b: Verifying Checksum 2024-12-18T00:01:44.6267071Z a1fa96c8688b: Download complete 2024-12-18T00:01:44.7430413Z 600e7beb6ed6: Verifying Checksum 2024-12-18T00:01:44.7431032Z 600e7beb6ed6: Download complete 2024-12-18T00:01:44.8162718Z 7fee2a426d6e: Download complete 2024-12-18T00:01:44.8257711Z 4f4fb700ef54: Verifying Checksum 2024-12-18T00:01:44.8258360Z 4f4fb700ef54: Download complete 2024-12-18T00:01:44.9261860Z 321081f536b0: Download complete 2024-12-18T00:01:45.0077514Z b2671014f386: Verifying Checksum 2024-12-18T00:01:45.0078190Z b2671014f386: Download complete 2024-12-18T00:01:45.5088165Z 94a3028693ad: Verifying Checksum 2024-12-18T00:01:45.5089079Z 94a3028693ad: Download complete 2024-12-18T00:01:45.6581262Z 838f3043a48a: Verifying Checksum 2024-12-18T00:01:45.6581921Z 838f3043a48a: Download complete 2024-12-18T00:01:45.7370404Z 3bf4f9678a11: Download complete 2024-12-18T00:01:45.8229820Z cbd527c6febd: Verifying Checksum 2024-12-18T00:01:45.8230489Z cbd527c6febd: Download complete 2024-12-18T00:01:45.9384327Z 70c75642ca15: Download complete 2024-12-18T00:01:46.5919403Z d67c111fa588: Verifying Checksum 2024-12-18T00:01:46.5920240Z d67c111fa588: Download complete 2024-12-18T00:01:46.7168332Z 1de45071cffa: Verifying Checksum 2024-12-18T00:01:46.7168965Z 1de45071cffa: Download complete 2024-12-18T00:01:46.8655097Z 6f1f1c737991: Verifying Checksum 2024-12-18T00:01:46.8655737Z 6f1f1c737991: Download complete 2024-12-18T00:01:46.9528236Z 8da387044f45: Verifying Checksum 2024-12-18T00:01:46.9529088Z 8da387044f45: Download complete 2024-12-18T00:01:47.0551313Z 25b1aa58f5ca: Verifying Checksum 2024-12-18T00:01:47.0551947Z 25b1aa58f5ca: Download complete 2024-12-18T00:01:47.1649264Z f4c8ff23e407: Verifying Checksum 2024-12-18T00:01:47.1649912Z f4c8ff23e407: Download complete 2024-12-18T00:01:49.2081671Z d9a1d16ac9fb: Verifying Checksum 2024-12-18T00:01:49.2082119Z d9a1d16ac9fb: Download complete 2024-12-18T00:01:49.3354549Z 4c2486120303: Verifying Checksum 2024-12-18T00:01:49.3355052Z 4c2486120303: Download complete 2024-12-18T00:01:49.4943230Z 1c1a1b6b59a8: Verifying Checksum 2024-12-18T00:01:49.4944189Z 1c1a1b6b59a8: Download complete 2024-12-18T00:01:49.6022018Z 65ab75609fc9: Download complete 2024-12-18T00:01:49.6774713Z 41fe6d82c37a: Verifying Checksum 2024-12-18T00:01:49.6775647Z 41fe6d82c37a: Download complete 2024-12-18T00:01:49.8643023Z 8f6c90485347: Pull complete 2024-12-18T00:01:49.8993764Z 9456ad12e83f: Verifying Checksum 2024-12-18T00:01:49.8994271Z 9456ad12e83f: Download complete 2024-12-18T00:01:49.9876953Z 7dd812a14957: Verifying Checksum 2024-12-18T00:01:49.9877931Z 7dd812a14957: Download complete 2024-12-18T00:01:50.2949383Z ab17245097e4: Pull complete 2024-12-18T00:01:50.6146753Z e47d2dd30bb9: Verifying Checksum 2024-12-18T00:01:50.6147195Z e47d2dd30bb9: Download complete 2024-12-18T00:01:50.6986668Z 9a255e925aee: Download complete 2024-12-18T00:01:50.7017751Z dfecd7e9912b: Pull complete 2024-12-18T00:01:51.1633556Z 464a8f745445: Pull complete 2024-12-18T00:02:00.6139877Z f900f3284582: Verifying Checksum 2024-12-18T00:02:00.6140395Z f900f3284582: Download complete 2024-12-18T00:02:00.6998315Z 100d759b6098: Verifying Checksum 2024-12-18T00:02:00.6998883Z 100d759b6098: Download complete 2024-12-18T00:02:00.7696326Z a3111847bfa2: Verifying Checksum 2024-12-18T00:02:00.7696755Z a3111847bfa2: Download complete 2024-12-18T00:02:01.2096590Z 7e6ea32f1b1c: Verifying Checksum 2024-12-18T00:02:01.2097043Z 7e6ea32f1b1c: Download complete 2024-12-18T00:02:01.3227732Z 38bf4ff6a6ac: Verifying Checksum 2024-12-18T00:02:01.3228200Z 38bf4ff6a6ac: Download complete 2024-12-18T00:02:01.4701534Z 231d0f857789: Verifying Checksum 2024-12-18T00:02:01.4702020Z 231d0f857789: Download complete 2024-12-18T00:02:01.5575296Z 0fe4c18889ad: Download complete 2024-12-18T00:02:01.7135220Z c5d910c7fb9c: Verifying Checksum 2024-12-18T00:02:01.7135671Z c5d910c7fb9c: Download complete 2024-12-18T00:02:04.5034999Z 266e2409e545: Verifying Checksum 2024-12-18T00:02:04.5035460Z 266e2409e545: Download complete 2024-12-18T00:02:16.2265246Z d67c111fa588: Pull complete 2024-12-18T00:02:16.6396188Z 0723abde19c0: Pull complete 2024-12-18T00:02:17.0944352Z 38b3bd834f10: Pull complete 2024-12-18T00:02:26.5848871Z e45523662aa5: Pull complete 2024-12-18T00:02:27.0463993Z 5a6d97ee6e47: Pull complete 2024-12-18T00:02:27.4827748Z 8b9cee92f408: Pull complete 2024-12-18T00:02:27.9404693Z 05a25e0e092f: Pull complete 2024-12-18T00:02:30.2142281Z 32f9bd3454b2: Pull complete 2024-12-18T00:02:30.6038118Z 7cef0be11b19: Pull complete 2024-12-18T00:02:31.0579931Z 3a7f27881c57: Pull complete 2024-12-18T00:02:31.4807982Z 7feada69620e: Pull complete 2024-12-18T00:02:31.9036190Z 1f019fb4d944: Pull complete 2024-12-18T00:02:35.0036979Z 975ffa249994: Verifying Checksum 2024-12-18T00:02:35.0037431Z 975ffa249994: Download complete 2024-12-18T00:03:13.6645546Z f900f3284582: Pull complete 2024-12-18T00:03:13.9818913Z 03ef53c6b863: Pull complete 2024-12-18T00:03:14.3395217Z 8c33d01f9e39: Pull complete 2024-12-18T00:03:14.7931047Z 49be5867b6ab: Pull complete 2024-12-18T00:03:15.1883938Z 2c921c2b9262: Pull complete 2024-12-18T00:03:15.2896216Z f276569c48f3: Pull complete 2024-12-18T00:03:18.1776711Z cac6488770e6: Pull complete 2024-12-18T00:03:18.6504246Z 9695d05a4951: Pull complete 2024-12-18T00:03:19.1032039Z 2857ab4d43aa: Pull complete 2024-12-18T00:03:19.5704150Z fcdec1f5e2fd: Pull complete 2024-12-18T00:03:20.0213005Z ff1478fbefd9: Pull complete 2024-12-18T00:03:20.4860366Z f8a6d90f0b3e: Pull complete 2024-12-18T00:03:26.9057818Z 011542937262: Pull complete 2024-12-18T00:03:27.3589945Z 812caf3e4338: Pull complete 2024-12-18T00:03:27.8373896Z 7e2aa240b70c: Pull complete 2024-12-18T00:03:28.8736161Z a0e5fd2630de: Pull complete 2024-12-18T00:03:29.3394196Z a7253f4f1310: Pull complete 2024-12-18T00:03:29.7962205Z d4bbe33feb5a: Pull complete 2024-12-18T00:03:30.4165869Z c7da3e1fda20: Pull complete 2024-12-18T00:03:30.8331147Z a1fa96c8688b: Pull complete 2024-12-18T00:03:31.4055700Z 600e7beb6ed6: Pull complete 2024-12-18T00:03:31.8505795Z 7fee2a426d6e: Pull complete 2024-12-18T00:03:32.2542732Z 4f4fb700ef54: Pull complete 2024-12-18T00:03:32.7206040Z 321081f536b0: Pull complete 2024-12-18T00:03:33.1558524Z b2671014f386: Pull complete 2024-12-18T00:03:35.2461924Z 94a3028693ad: Pull complete 2024-12-18T00:03:35.7053137Z 838f3043a48a: Pull complete 2024-12-18T00:03:36.1549922Z 3bf4f9678a11: Pull complete 2024-12-18T00:03:37.0865168Z cbd527c6febd: Pull complete 2024-12-18T00:03:37.5216179Z 70c75642ca15: Pull complete 2024-12-18T00:04:50.8868092Z 975ffa249994: Pull complete 2024-12-18T00:04:51.3434166Z 1de45071cffa: Pull complete 2024-12-18T00:04:51.8046075Z 6f1f1c737991: Pull complete 2024-12-18T00:04:52.7101212Z 8da387044f45: Pull complete 2024-12-18T00:04:53.5760364Z 25b1aa58f5ca: Pull complete 2024-12-18T00:04:54.0509503Z f4c8ff23e407: Pull complete 2024-12-18T00:04:59.3129236Z d9a1d16ac9fb: Pull complete 2024-12-18T00:04:59.5092587Z 4c2486120303: Pull complete 2024-12-18T00:04:59.8732803Z 1c1a1b6b59a8: Pull complete 2024-12-18T00:05:00.3043398Z 65ab75609fc9: Pull complete 2024-12-18T00:05:00.7639334Z 41fe6d82c37a: Pull complete 2024-12-18T00:05:01.9426033Z 9456ad12e83f: Pull complete 2024-12-18T00:05:02.3720660Z 7dd812a14957: Pull complete 2024-12-18T00:05:04.2760907Z e47d2dd30bb9: Pull complete 2024-12-18T00:05:04.6738022Z 9a255e925aee: Pull complete 2024-12-18T00:05:17.3529033Z 266e2409e545: Pull complete 2024-12-18T00:05:17.7978878Z 100d759b6098: Pull complete 2024-12-18T00:05:18.2509405Z a3111847bfa2: Pull complete 2024-12-18T00:05:19.1465301Z 7e6ea32f1b1c: Pull complete 2024-12-18T00:05:19.5601206Z 38bf4ff6a6ac: Pull complete 2024-12-18T00:05:19.9913142Z 231d0f857789: Pull complete 2024-12-18T00:05:20.4508717Z 0fe4c18889ad: Pull complete 2024-12-18T00:05:20.8857621Z c5d910c7fb9c: Pull complete 2024-12-18T00:05:22.8952675Z Digest: sha256:52918bcdb0e370d1b9413099049464295e6c36e0a09cff6cff645b519e6da502 2024-12-18T00:05:22.9819733Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:05:23.0317593Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:05:23.0369858Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:05:23.0370948Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:05:23.0380300Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:05:23.0380747Z env: 2024-12-18T00:05:23.0381021Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:05:23.0381342Z ##[endgroup] 2024-12-18T00:05:23.0640123Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@release/2.6 2024-12-18T00:05:23.0640596Z with: 2024-12-18T00:05:23.0640839Z driver-version: 550.54.15 2024-12-18T00:05:23.0641111Z env: 2024-12-18T00:05:23.0641342Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:05:23.0641625Z ##[endgroup] 2024-12-18T00:05:23.0802789Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-12-18T00:05:23.0803228Z with: 2024-12-18T00:05:23.0803464Z timeout_minutes: 10 2024-12-18T00:05:23.0803734Z max_attempts: 3 2024-12-18T00:05:23.0833854Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-12-18T00:05:23.0863089Z retry_wait_seconds: 10 2024-12-18T00:05:23.0863428Z polling_interval_seconds: 1 2024-12-18T00:05:23.0863777Z warning_on_retry: true 2024-12-18T00:05:23.0864095Z continue_on_error: false 2024-12-18T00:05:23.0864398Z env: 2024-12-18T00:05:23.0864655Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:05:23.0864978Z DRIVER_VERSION: 550.54.15 2024-12-18T00:05:23.0865294Z ##[endgroup] 2024-12-18T00:05:23.2306903Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-12-18T00:05:23.2307440Z + pre_install_nvidia_driver_amzn2 2024-12-18T00:05:23.2309269Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-12-18T00:05:23.7011650Z No match for argument: nvidia-driver-latest-dkms 2024-12-18T00:05:23.7012200Z No packages marked for removal. 2024-12-18T00:05:23.7080565Z Dependencies resolved. 2024-12-18T00:05:23.7088268Z Nothing to do. 2024-12-18T00:05:23.7088986Z Complete! 2024-12-18T00:05:23.8522063Z + install_nvidia_driver_common 2024-12-18T00:05:23.8523598Z + echo 'Before installing NVIDIA driver' 2024-12-18T00:05:23.8525003Z + lspci 2024-12-18T00:05:23.8526727Z Before installing NVIDIA driver 2024-12-18T00:05:23.9689585Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-12-18T00:05:23.9690630Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-12-18T00:05:23.9691398Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-12-18T00:05:23.9692068Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-12-18T00:05:23.9692669Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-12-18T00:05:23.9693337Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-12-18T00:05:23.9693986Z 00:1b.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:23.9694568Z 00:1c.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:23.9695329Z 00:1d.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:23.9695911Z 00:1e.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:23.9696534Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-12-18T00:05:23.9697035Z + lsmod 2024-12-18T00:05:23.9726646Z Module Size Used by 2024-12-18T00:05:23.9727115Z xt_conntrack 16384 1 2024-12-18T00:05:23.9727437Z nft_chain_nat 16384 3 2024-12-18T00:05:23.9727799Z xt_MASQUERADE 20480 1 2024-12-18T00:05:23.9728177Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-12-18T00:05:23.9728596Z nf_conntrack_netlink 57344 0 2024-12-18T00:05:23.9729098Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-12-18T00:05:23.9729664Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-12-18T00:05:23.9730051Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-12-18T00:05:23.9730430Z xfrm_user 57344 1 2024-12-18T00:05:23.9730767Z xfrm_algo 16384 1 xfrm_user 2024-12-18T00:05:23.9731132Z xt_addrtype 16384 2 2024-12-18T00:05:23.9731652Z nft_compat 20480 4 2024-12-18T00:05:23.9732035Z nf_tables 311296 57 nft_compat,nft_chain_nat 2024-12-18T00:05:23.9732555Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-12-18T00:05:23.9733029Z br_netfilter 36864 0 2024-12-18T00:05:23.9733377Z bridge 323584 1 br_netfilter 2024-12-18T00:05:23.9733747Z stp 16384 1 bridge 2024-12-18T00:05:23.9734336Z llc 16384 2 bridge,stp 2024-12-18T00:05:23.9734799Z overlay 167936 0 2024-12-18T00:05:23.9747619Z tls 114688 0 2024-12-18T00:05:23.9748206Z nls_ascii 16384 1 2024-12-18T00:05:23.9748536Z nls_cp437 20480 1 2024-12-18T00:05:23.9748869Z vfat 24576 1 2024-12-18T00:05:23.9749216Z fat 86016 1 vfat 2024-12-18T00:05:23.9749573Z sunrpc 692224 1 2024-12-18T00:05:23.9749889Z ena 167936 0 2024-12-18T00:05:23.9750208Z i8042 45056 0 2024-12-18T00:05:23.9750538Z serio 28672 3 i8042 2024-12-18T00:05:23.9751011Z skx_edac_common 28672 0 2024-12-18T00:05:23.9751346Z ghash_clmulni_intel 16384 0 2024-12-18T00:05:23.9751662Z button 24576 0 2024-12-18T00:05:23.9751985Z sch_fq_codel 20480 33 2024-12-18T00:05:23.9752316Z dm_mod 188416 0 2024-12-18T00:05:23.9752633Z fuse 163840 1 2024-12-18T00:05:23.9752953Z configfs 57344 1 2024-12-18T00:05:23.9753263Z dax 45056 1 dm_mod 2024-12-18T00:05:23.9753613Z loop 36864 0 2024-12-18T00:05:23.9753933Z dmi_sysfs 20480 0 2024-12-18T00:05:23.9754252Z crc32_pclmul 16384 0 2024-12-18T00:05:23.9754576Z crc32c_intel 24576 0 2024-12-18T00:05:23.9754888Z efivarfs 24576 1 2024-12-18T00:05:23.9755203Z + modinfo nvidia 2024-12-18T00:05:23.9755722Z filename: /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-12-18T00:05:23.9756296Z alias: char-major-195-* 2024-12-18T00:05:23.9756636Z version: 550.54.15 2024-12-18T00:05:23.9756946Z supported: external 2024-12-18T00:05:23.9757237Z license: NVIDIA 2024-12-18T00:05:23.9757576Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-12-18T00:05:23.9758000Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-12-18T00:05:23.9758397Z srcversion: 833721318DA517F0C2FEC97 2024-12-18T00:05:23.9758796Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-12-18T00:05:23.9759210Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-12-18T00:05:23.9759637Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-12-18T00:05:23.9760030Z depends: i2c-core,drm 2024-12-18T00:05:23.9760355Z retpoline: Y 2024-12-18T00:05:23.9760746Z name: nvidia 2024-12-18T00:05:23.9761193Z vermagic: 6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-12-18T00:05:23.9761785Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-12-18T00:05:23.9762339Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-12-18T00:05:23.9762865Z parm: NVreg_ResmanDebugLevel:int 2024-12-18T00:05:23.9763254Z parm: NVreg_RmLogonRC:int 2024-12-18T00:05:23.9763618Z parm: NVreg_ModifyDeviceFiles:int 2024-12-18T00:05:23.9764012Z parm: NVreg_DeviceFileUID:int 2024-12-18T00:05:23.9764392Z parm: NVreg_DeviceFileGID:int 2024-12-18T00:05:23.9764774Z parm: NVreg_DeviceFileMode:int 2024-12-18T00:05:23.9765216Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-12-18T00:05:23.9765680Z parm: NVreg_UsePageAttributeTable:int 2024-12-18T00:05:23.9766090Z parm: NVreg_EnablePCIeGen3:int 2024-12-18T00:05:23.9766459Z parm: NVreg_EnableMSI:int 2024-12-18T00:05:23.9766930Z parm: NVreg_TCEBypassMode:int 2024-12-18T00:05:23.9767314Z parm: NVreg_EnableStreamMemOPs:int 2024-12-18T00:05:23.9767737Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-12-18T00:05:23.9768214Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-12-18T00:05:23.9768845Z parm: NVreg_EnableS0ixPowerManagement:int 2024-12-18T00:05:23.9769352Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-12-18T00:05:23.9769855Z parm: NVreg_DynamicPowerManagement:int 2024-12-18T00:05:23.9770368Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-12-18T00:05:23.9770859Z parm: NVreg_EnableGpuFirmware:int 2024-12-18T00:05:23.9771382Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-12-18T00:05:23.9771843Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-12-18T00:05:23.9772311Z parm: NVreg_EnableUserNUMAManagement:int 2024-12-18T00:05:23.9772744Z parm: NVreg_MemoryPoolSize:int 2024-12-18T00:05:23.9773137Z parm: NVreg_KMallocHeapMaxSize:int 2024-12-18T00:05:23.9773552Z parm: NVreg_VMallocHeapMaxSize:int 2024-12-18T00:05:23.9773958Z parm: NVreg_IgnoreMMIOCheck:int 2024-12-18T00:05:23.9774350Z parm: NVreg_NvLinkDisable:int 2024-12-18T00:05:23.9774778Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-12-18T00:05:23.9775207Z parm: NVreg_RegisterPCIDriver:int 2024-12-18T00:05:23.9775614Z parm: NVreg_EnableResizableBar:int 2024-12-18T00:05:23.9776029Z parm: NVreg_EnableDbgBreakpoint:int 2024-12-18T00:05:23.9776457Z parm: NVreg_EnableNonblockingOpen:int 2024-12-18T00:05:23.9776877Z parm: NVreg_RegistryDwords:charp 2024-12-18T00:05:23.9777294Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-12-18T00:05:23.9777706Z parm: NVreg_RmMsg:charp 2024-12-18T00:05:23.9778071Z parm: NVreg_GpuBlacklist:charp 2024-12-18T00:05:23.9778479Z parm: NVreg_TemporaryFilePath:charp 2024-12-18T00:05:23.9778990Z parm: NVreg_ExcludedGpus:charp 2024-12-18T00:05:23.9779550Z parm: NVreg_DmaRemapPeerMmio:int 2024-12-18T00:05:23.9780004Z parm: NVreg_RmNvlinkBandwidth:charp 2024-12-18T00:05:23.9780437Z parm: NVreg_ImexChannelCount:int 2024-12-18T00:05:23.9780841Z parm: rm_firmware_active:charp 2024-12-18T00:05:23.9781201Z + HAS_NVIDIA_DRIVER=0 2024-12-18T00:05:23.9781514Z ++ command -v nvidia-smi 2024-12-18T00:05:23.9781846Z + '[' -x /usr/bin/nvidia-smi ']' 2024-12-18T00:05:23.9782178Z + set +e 2024-12-18T00:05:23.9782563Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-12-18T00:05:28.9239130Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-12-18T00:05:28.9239650Z + NVIDIA_SMI_STATUS=0 2024-12-18T00:05:28.9239969Z + '[' 0 -ne 0 ']' 2024-12-18T00:05:28.9240250Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-12-18T00:05:28.9240589Z + HAS_NVIDIA_DRIVER=1 2024-12-18T00:05:28.9241776Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-12-18T00:05:28.9242432Z + set -e 2024-12-18T00:05:28.9242687Z + '[' 1 -eq 0 ']' 2024-12-18T00:05:28.9243173Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-12-18T00:05:28.9243754Z + post_install_nvidia_driver_common 2024-12-18T00:05:28.9244111Z + sudo modprobe nvidia 2024-12-18T00:05:29.1105288Z + echo 'After installing NVIDIA driver' 2024-12-18T00:05:29.1105701Z + lspci 2024-12-18T00:05:29.1105992Z After installing NVIDIA driver 2024-12-18T00:05:29.1234604Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-12-18T00:05:29.1235248Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-12-18T00:05:29.1235982Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-12-18T00:05:29.1236631Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-12-18T00:05:29.1237249Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-12-18T00:05:29.1237921Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-12-18T00:05:29.1238548Z 00:1b.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:29.1239136Z 00:1c.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:29.1239709Z 00:1d.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:29.1240289Z 00:1e.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:29.1240917Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-12-18T00:05:29.1241429Z + lsmod 2024-12-18T00:05:29.1258897Z Module Size Used by 2024-12-18T00:05:29.1259676Z nvidia_uvm 4706304 0 2024-12-18T00:05:29.1260025Z nvidia 54071296 1 nvidia_uvm 2024-12-18T00:05:29.1260408Z drm 602112 1 nvidia 2024-12-18T00:05:29.1260817Z drm_panel_orientation_quirks 32768 1 drm 2024-12-18T00:05:29.1261219Z backlight 24576 1 drm 2024-12-18T00:05:29.1261597Z i2c_core 106496 2 nvidia,drm 2024-12-18T00:05:29.1261967Z xt_conntrack 16384 1 2024-12-18T00:05:29.1262313Z nft_chain_nat 16384 3 2024-12-18T00:05:29.1262657Z xt_MASQUERADE 20480 1 2024-12-18T00:05:29.1263048Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-12-18T00:05:29.1263483Z nf_conntrack_netlink 57344 0 2024-12-18T00:05:29.1263991Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-12-18T00:05:29.1264559Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-12-18T00:05:29.1264972Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-12-18T00:05:29.1265364Z xfrm_user 57344 1 2024-12-18T00:05:29.1265717Z xfrm_algo 16384 1 xfrm_user 2024-12-18T00:05:29.1266095Z xt_addrtype 16384 2 2024-12-18T00:05:29.1266431Z nft_compat 20480 4 2024-12-18T00:05:29.1266836Z nf_tables 311296 57 nft_compat,nft_chain_nat 2024-12-18T00:05:29.1267394Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-12-18T00:05:29.1267887Z br_netfilter 36864 0 2024-12-18T00:05:29.1268251Z bridge 323584 1 br_netfilter 2024-12-18T00:05:29.1268631Z stp 16384 1 bridge 2024-12-18T00:05:29.1269014Z llc 16384 2 bridge,stp 2024-12-18T00:05:29.1269393Z overlay 167936 0 2024-12-18T00:05:29.1269729Z tls 114688 0 2024-12-18T00:05:29.1270059Z nls_ascii 16384 1 2024-12-18T00:05:29.1270378Z nls_cp437 20480 1 2024-12-18T00:05:29.1270707Z vfat 24576 1 2024-12-18T00:05:29.1271045Z fat 86016 1 vfat 2024-12-18T00:05:29.1271529Z sunrpc 692224 1 2024-12-18T00:05:29.1271855Z ena 167936 0 2024-12-18T00:05:29.1272273Z i8042 45056 0 2024-12-18T00:05:29.1272704Z serio 28672 3 i8042 2024-12-18T00:05:29.1273052Z skx_edac_common 28672 0 2024-12-18T00:05:29.1273380Z ghash_clmulni_intel 16384 0 2024-12-18T00:05:29.1273705Z button 24576 0 2024-12-18T00:05:29.1274011Z sch_fq_codel 20480 33 2024-12-18T00:05:29.1274613Z dm_mod 188416 0 2024-12-18T00:05:29.1274927Z fuse 163840 1 2024-12-18T00:05:29.1275406Z configfs 57344 1 2024-12-18T00:05:29.1275734Z dax 45056 1 dm_mod 2024-12-18T00:05:29.1276083Z loop 36864 0 2024-12-18T00:05:29.1276411Z dmi_sysfs 20480 0 2024-12-18T00:05:29.1276738Z crc32_pclmul 16384 0 2024-12-18T00:05:29.1277075Z crc32c_intel 24576 0 2024-12-18T00:05:29.1277404Z efivarfs 24576 1 2024-12-18T00:05:29.1277712Z + modinfo nvidia 2024-12-18T00:05:29.1278222Z filename: /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-12-18T00:05:29.1278795Z alias: char-major-195-* 2024-12-18T00:05:29.1279140Z version: 550.54.15 2024-12-18T00:05:29.1279457Z supported: external 2024-12-18T00:05:29.1279765Z license: NVIDIA 2024-12-18T00:05:29.1280093Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-12-18T00:05:29.1280523Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-12-18T00:05:29.1280923Z srcversion: 833721318DA517F0C2FEC97 2024-12-18T00:05:29.1281329Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-12-18T00:05:29.1281760Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-12-18T00:05:29.1282171Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-12-18T00:05:29.1282572Z depends: i2c-core,drm 2024-12-18T00:05:29.1282976Z retpoline: Y 2024-12-18T00:05:29.1283262Z name: nvidia 2024-12-18T00:05:29.1283724Z vermagic: 6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-12-18T00:05:29.1284313Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-12-18T00:05:29.1284870Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-12-18T00:05:29.1285397Z parm: NVreg_ResmanDebugLevel:int 2024-12-18T00:05:29.1285789Z parm: NVreg_RmLogonRC:int 2024-12-18T00:05:29.1286171Z parm: NVreg_ModifyDeviceFiles:int 2024-12-18T00:05:29.1286562Z parm: NVreg_DeviceFileUID:int 2024-12-18T00:05:29.1286945Z parm: NVreg_DeviceFileGID:int 2024-12-18T00:05:29.1287330Z parm: NVreg_DeviceFileMode:int 2024-12-18T00:05:29.1287787Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-12-18T00:05:29.1288265Z parm: NVreg_UsePageAttributeTable:int 2024-12-18T00:05:29.1288679Z parm: NVreg_EnablePCIeGen3:int 2024-12-18T00:05:29.1289071Z parm: NVreg_EnableMSI:int 2024-12-18T00:05:29.1289444Z parm: NVreg_TCEBypassMode:int 2024-12-18T00:05:29.1289851Z parm: NVreg_EnableStreamMemOPs:int 2024-12-18T00:05:29.1290308Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-12-18T00:05:29.1290788Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-12-18T00:05:29.1291269Z parm: NVreg_EnableS0ixPowerManagement:int 2024-12-18T00:05:29.1291777Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-12-18T00:05:29.1292282Z parm: NVreg_DynamicPowerManagement:int 2024-12-18T00:05:29.1292799Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-12-18T00:05:29.1293306Z parm: NVreg_EnableGpuFirmware:int 2024-12-18T00:05:29.1293719Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-12-18T00:05:29.1294185Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-12-18T00:05:29.1294656Z parm: NVreg_EnableUserNUMAManagement:int 2024-12-18T00:05:29.1295096Z parm: NVreg_MemoryPoolSize:int 2024-12-18T00:05:29.1295511Z parm: NVreg_KMallocHeapMaxSize:int 2024-12-18T00:05:29.1295986Z parm: NVreg_VMallocHeapMaxSize:int 2024-12-18T00:05:29.1296405Z parm: NVreg_IgnoreMMIOCheck:int 2024-12-18T00:05:29.1296803Z parm: NVreg_NvLinkDisable:int 2024-12-18T00:05:29.1297243Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-12-18T00:05:29.1297702Z parm: NVreg_RegisterPCIDriver:int 2024-12-18T00:05:29.1298106Z parm: NVreg_EnableResizableBar:int 2024-12-18T00:05:29.1298532Z parm: NVreg_EnableDbgBreakpoint:int 2024-12-18T00:05:29.1299050Z parm: NVreg_EnableNonblockingOpen:int 2024-12-18T00:05:29.1299653Z parm: NVreg_RegistryDwords:charp 2024-12-18T00:05:29.1300128Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-12-18T00:05:29.1300553Z parm: NVreg_RmMsg:charp 2024-12-18T00:05:29.1300936Z parm: NVreg_GpuBlacklist:charp 2024-12-18T00:05:29.1301366Z parm: NVreg_TemporaryFilePath:charp 2024-12-18T00:05:29.1301794Z parm: NVreg_ExcludedGpus:charp 2024-12-18T00:05:29.1302213Z parm: NVreg_DmaRemapPeerMmio:int 2024-12-18T00:05:29.1302633Z parm: NVreg_RmNvlinkBandwidth:charp 2024-12-18T00:05:29.1303064Z parm: NVreg_ImexChannelCount:int 2024-12-18T00:05:29.1303473Z parm: rm_firmware_active:charp 2024-12-18T00:05:29.1303850Z + set +e 2024-12-18T00:05:29.1304107Z + nvidia-smi 2024-12-18T00:05:31.6806151Z Wed Dec 18 00:05:31 2024 2024-12-18T00:05:31.6806672Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:05:31.6807321Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-12-18T00:05:31.6807949Z |-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:05:31.6808587Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-12-18T00:05:31.6809503Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-12-18T00:05:31.6810082Z | | | MIG M. | 2024-12-18T00:05:31.6810521Z |=========================================+========================+======================| 2024-12-18T00:05:31.6900028Z | 0 Tesla T4 Off | 00000000:00:1B.0 Off | 0 | 2024-12-18T00:05:31.6900709Z | N/A 33C P0 24W / 70W | 0MiB / 15360MiB | 0% Default | 2024-12-18T00:05:31.6901226Z | | | N/A | 2024-12-18T00:05:31.6901757Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:05:31.6988027Z | 1 Tesla T4 Off | 00000000:00:1C.0 Off | 0 | 2024-12-18T00:05:31.6988655Z | N/A 32C P0 25W / 70W | 0MiB / 15360MiB | 0% Default | 2024-12-18T00:05:31.6989187Z | | | N/A | 2024-12-18T00:05:31.6989720Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:05:31.7078772Z | 2 Tesla T4 Off | 00000000:00:1D.0 Off | 0 | 2024-12-18T00:05:31.7079371Z | N/A 34C P0 25W / 70W | 0MiB / 15360MiB | 0% Default | 2024-12-18T00:05:31.7079874Z | | | N/A | 2024-12-18T00:05:31.7080374Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:05:31.7170914Z | 3 Tesla T4 Off | 00000000:00:1E.0 Off | 0 | 2024-12-18T00:05:31.7171973Z | N/A 34C P0 25W / 70W | 0MiB / 15360MiB | 7% Default | 2024-12-18T00:05:31.7172524Z | | | N/A | 2024-12-18T00:05:31.7173791Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:05:31.7175416Z 2024-12-18T00:05:31.7175915Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:05:31.7176459Z | Processes: | 2024-12-18T00:05:31.7177028Z | GPU GI CI PID Type Process name GPU Memory | 2024-12-18T00:05:31.7177561Z | ID ID Usage | 2024-12-18T00:05:31.7177997Z |=========================================================================================| 2024-12-18T00:05:31.7194992Z | No running processes found | 2024-12-18T00:05:31.7195634Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:05:34.1242319Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-12-18T00:05:36.6374615Z Tesla T4 2024-12-18T00:05:38.6691383Z + NVIDIA_SMI_STATUS=0 2024-12-18T00:05:38.6691765Z + '[' 0 -eq 0 ']' 2024-12-18T00:05:38.6692087Z + echo 'INFO: Ignoring allowed status 0' 2024-12-18T00:05:38.6692458Z + set -e 2024-12-18T00:05:38.6692730Z INFO: Ignoring allowed status 0 2024-12-18T00:05:38.6697299Z == Installing nvidia container toolkit for amzn2023 == 2024-12-18T00:05:38.6701766Z + sudo yum install -y yum-utils 2024-12-18T00:05:39.1086223Z Last metadata expiration check: 1:07:00 ago on Tue Dec 17 22:58:39 2024. 2024-12-18T00:05:39.1350298Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-12-18T00:05:39.1748760Z Dependencies resolved. 2024-12-18T00:05:39.1932743Z Nothing to do. 2024-12-18T00:05:39.1933142Z Complete! 2024-12-18T00:05:39.3421180Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-12-18T00:05:39.3421992Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-12-18T00:05:39.3423093Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-12-18T00:05:39.6657364Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-12-18T00:05:39.7387048Z + sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 2024-12-18T00:05:40.2729046Z nvidia-container-toolkit 10 kB/s | 833 B 00:00 2024-12-18T00:05:40.3001989Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-12-18T00:05:40.3411324Z Dependencies resolved. 2024-12-18T00:05:40.3590863Z ================================================================================ 2024-12-18T00:05:40.3591593Z Package Arch Version Repository Size 2024-12-18T00:05:40.3592092Z ================================================================================ 2024-12-18T00:05:40.3592462Z Downgrading: 2024-12-18T00:05:40.3592922Z nvidia-container-toolkit x86_64 1.16.2-1 nvidia-container-toolkit 1.2 M 2024-12-18T00:05:40.3593656Z nvidia-container-toolkit-base x86_64 1.16.2-1 nvidia-container-toolkit 5.6 M 2024-12-18T00:05:40.3594105Z 2024-12-18T00:05:40.3594220Z Transaction Summary 2024-12-18T00:05:40.3594536Z ================================================================================ 2024-12-18T00:05:40.3594924Z Downgrade 2 Packages 2024-12-18T00:05:40.3595105Z 2024-12-18T00:05:40.3595231Z Total download size: 6.8 M 2024-12-18T00:05:40.3595557Z Downloading Packages: 2024-12-18T00:05:40.5549325Z (1/2): nvidia-container-toolkit-base-1.16.2-1.x 29 MB/s | 5.6 MB 00:00 2024-12-18T00:05:40.6444231Z (2/2): nvidia-container-toolkit-1.16.2-1.x86_64 4.3 MB/s | 1.2 MB 00:00 2024-12-18T00:05:40.6450587Z -------------------------------------------------------------------------------- 2024-12-18T00:05:40.6453735Z Total 24 MB/s | 6.8 MB 00:00 2024-12-18T00:05:40.6455300Z Running transaction check 2024-12-18T00:05:40.6565865Z Transaction check succeeded. 2024-12-18T00:05:40.6566573Z Running transaction test 2024-12-18T00:05:40.6876963Z Transaction test succeeded. 2024-12-18T00:05:40.6877798Z Running transaction 2024-12-18T00:05:41.6138613Z Preparing : 1/1 2024-12-18T00:05:41.8360663Z Downgrading : nvidia-container-toolkit-base-1.16.2-1.x86_64 1/4 2024-12-18T00:05:41.8886315Z Downgrading : nvidia-container-toolkit-1.16.2-1.x86_64 2/4 2024-12-18T00:05:41.9719595Z Running scriptlet: nvidia-container-toolkit-1.16.2-1.x86_64 2/4 2024-12-18T00:05:41.9720357Z Cleanup : nvidia-container-toolkit-1.17.3-1.x86_64 3/4 2024-12-18T00:05:42.0301087Z Running scriptlet: nvidia-container-toolkit-1.17.3-1.x86_64 3/4 2024-12-18T00:05:42.0791017Z Cleanup : nvidia-container-toolkit-base-1.17.3-1.x86_64 4/4 2024-12-18T00:07:10.9894280Z Running scriptlet: nvidia-container-toolkit-1.16.2-1.x86_64 4/4 2024-12-18T00:07:10.9895026Z Verifying : nvidia-container-toolkit-1.16.2-1.x86_64 1/4 2024-12-18T00:07:10.9895790Z Verifying : nvidia-container-toolkit-1.17.3-1.x86_64 2/4 2024-12-18T00:07:10.9896496Z Verifying : nvidia-container-toolkit-base-1.16.2-1.x86_64 3/4 2024-12-18T00:07:11.1443434Z Verifying : nvidia-container-toolkit-base-1.17.3-1.x86_64 4/4================================================================================ 2024-12-18T00:07:11.1444152Z WARNING: 2024-12-18T00:07:11.1444452Z A newer release of "Amazon Linux" is available. 2024-12-18T00:07:11.1445025Z 2024-12-18T00:07:11.1445141Z Available Versions: 2024-12-18T00:07:11.1445338Z 2024-12-18T00:07:11.1445450Z Version 2023.6.20241010: 2024-12-18T00:07:11.1445855Z Run the following command to upgrade to 2023.6.20241010: 2024-12-18T00:07:11.1446172Z 2024-12-18T00:07:11.1446334Z dnf upgrade --releasever=2023.6.20241010 2024-12-18T00:07:11.1446597Z 2024-12-18T00:07:11.1446716Z Release notes: 2024-12-18T00:07:11.1447214Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241010.html 2024-12-18T00:07:11.1447680Z 2024-12-18T00:07:11.1447790Z Version 2023.6.20241028: 2024-12-18T00:07:11.1448175Z Run the following command to upgrade to 2023.6.20241028: 2024-12-18T00:07:11.1448484Z 2024-12-18T00:07:11.1448640Z dnf upgrade --releasever=2023.6.20241028 2024-12-18T00:07:11.1448898Z 2024-12-18T00:07:11.1449016Z Release notes: 2024-12-18T00:07:11.1449511Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241028.html 2024-12-18T00:07:11.1449967Z 2024-12-18T00:07:11.1450078Z Version 2023.6.20241031: 2024-12-18T00:07:11.1450466Z Run the following command to upgrade to 2023.6.20241031: 2024-12-18T00:07:11.1450792Z 2024-12-18T00:07:11.1450933Z dnf upgrade --releasever=2023.6.20241031 2024-12-18T00:07:11.1451191Z 2024-12-18T00:07:11.1451309Z Release notes: 2024-12-18T00:07:11.1451797Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241031.html 2024-12-18T00:07:11.1452244Z 2024-12-18T00:07:11.1452367Z Version 2023.6.20241111: 2024-12-18T00:07:11.1452738Z Run the following command to upgrade to 2023.6.20241111: 2024-12-18T00:07:11.1453058Z 2024-12-18T00:07:11.1453198Z dnf upgrade --releasever=2023.6.20241111 2024-12-18T00:07:11.1453469Z 2024-12-18T00:07:11.1453573Z Release notes: 2024-12-18T00:07:11.1454059Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241111.html 2024-12-18T00:07:11.1454509Z 2024-12-18T00:07:11.1454630Z Version 2023.6.20241121: 2024-12-18T00:07:11.1455121Z Run the following command to upgrade to 2023.6.20241121: 2024-12-18T00:07:11.1455418Z 2024-12-18T00:07:11.1455817Z dnf upgrade --releasever=2023.6.20241121 2024-12-18T00:07:11.1456074Z 2024-12-18T00:07:11.1456173Z Release notes: 2024-12-18T00:07:11.1456626Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241121.html 2024-12-18T00:07:11.1457037Z 2024-12-18T00:07:11.1457159Z Version 2023.6.20241212: 2024-12-18T00:07:11.1457521Z Run the following command to upgrade to 2023.6.20241212: 2024-12-18T00:07:11.1457802Z 2024-12-18T00:07:11.1457946Z dnf upgrade --releasever=2023.6.20241212 2024-12-18T00:07:11.1458184Z 2024-12-18T00:07:11.1458281Z Release notes: 2024-12-18T00:07:11.1458811Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241212.html 2024-12-18T00:07:11.1459427Z 2024-12-18T00:07:11.1459571Z ================================================================================ 2024-12-18T00:07:11.1779104Z 2024-12-18T00:07:11.1779338Z 2024-12-18T00:07:11.1780021Z Downgraded: 2024-12-18T00:07:11.1780563Z nvidia-container-toolkit-1.16.2-1.x86_64 2024-12-18T00:07:11.1781307Z nvidia-container-toolkit-base-1.16.2-1.x86_64 2024-12-18T00:07:11.1781744Z 2024-12-18T00:07:11.1781870Z Complete! 2024-12-18T00:07:11.2285958Z + sudo systemctl restart docker 2024-12-18T00:07:19.7691487Z Wed Dec 18 00:07:19 2024 2024-12-18T00:07:19.7692027Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:07:19.7692648Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-12-18T00:07:19.7693288Z |-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:07:19.7693923Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-12-18T00:07:19.7694949Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-12-18T00:07:19.7695516Z | | | MIG M. | 2024-12-18T00:07:19.7695946Z |=========================================+========================+======================| 2024-12-18T00:07:19.7800581Z | 0 Tesla T4 On | 00000000:00:1B.0 Off | 0 | 2024-12-18T00:07:19.7801605Z | N/A 29C P0 24W / 70W | 0MiB / 15360MiB | 0% Default | 2024-12-18T00:07:19.7802130Z | | | N/A | 2024-12-18T00:07:19.7891153Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:07:19.7891763Z | 1 Tesla T4 On | 00000000:00:1C.0 Off | 0 | 2024-12-18T00:07:19.7892332Z | N/A 28C P0 25W / 70W | 0MiB / 15360MiB | 0% Default | 2024-12-18T00:07:19.7892820Z | | | N/A | 2024-12-18T00:07:19.7893340Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:07:19.7981306Z | 2 Tesla T4 On | 00000000:00:1D.0 Off | 0 | 2024-12-18T00:07:19.7981901Z | N/A 30C P0 24W / 70W | 0MiB / 15360MiB | 0% Default | 2024-12-18T00:07:19.7982412Z | | | N/A | 2024-12-18T00:07:19.7982938Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:07:19.8073878Z | 3 Tesla T4 On | 00000000:00:1E.0 Off | 0 | 2024-12-18T00:07:19.8074461Z | N/A 30C P0 25W / 70W | 0MiB / 15360MiB | 7% Default | 2024-12-18T00:07:19.8075077Z | | | N/A | 2024-12-18T00:07:19.8075805Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:07:19.8077907Z 2024-12-18T00:07:19.8078408Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:07:19.8078952Z | Processes: | 2024-12-18T00:07:19.8079509Z | GPU GI CI PID Type Process name GPU Memory | 2024-12-18T00:07:19.8080048Z | ID ID Usage | 2024-12-18T00:07:19.8080490Z |=========================================================================================| 2024-12-18T00:07:19.8098246Z | No running processes found | 2024-12-18T00:07:19.8099005Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:07:20.3400002Z Command completed after 1 attempt(s). 2024-12-18T00:07:20.3486271Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-12-18T00:07:20.3486947Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-12-18T00:07:20.3487539Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-12-18T00:07:20.3488078Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:07:20.3498507Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:20.3499024Z env: 2024-12-18T00:07:20.3499461Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:20.3499867Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:20.3500380Z ##[endgroup] 2024-12-18T00:07:20.6768601Z Defaulting to user installation because normal site-packages is not writeable 2024-12-18T00:07:21.0933743Z Collecting psutil==5.9.1 2024-12-18T00:07:21.1368404Z Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-12-18T00:07:21.1913873Z Collecting nvidia-ml-py==11.525.84 2024-12-18T00:07:21.1954989Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-12-18T00:07:21.2883778Z Installing collected packages: psutil, nvidia-ml-py 2024-12-18T00:07:21.4583773Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-12-18T00:07:21.6591346Z Prepare all required actions 2024-12-18T00:07:21.6592311Z Getting action download info 2024-12-18T00:07:21.8186683Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-12-18T00:07:22.0595956Z Download action repository 'actions/download-artifact@v4' (SHA:fa0a91b85d4f404e444e00e005971372dc801d16) 2024-12-18T00:07:22.4039813Z ##[group]Run ./.github/actions/download-build-artifacts 2024-12-18T00:07:22.4040407Z with: 2024-12-18T00:07:22.4040693Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-12-18T00:07:22.4041083Z s3-bucket: gha-artifacts 2024-12-18T00:07:22.4041391Z env: 2024-12-18T00:07:22.4041633Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:22.4042029Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:22.4042450Z ##[endgroup] 2024-12-18T00:07:22.4079030Z ##[group]Run seemethere/download-artifact-s3@v4 2024-12-18T00:07:22.4079430Z with: 2024-12-18T00:07:22.4079759Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-12-18T00:07:22.4080142Z s3-bucket: gha-artifacts 2024-12-18T00:07:22.4080446Z region: us-east-1 2024-12-18T00:07:22.4080721Z env: 2024-12-18T00:07:22.4080975Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:22.4081366Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:22.4081787Z ##[endgroup] 2024-12-18T00:07:22.9157800Z (node:59015) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-12-18T00:07:22.9158565Z 2024-12-18T00:07:22.9158801Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-12-18T00:07:22.9159418Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-12-18T00:07:22.9160069Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-12-18T00:07:23.1490639Z Found 1 objects with prefix pytorch/pytorch/12383255652/linux-focal-cuda11.8-py3.10-gcc9/ 2024-12-18T00:07:23.1491581Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-12-18T00:07:38.4283248Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-12-18T00:07:38.4288196Z Artifact download has finished successfully 2024-12-18T00:07:38.4513141Z ##[group]Run unzip -o artifacts.zip 2024-12-18T00:07:38.4513511Z unzip -o artifacts.zip 2024-12-18T00:07:38.4520512Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:38.4520929Z env: 2024-12-18T00:07:38.4521175Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:38.4521543Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:38.4521937Z ##[endgroup] 2024-12-18T00:07:38.4599085Z Archive: artifacts.zip 2024-12-18T00:07:38.4599517Z creating: dist/ 2024-12-18T00:07:40.4989696Z inflating: dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl 2024-12-18T00:07:40.4990334Z creating: build/custom_test_artifacts/ 2024-12-18T00:07:40.4990927Z creating: build/custom_test_artifacts/custom-op-build/ 2024-12-18T00:07:40.4991520Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-12-18T00:07:40.4992203Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-12-18T00:07:40.4997951Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T00:07:40.4998729Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-12-18T00:07:40.4999825Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T00:07:40.5000657Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T00:07:40.5001467Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T00:07:40.5002377Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T00:07:40.5003504Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T00:07:40.5004339Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T00:07:40.5005164Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T00:07:40.5006280Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T00:07:40.5007682Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T00:07:40.5009380Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T00:07:40.5010342Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T00:07:40.5011405Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T00:07:40.5012534Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T00:07:40.5013404Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-12-18T00:07:40.5014245Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-12-18T00:07:40.5061805Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-12-18T00:07:40.5063085Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-12-18T00:07:40.5064343Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-12-18T00:07:40.5065635Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-12-18T00:07:40.5066900Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-12-18T00:07:40.5068106Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-12-18T00:07:40.5069313Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-12-18T00:07:40.5070533Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-12-18T00:07:40.5071836Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-12-18T00:07:40.5107522Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-12-18T00:07:40.5149514Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-12-18T00:07:40.5150779Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-12-18T00:07:40.5152002Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-12-18T00:07:40.5153045Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-12-18T00:07:40.5154091Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-12-18T00:07:40.5155337Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-12-18T00:07:40.5156351Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-12-18T00:07:40.5157381Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-12-18T00:07:40.5218830Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-12-18T00:07:40.5284326Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-12-18T00:07:40.5285521Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-12-18T00:07:40.5286346Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-12-18T00:07:40.5287080Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-12-18T00:07:40.5287829Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-12-18T00:07:40.5288606Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-12-18T00:07:40.5289464Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-12-18T00:07:40.5290454Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-12-18T00:07:40.5291388Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-12-18T00:07:40.5292258Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-12-18T00:07:40.5293155Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-12-18T00:07:40.5294071Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-12-18T00:07:40.5294977Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-12-18T00:07:40.5295887Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-12-18T00:07:40.5296780Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-12-18T00:07:40.5312835Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-12-18T00:07:40.5458796Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-12-18T00:07:40.5459856Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-12-18T00:07:40.5460806Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-12-18T00:07:40.5461867Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-12-18T00:07:40.5462886Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-12-18T00:07:40.5463834Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-12-18T00:07:40.5464808Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-12-18T00:07:40.5465801Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-12-18T00:07:40.5466789Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-12-18T00:07:40.5467775Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-12-18T00:07:40.5468744Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-12-18T00:07:40.5484582Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-12-18T00:07:40.5569993Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-12-18T00:07:40.5571186Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T00:07:40.5572093Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-12-18T00:07:40.5573113Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-12-18T00:07:40.5573879Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-12-18T00:07:40.5574633Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-12-18T00:07:40.5575386Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-12-18T00:07:40.5576093Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-12-18T00:07:40.5576732Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-12-18T00:07:40.5577394Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-12-18T00:07:40.5700573Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-12-18T00:07:40.5765977Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-12-18T00:07:40.5766768Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-12-18T00:07:40.5767346Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-12-18T00:07:40.5768016Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-12-18T00:07:40.5772620Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T00:07:40.5773422Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-12-18T00:07:40.5774191Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T00:07:40.5775008Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T00:07:40.5775791Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T00:07:40.5776704Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T00:07:40.5777968Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T00:07:40.5778928Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T00:07:40.5779937Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T00:07:40.5780949Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T00:07:40.5782410Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T00:07:40.5784117Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T00:07:40.5785103Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T00:07:40.5786259Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T00:07:40.5787269Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T00:07:40.5788159Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-12-18T00:07:40.5788993Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-12-18T00:07:40.5835552Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-12-18T00:07:40.5836965Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-12-18T00:07:40.5838167Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-12-18T00:07:40.5839394Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-12-18T00:07:40.5840613Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-12-18T00:07:40.5841861Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-12-18T00:07:40.5843005Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-12-18T00:07:40.5844169Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-12-18T00:07:40.5845329Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-12-18T00:07:40.5882590Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-12-18T00:07:40.5924000Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-12-18T00:07:40.5925218Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-12-18T00:07:40.5926306Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-12-18T00:07:40.5927349Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-12-18T00:07:40.5928359Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-12-18T00:07:40.5929404Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-12-18T00:07:40.5930401Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-12-18T00:07:40.5931411Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-12-18T00:07:40.5993880Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-12-18T00:07:40.6056772Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-12-18T00:07:40.6057793Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-12-18T00:07:40.6058736Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-12-18T00:07:40.6059620Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-12-18T00:07:40.6060382Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-12-18T00:07:40.6061188Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-12-18T00:07:40.6062099Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-12-18T00:07:40.6063134Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-12-18T00:07:40.6064123Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-12-18T00:07:40.6065046Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-12-18T00:07:40.6065985Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-12-18T00:07:40.6066955Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-12-18T00:07:40.6068108Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-12-18T00:07:40.6069074Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-12-18T00:07:40.6070023Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-12-18T00:07:40.6085530Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-12-18T00:07:40.6153846Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-12-18T00:07:40.6154865Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T00:07:40.6155768Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-12-18T00:07:40.6156583Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-12-18T00:07:40.6157338Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-12-18T00:07:40.6158068Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-12-18T00:07:40.6158812Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-12-18T00:07:40.6159499Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-12-18T00:07:40.6160133Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-12-18T00:07:40.6160788Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-12-18T00:07:40.6211059Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-12-18T00:07:40.6211716Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-12-18T00:07:40.6212342Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-12-18T00:07:40.6213076Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-12-18T00:07:40.6217940Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T00:07:40.6218883Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-12-18T00:07:40.6219881Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T00:07:40.6220790Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T00:07:40.6221684Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T00:07:40.6222695Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T00:07:40.6223722Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T00:07:40.6224629Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T00:07:40.6225538Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T00:07:40.6226613Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T00:07:40.6228034Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T00:07:40.6229659Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T00:07:40.6230722Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T00:07:40.6232126Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T00:07:40.6233187Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T00:07:40.6234331Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-12-18T00:07:40.6235203Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-12-18T00:07:40.6283329Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-12-18T00:07:40.6284625Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-12-18T00:07:40.6286106Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-12-18T00:07:40.6287420Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-12-18T00:07:40.6288715Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-12-18T00:07:40.6289963Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-12-18T00:07:40.6291203Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-12-18T00:07:40.6292446Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-12-18T00:07:40.6293688Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-12-18T00:07:40.6328537Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-12-18T00:07:40.6370145Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-12-18T00:07:40.6371424Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-12-18T00:07:40.6372592Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-12-18T00:07:40.6373710Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-12-18T00:07:40.6374809Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-12-18T00:07:40.6375911Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-12-18T00:07:40.6376989Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-12-18T00:07:40.6378077Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-12-18T00:07:40.6438109Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-12-18T00:07:40.6500521Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-12-18T00:07:40.6501645Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-12-18T00:07:40.6502577Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-12-18T00:07:40.6503357Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-12-18T00:07:40.6504195Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-12-18T00:07:40.6505074Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-12-18T00:07:40.6506063Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-12-18T00:07:40.6507184Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-12-18T00:07:40.6508447Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-12-18T00:07:40.6509445Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-12-18T00:07:40.6510465Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-12-18T00:07:40.6511700Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-12-18T00:07:40.6512720Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-12-18T00:07:40.6513729Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-12-18T00:07:40.6514728Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-12-18T00:07:40.6515799Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-12-18T00:07:40.6636491Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-12-18T00:07:40.6637520Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-12-18T00:07:40.6638526Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-12-18T00:07:40.6639678Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-12-18T00:07:40.6640776Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-12-18T00:07:40.6641810Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-12-18T00:07:40.6642878Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-12-18T00:07:40.6643947Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-12-18T00:07:40.6645010Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-12-18T00:07:40.6646059Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-12-18T00:07:40.6647109Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-12-18T00:07:40.6661867Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-12-18T00:07:40.6721151Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-12-18T00:07:40.6722313Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T00:07:40.6723301Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-12-18T00:07:40.6724184Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-12-18T00:07:40.6724981Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-12-18T00:07:40.6725783Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-12-18T00:07:40.6726595Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-12-18T00:07:40.6727351Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-12-18T00:07:40.6728046Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-12-18T00:07:40.6728759Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-12-18T00:07:40.6829759Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-12-18T00:07:40.6876896Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-12-18T00:07:40.6877655Z creating: build/lib/ 2024-12-18T00:07:40.6886122Z inflating: build/lib/libpthreadpool.a 2024-12-18T00:07:40.6894755Z inflating: build/lib/libcpuinfo.a 2024-12-18T00:07:40.6903505Z inflating: build/lib/libcpuinfo_internals.a 2024-12-18T00:07:40.6904381Z inflating: build/lib/libclog.a 2024-12-18T00:07:40.6997155Z inflating: build/lib/libprotobuf-lite.a 2024-12-18T00:07:40.6997834Z inflating: build/lib/libnnpack_reference_layers.a 2024-12-18T00:07:40.7469306Z inflating: build/lib/libprotobuf.a 2024-12-18T00:07:40.7539602Z inflating: build/lib/libgtest.a 2024-12-18T00:07:40.7617957Z inflating: build/lib/libbenchmark.a 2024-12-18T00:07:40.7624485Z inflating: build/lib/libittnotify.a 2024-12-18T00:07:40.7655512Z inflating: build/lib/libtensorpipe_uv.a 2024-12-18T00:07:40.7722096Z inflating: build/lib/libasmjit.a 2024-12-18T00:07:40.7740258Z inflating: build/lib/libnnpack.a 2024-12-18T00:07:40.7873633Z inflating: build/lib/libgloo.a 2024-12-18T00:07:40.7895290Z inflating: build/lib/libfmt.a 2024-12-18T00:07:40.7895820Z inflating: build/lib/libcaffe2_nvrtc.so 2024-12-18T00:07:40.8000543Z inflating: build/lib/libc10.so 2024-12-18T00:07:40.8194441Z inflating: build/lib/libmicrokernels-prod.a 2024-12-18T00:07:40.8195024Z inflating: build/lib/libtorch_global_deps.so 2024-12-18T00:07:40.8714539Z inflating: build/lib/libprotoc.a 2024-12-18T00:07:40.8734157Z inflating: build/lib/libpytorch_qnnpack.a 2024-12-18T00:07:40.8754564Z inflating: build/lib/libgmock.a 2024-12-18T00:07:40.8755284Z inflating: build/lib/libgtest_main.a 2024-12-18T00:07:40.8755693Z inflating: build/lib/libbenchmark_main.a 2024-12-18T00:07:41.9096177Z inflating: build/lib/libdnnl.a 2024-12-18T00:07:41.9682732Z inflating: build/lib/libtensorpipe.a 2024-12-18T00:07:41.9744290Z inflating: build/lib/libc10_cuda.so 2024-12-18T00:07:42.1002526Z inflating: build/lib/libfbgemm.a 2024-12-18T00:07:42.1003220Z inflating: build/lib/libgmock_main.a 2024-12-18T00:07:42.1529949Z inflating: build/lib/libkineto.a 2024-12-18T00:07:42.1789706Z inflating: build/lib/libtensorpipe_cuda.a 2024-12-18T00:07:42.1883707Z inflating: build/lib/libXNNPACK.a 2024-12-18T00:07:42.2274617Z inflating: build/lib/libgloo_cuda.a 2024-12-18T00:07:42.2318511Z inflating: build/lib/libonnx_proto.a 2024-12-18T00:07:42.3060276Z inflating: build/lib/libonnx.a 2024-12-18T00:07:42.3951685Z inflating: build/lib/libmicrokernels-all.a 2024-12-18T00:07:44.9690504Z inflating: build/lib/libtorch_cpu.so 2024-12-18T00:07:44.9694008Z inflating: build/lib/libunbox_lib.a 2024-12-18T00:07:44.9698244Z inflating: build/lib/libshm.so 2024-12-18T00:07:46.6837075Z inflating: build/lib/libtorch_cuda.so 2024-12-18T00:07:46.6837587Z inflating: build/lib/libtorch.so 2024-12-18T00:07:46.6839083Z inflating: build/lib/libc10d_cuda_test.so 2024-12-18T00:07:47.6400468Z inflating: build/lib/libtorch_cuda_linalg.so 2024-12-18T00:07:47.6420594Z inflating: build/lib/libjitbackend_test.so 2024-12-18T00:07:47.6494588Z inflating: build/lib/libtorchbind_test.so 2024-12-18T00:07:47.6521532Z inflating: build/lib/libbackend_with_compiler.so 2024-12-18T00:07:47.6547133Z inflating: build/lib/libaoti_custom_ops.so 2024-12-18T00:07:47.8686837Z inflating: build/lib/libtorch_python.so 2024-12-18T00:07:47.8723779Z inflating: build/lib/libnnapi_backend.so 2024-12-18T00:07:47.8724446Z creating: build/bin/ 2024-12-18T00:07:47.8780100Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-12-18T00:07:47.8837282Z inflating: build/bin/c10_DeviceGuard_test 2024-12-18T00:07:47.8895850Z inflating: build/bin/c10_Device_test 2024-12-18T00:07:47.8963279Z inflating: build/bin/c10_DispatchKeySet_test 2024-12-18T00:07:47.9021741Z inflating: build/bin/c10_Scalar_test 2024-12-18T00:07:47.9078558Z inflating: build/bin/c10_StreamGuard_test 2024-12-18T00:07:47.9134226Z inflating: build/bin/c10_SymInt_test 2024-12-18T00:07:47.9195814Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-12-18T00:07:47.9259355Z inflating: build/bin/c10_InlineStreamGuard_test 2024-12-18T00:07:47.9320974Z inflating: build/bin/c10_SizesAndStrides_test 2024-12-18T00:07:47.9376664Z inflating: build/bin/c10_ArrayRef_test 2024-12-18T00:07:47.9454574Z inflating: build/bin/c10_cow_test 2024-12-18T00:07:47.9508518Z inflating: build/bin/c10_ConstexprCrc_test 2024-12-18T00:07:47.9568931Z inflating: build/bin/c10_Bitset_test 2024-12-18T00:07:47.9623521Z inflating: build/bin/c10_DeadlockDetection_test 2024-12-18T00:07:47.9681635Z inflating: build/bin/c10_Half_test 2024-12-18T00:07:47.9743307Z inflating: build/bin/c10_LeftRight_test 2024-12-18T00:07:47.9805013Z inflating: build/bin/c10_Metaprogramming_test 2024-12-18T00:07:47.9863470Z inflating: build/bin/c10_NetworkFlow_test 2024-12-18T00:07:47.9921630Z inflating: build/bin/c10_Synchronized_test 2024-12-18T00:07:47.9981876Z inflating: build/bin/c10_ThreadLocal_test 2024-12-18T00:07:48.0040954Z inflating: build/bin/c10_TypeIndex_test 2024-12-18T00:07:48.0098914Z inflating: build/bin/c10_TypeList_test 2024-12-18T00:07:48.0154356Z inflating: build/bin/c10_TypeTraits_test 2024-12-18T00:07:48.0211216Z inflating: build/bin/c10_accumulate_test 2024-12-18T00:07:48.0276199Z inflating: build/bin/c10_bfloat16_test 2024-12-18T00:07:48.0332415Z inflating: build/bin/c10_bit_cast_test 2024-12-18T00:07:48.0395437Z inflating: build/bin/c10_complex_math_test 2024-12-18T00:07:48.0457097Z inflating: build/bin/c10_complex_test 2024-12-18T00:07:48.0513606Z inflating: build/bin/c10_error_test 2024-12-18T00:07:48.0572065Z inflating: build/bin/c10_exception_test 2024-12-18T00:07:48.0627113Z inflating: build/bin/c10_flags_test 2024-12-18T00:07:48.0685011Z inflating: build/bin/c10_generic_math_test 2024-12-18T00:07:48.0859847Z inflating: build/bin/c10_intrusive_ptr_test 2024-12-18T00:07:48.0916273Z inflating: build/bin/c10_irange_test 2024-12-18T00:07:48.0975976Z inflating: build/bin/c10_lazy_test 2024-12-18T00:07:48.1040197Z inflating: build/bin/c10_logging_test 2024-12-18T00:07:48.1123275Z inflating: build/bin/c10_optional_test 2024-12-18T00:07:48.1193536Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-12-18T00:07:48.1253840Z inflating: build/bin/c10_registry_test 2024-12-18T00:07:48.1415606Z inflating: build/bin/c10_small_vector_test 2024-12-18T00:07:48.1474944Z inflating: build/bin/c10_ssize_test 2024-12-18T00:07:48.1532902Z inflating: build/bin/c10_string_util_test 2024-12-18T00:07:48.1587792Z inflating: build/bin/c10_tempfile_test 2024-12-18T00:07:48.1655092Z inflating: build/bin/c10_string_view_test 2024-12-18T00:07:48.1718391Z inflating: build/bin/c10_typeid_test 2024-12-18T00:07:48.1770141Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-12-18T00:07:48.2235408Z inflating: build/bin/protoc-3.13.0.0 2024-12-18T00:07:48.2700215Z inflating: build/bin/protoc 2024-12-18T00:07:48.2760465Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-12-18T00:07:48.2818304Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-12-18T00:07:48.2878345Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-12-18T00:07:48.2934594Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-12-18T00:07:48.2989041Z inflating: build/bin/c10_cuda_CUDATest 2024-12-18T00:07:48.3050087Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-12-18T00:07:48.3107086Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-12-18T00:07:48.3167891Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-12-18T00:07:48.3575533Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-12-18T00:07:48.3996660Z inflating: build/bin/vec_test_all_types_AVX512 2024-12-18T00:07:48.4433752Z inflating: build/bin/vec_test_all_types_AVX2 2024-12-18T00:07:48.4491769Z inflating: build/bin/HashStoreTest 2024-12-18T00:07:48.4553220Z inflating: build/bin/FileStoreTest 2024-12-18T00:07:48.4613790Z inflating: build/bin/TCPStoreTest 2024-12-18T00:07:48.4673778Z inflating: build/bin/BackoffTest 2024-12-18T00:07:48.4689136Z inflating: build/bin/ProcessGroupMPITest 2024-12-18T00:07:48.4691443Z inflating: build/bin/torch_shm_manager 2024-12-18T00:07:48.4754002Z inflating: build/bin/test_edge_op_registration 2024-12-18T00:07:48.4755727Z inflating: build/bin/example_allreduce 2024-12-18T00:07:48.4813954Z inflating: build/bin/Dimname_test 2024-12-18T00:07:48.4895747Z inflating: build/bin/Dict_test 2024-12-18T00:07:48.4968108Z inflating: build/bin/MaybeOwned_test 2024-12-18T00:07:48.5031322Z inflating: build/bin/NamedTensor_test 2024-12-18T00:07:48.5098785Z inflating: build/bin/apply_utils_test 2024-12-18T00:07:48.5164776Z inflating: build/bin/atest 2024-12-18T00:07:48.5233953Z inflating: build/bin/basic 2024-12-18T00:07:48.5297393Z inflating: build/bin/broadcast_test 2024-12-18T00:07:48.5355554Z inflating: build/bin/cpu_allocator_test 2024-12-18T00:07:48.5419520Z inflating: build/bin/cpu_generator_test 2024-12-18T00:07:48.5479659Z inflating: build/bin/cpu_profiling_allocator_test 2024-12-18T00:07:48.5579823Z inflating: build/bin/cpu_rng_test 2024-12-18T00:07:48.5635286Z inflating: build/bin/dispatch_key_set_test 2024-12-18T00:07:48.5692222Z inflating: build/bin/dlconvertor_test 2024-12-18T00:07:48.5758132Z inflating: build/bin/extension_backend_test 2024-12-18T00:07:48.5818413Z inflating: build/bin/half_test 2024-12-18T00:07:48.5922740Z inflating: build/bin/ivalue_test 2024-12-18T00:07:48.5978749Z inflating: build/bin/lazy_tensor_test 2024-12-18T00:07:48.6038654Z inflating: build/bin/math_kernel_test 2024-12-18T00:07:48.6098623Z inflating: build/bin/memory_format_test 2024-12-18T00:07:48.6158662Z inflating: build/bin/memory_overlapping_test 2024-12-18T00:07:48.6216481Z inflating: build/bin/mobile_memory_cleanup 2024-12-18T00:07:48.6275788Z inflating: build/bin/operator_name_test 2024-12-18T00:07:48.6337667Z inflating: build/bin/native_test 2024-12-18T00:07:48.6396718Z inflating: build/bin/packedtensoraccessor_test 2024-12-18T00:07:48.6454822Z inflating: build/bin/operators_test 2024-12-18T00:07:48.6527834Z inflating: build/bin/pow_test 2024-12-18T00:07:48.6593502Z inflating: build/bin/quantized_test 2024-12-18T00:07:48.6650139Z inflating: build/bin/reduce_ops_test 2024-12-18T00:07:48.6706147Z inflating: build/bin/reportMemoryUsage_test 2024-12-18T00:07:48.6765659Z inflating: build/bin/StorageUtils_test 2024-12-18T00:07:48.6827643Z inflating: build/bin/scalar_tensor_test 2024-12-18T00:07:48.6893816Z inflating: build/bin/scalar_test 2024-12-18T00:07:48.6953775Z inflating: build/bin/stride_properties_test 2024-12-18T00:07:48.7040114Z inflating: build/bin/tensor_iterator_test 2024-12-18T00:07:48.7101103Z inflating: build/bin/test_parallel 2024-12-18T00:07:48.7164265Z inflating: build/bin/type_ptr_test 2024-12-18T00:07:48.7165794Z inflating: build/bin/thread_init_test 2024-12-18T00:07:48.7232636Z inflating: build/bin/type_test 2024-12-18T00:07:48.7293299Z inflating: build/bin/undefined_tensor_test 2024-12-18T00:07:48.7293878Z inflating: build/bin/verify_api_visibility 2024-12-18T00:07:48.7353394Z inflating: build/bin/wrapdim_test 2024-12-18T00:07:48.7430392Z inflating: build/bin/legacy_vmap_test 2024-12-18T00:07:48.7488397Z inflating: build/bin/weakref_test 2024-12-18T00:07:48.7545940Z inflating: build/bin/xla_tensor_test 2024-12-18T00:07:48.7614110Z inflating: build/bin/IListRef_test 2024-12-18T00:07:48.7728408Z inflating: build/bin/List_test 2024-12-18T00:07:48.7857996Z inflating: build/bin/kernel_function_legacy_test 2024-12-18T00:07:48.7962356Z inflating: build/bin/kernel_function_test 2024-12-18T00:07:48.8034177Z inflating: build/bin/KernelFunction_test 2024-12-18T00:07:48.8145336Z inflating: build/bin/kernel_lambda_test 2024-12-18T00:07:48.8283218Z inflating: build/bin/kernel_lambda_legacy_test 2024-12-18T00:07:48.8353490Z inflating: build/bin/kernel_stackbased_test 2024-12-18T00:07:48.8409049Z inflating: build/bin/CppSignature_test 2024-12-18T00:07:48.8514446Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-12-18T00:07:48.8569168Z inflating: build/bin/op_allowlist_test 2024-12-18T00:07:48.8883819Z inflating: build/bin/op_registration_test 2024-12-18T00:07:48.8946015Z inflating: build/bin/backend_fallback_test 2024-12-18T00:07:48.9016124Z inflating: build/bin/inline_container_test 2024-12-18T00:07:48.9076653Z inflating: build/bin/cuda_apply_test 2024-12-18T00:07:48.9133449Z inflating: build/bin/cuda_allocator_test 2024-12-18T00:07:48.9200064Z inflating: build/bin/cuda_atomic_ops_test 2024-12-18T00:07:48.9261315Z inflating: build/bin/cuda_caching_host_allocator_test 2024-12-18T00:07:48.9336923Z inflating: build/bin/cuda_complex_math_test 2024-12-18T00:07:48.9403159Z inflating: build/bin/cuda_complex_test 2024-12-18T00:07:48.9459496Z inflating: build/bin/cuda_device_test 2024-12-18T00:07:48.9523954Z inflating: build/bin/cuda_cub_test 2024-12-18T00:07:48.9580250Z inflating: build/bin/cuda_dlconvertor_test 2024-12-18T00:07:48.9637448Z inflating: build/bin/cuda_integer_divider_test 2024-12-18T00:07:48.9708484Z inflating: build/bin/cuda_distributions_test 2024-12-18T00:07:48.9772636Z inflating: build/bin/cuda_generator_test 2024-12-18T00:07:48.9826720Z inflating: build/bin/cuda_half_test 2024-12-18T00:07:48.9886621Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-12-18T00:07:48.9941301Z inflating: build/bin/cuda_optional_test 2024-12-18T00:07:48.9998061Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-12-18T00:07:49.0064685Z inflating: build/bin/cuda_stream_test 2024-12-18T00:07:49.0124224Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-12-18T00:07:49.0180218Z inflating: build/bin/cuda_cudnn_test 2024-12-18T00:07:49.0236932Z inflating: build/bin/cuda_vectorized_test 2024-12-18T00:07:49.0309208Z inflating: build/bin/ProcessGroupGlooTest 2024-12-18T00:07:49.0326963Z inflating: build/bin/tutorial_tensorexpr 2024-12-18T00:07:49.0389619Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-12-18T00:07:49.0460376Z inflating: build/bin/ProcessGroupNCCLTest 2024-12-18T00:07:49.0528800Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-12-18T00:07:49.0585612Z inflating: build/bin/ProcessGroupUCCTest 2024-12-18T00:07:49.0649368Z inflating: build/bin/test_dist_autograd 2024-12-18T00:07:49.0650083Z inflating: build/bin/parallel_benchmark 2024-12-18T00:07:49.0727523Z inflating: build/bin/test_cpp_rpc 2024-12-18T00:07:49.0801411Z inflating: build/bin/test_mobile_nnc 2024-12-18T00:07:49.0809526Z inflating: build/bin/aot_model_compiler_test 2024-12-18T00:07:49.1171644Z inflating: build/bin/test_lazy 2024-12-18T00:07:49.2038882Z inflating: build/bin/test_tensorexpr 2024-12-18T00:07:49.3265602Z inflating: build/bin/test_api 2024-12-18T00:07:49.3870814Z inflating: build/bin/test_jit 2024-12-18T00:07:49.3871414Z creating: .additional_ci_files/ 2024-12-18T00:07:49.3965780Z inflating: .additional_ci_files/test-times.json 2024-12-18T00:07:49.4336519Z inflating: .additional_ci_files/test-class-times.json 2024-12-18T00:07:49.4379610Z ##[group]Run rm artifacts.zip 2024-12-18T00:07:49.4379991Z rm artifacts.zip 2024-12-18T00:07:49.4386350Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:49.4386897Z env: 2024-12-18T00:07:49.4387144Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:49.4387506Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:49.4387898Z ##[endgroup] 2024-12-18T00:07:49.5023852Z ##[group]Run df -H 2024-12-18T00:07:49.5024158Z df -H 2024-12-18T00:07:49.5029888Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:49.5030334Z env: 2024-12-18T00:07:49.5030601Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:49.5031103Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:49.5031894Z ##[endgroup] 2024-12-18T00:07:49.5078241Z Filesystem Size Used Avail Use% Mounted on 2024-12-18T00:07:49.5078819Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-12-18T00:07:49.5079219Z tmpfs 101G 0 101G 0% /dev/shm 2024-12-18T00:07:49.5079626Z tmpfs 41G 697k 41G 1% /run 2024-12-18T00:07:49.5080032Z /dev/nvme0n1p1 161G 53G 109G 33% / 2024-12-18T00:07:49.5080602Z tmpfs 101G 13k 101G 1% /tmp 2024-12-18T00:07:49.5081011Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-12-18T00:07:49.5081437Z tmpfs 21G 0 21G 0% /run/user/0 2024-12-18T00:07:49.5116611Z Prepare all required actions 2024-12-18T00:07:49.5117443Z Getting action download info 2024-12-18T00:07:49.6515137Z ##[group]Run ./.github/actions/download-td-artifacts 2024-12-18T00:07:49.6515538Z with: 2024-12-18T00:07:49.6515765Z env: 2024-12-18T00:07:49.6515999Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:49.6516350Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:49.6516731Z ##[endgroup] 2024-12-18T00:07:49.6551461Z ##[group]Run seemethere/download-artifact-s3@v4 2024-12-18T00:07:49.6551957Z with: 2024-12-18T00:07:49.6552192Z name: td_results 2024-12-18T00:07:49.6552461Z s3-bucket: gha-artifacts 2024-12-18T00:07:49.6552737Z region: us-east-1 2024-12-18T00:07:49.6552989Z env: 2024-12-18T00:07:49.6553225Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:49.6553603Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:49.6553990Z ##[endgroup] 2024-12-18T00:07:50.1324436Z (node:59036) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-12-18T00:07:50.1325016Z 2024-12-18T00:07:50.1325253Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-12-18T00:07:50.1325864Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-12-18T00:07:50.1326514Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-12-18T00:07:50.2383495Z Found 1 objects with prefix pytorch/pytorch/12383255652/td_results/ 2024-12-18T00:07:50.2384273Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-12-18T00:07:50.3271220Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-12-18T00:07:50.3275141Z Artifact download has finished successfully 2024-12-18T00:07:50.3462922Z ##[group]Run mkdir -p .additional_ci_files 2024-12-18T00:07:50.3463379Z mkdir -p .additional_ci_files 2024-12-18T00:07:50.3463922Z mv td_results.json .additional_ci_files/td_results.json || true 2024-12-18T00:07:50.3471123Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:50.3471538Z env: 2024-12-18T00:07:50.3471791Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:50.3472153Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:50.3472544Z ##[endgroup] 2024-12-18T00:07:50.3561236Z ##[group]Run .github/scripts/parse_ref.py 2024-12-18T00:07:50.3561668Z .github/scripts/parse_ref.py 2024-12-18T00:07:50.3567251Z shell: /usr/bin/bash -e {0} 2024-12-18T00:07:50.3567572Z env: 2024-12-18T00:07:50.3567828Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:50.3568216Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:50.3568705Z ##[endgroup] 2024-12-18T00:07:50.3852022Z Prepare all required actions 2024-12-18T00:07:50.3891880Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-12-18T00:07:50.3892278Z with: 2024-12-18T00:07:50.3892698Z github-token: *** 2024-12-18T00:07:50.3892974Z env: 2024-12-18T00:07:50.3893226Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:50.3893610Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:50.3894017Z ##[endgroup] 2024-12-18T00:07:50.3916520Z ##[group]Run set -eux 2024-12-18T00:07:50.3916828Z set -eux 2024-12-18T00:07:50.3917313Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-12-18T00:07:50.3922947Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:50.3923349Z env: 2024-12-18T00:07:50.3923592Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:50.3923954Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:50.3924497Z GITHUB_TOKEN: *** 2024-12-18T00:07:50.3924756Z ##[endgroup] 2024-12-18T00:07:50.3948493Z + python3 .github/scripts/get_workflow_job_id.py 12383255652 i-0d4c5ae1b54bc839d 2024-12-18T00:07:52.5963245Z setting job-id=34566462667 2024-12-18T00:07:52.5964185Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:07:52.6151477Z Prepare all required actions 2024-12-18T00:07:52.6151917Z Getting action download info 2024-12-18T00:07:52.7591192Z ##[group]Run ./.github/actions/filter-test-configs 2024-12-18T00:07:52.7591686Z with: 2024-12-18T00:07:52.7592094Z github-token: *** 2024-12-18T00:07:52.7593302Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]} 2024-12-18T00:07:52.7594818Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:07:52.7595411Z env: 2024-12-18T00:07:52.7595635Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:52.7595994Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:52.7596379Z ##[endgroup] 2024-12-18T00:07:52.7666899Z ##[group]Run nick-fields/retry@v3.0.0 2024-12-18T00:07:52.7667272Z with: 2024-12-18T00:07:52.7667510Z shell: bash 2024-12-18T00:07:52.7667773Z timeout_minutes: 10 2024-12-18T00:07:52.7668062Z max_attempts: 5 2024-12-18T00:07:52.7668345Z retry_wait_seconds: 30 2024-12-18T00:07:52.7669281Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-12-18T00:07:52.7670255Z polling_interval_seconds: 1 2024-12-18T00:07:52.7670596Z warning_on_retry: true 2024-12-18T00:07:52.7671017Z continue_on_error: false 2024-12-18T00:07:52.7671409Z env: 2024-12-18T00:07:52.7671631Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:52.7671974Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:52.7672515Z GITHUB_TOKEN: *** 2024-12-18T00:07:52.7672763Z ##[endgroup] 2024-12-18T00:07:52.8665259Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-12-18T00:07:53.1220194Z Defaulting to user installation because normal site-packages is not writeable 2024-12-18T00:07:53.2700196Z Collecting requests==2.27.1 2024-12-18T00:07:53.3160849Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-12-18T00:07:53.5050336Z Collecting pyyaml==6.0.1 2024-12-18T00:07:53.5097116Z Downloading PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB) 2024-12-18T00:07:53.5411050Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-12-18T00:07:53.8615081Z Collecting charset-normalizer~=2.0.0 2024-12-18T00:07:53.8669218Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-12-18T00:07:53.8728106Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-12-18T00:07:53.9220271Z Collecting certifi>=2017.4.17 2024-12-18T00:07:53.9277833Z Downloading certifi-2024.12.14-py3-none-any.whl (164 kB) 2024-12-18T00:07:54.0147416Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2024-12-18T00:07:54.1405017Z Successfully installed certifi-2024.12.14 charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 2024-12-18T00:07:54.8486153Z Command completed after 1 attempt(s). 2024-12-18T00:07:54.8535906Z ##[group]Run set -x 2024-12-18T00:07:54.8536224Z set -x 2024-12-18T00:07:54.8536499Z  2024-12-18T00:07:54.8536951Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-12-18T00:07:54.8537512Z # in runner workspace 2024-12-18T00:07:54.8537968Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-12-18T00:07:54.8545053Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:54.8545477Z env: 2024-12-18T00:07:54.8545722Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:54.8546088Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:54.8546480Z ##[endgroup] 2024-12-18T00:07:54.8571939Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-12-18T00:07:54.8829279Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-12-18T00:07:54.8829709Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-12-18T00:07:54.8830077Z echo "Job name: ${JOB_NAME}" 2024-12-18T00:07:54.8830400Z  2024-12-18T00:07:54.8830810Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-12-18T00:07:54.8831320Z # in runner workspace 2024-12-18T00:07:54.8832109Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-12-18T00:07:54.8832669Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-12-18T00:07:54.8833080Z  --job-name "${JOB_NAME}" \ 2024-12-18T00:07:54.8834506Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]}" \ 2024-12-18T00:07:54.8835952Z  --selected-test-configs "" \ 2024-12-18T00:07:54.8836355Z  --pr-number "${PR_NUMBER}" \ 2024-12-18T00:07:54.8836729Z  --tag "${TAG}" \ 2024-12-18T00:07:54.8837062Z  --event-name "${EVENT_NAME}" \ 2024-12-18T00:07:54.8837448Z  --schedule "${SCHEDULE}" \ 2024-12-18T00:07:54.8837817Z  --branch "${HEAD_BRANCH}" 2024-12-18T00:07:54.8843843Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:54.8844396Z env: 2024-12-18T00:07:54.8844643Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:54.8845013Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:54.8845604Z GITHUB_TOKEN: *** 2024-12-18T00:07:54.8846138Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:07:54.8846708Z PR_NUMBER: 2024-12-18T00:07:54.8846953Z TAG: 2024-12-18T00:07:54.8847187Z EVENT_NAME: push 2024-12-18T00:07:54.8847449Z SCHEDULE: 2024-12-18T00:07:54.8847689Z HEAD_BRANCH: 2024-12-18T00:07:54.8847921Z ##[endgroup] 2024-12-18T00:07:54.8871094Z Workflow: pull 2024-12-18T00:07:54.8871711Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:07:55.1552944Z ##[group]Run echo "Filtered matrix:" 2024-12-18T00:07:55.1553321Z echo "Filtered matrix:" 2024-12-18T00:07:55.1554582Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]}" 2024-12-18T00:07:55.1555992Z  2024-12-18T00:07:55.1556240Z echo 2024-12-18T00:07:55.1556549Z echo "Is the current job unstable? False" 2024-12-18T00:07:55.1556920Z  2024-12-18T00:07:55.1557154Z echo 2024-12-18T00:07:55.1557434Z echo "Is keep-going label set? False" 2024-12-18T00:07:55.1557784Z  2024-12-18T00:07:55.1558015Z echo 2024-12-18T00:07:55.1558278Z echo "Renabled issues? " 2024-12-18T00:07:55.1563877Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:55.1564270Z env: 2024-12-18T00:07:55.1564515Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:55.1564878Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:55.1565261Z ##[endgroup] 2024-12-18T00:07:55.1589646Z Filtered matrix: 2024-12-18T00:07:55.1591333Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.g4dn.12xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: linux.g4dn.12xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: linux.g4dn.12xlarge.nvidia.gpu}]} 2024-12-18T00:07:55.1592576Z 2024-12-18T00:07:55.1592712Z Is the current job unstable? False 2024-12-18T00:07:55.1592964Z 2024-12-18T00:07:55.1593096Z Is keep-going label set? False 2024-12-18T00:07:55.1593327Z 2024-12-18T00:07:55.1593439Z Renabled issues? 2024-12-18T00:07:55.1642073Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-12-18T00:07:55.1642670Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-12-18T00:07:55.1648527Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:55.1648943Z env: 2024-12-18T00:07:55.1649185Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:55.1649552Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:55.1649954Z JOB_TIMEOUT: 360 2024-12-18T00:07:55.1650196Z ##[endgroup] 2024-12-18T00:07:55.1729268Z ##[group]Run set -x 2024-12-18T00:07:55.1729612Z set -x 2024-12-18T00:07:55.1729869Z  2024-12-18T00:07:55.1730157Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-12-18T00:07:55.1730575Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-12-18T00:07:55.1731194Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-12-18T00:07:55.1731791Z  TEST_COMMAND=.ci/onnx/test.sh 2024-12-18T00:07:55.1732321Z else 2024-12-18T00:07:55.1732700Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-12-18T00:07:55.1733061Z fi 2024-12-18T00:07:55.1733317Z  2024-12-18T00:07:55.1733719Z # detached container should get cleaned up by teardown_ec2_linux 2024-12-18T00:07:55.1734346Z # TODO: Stop building test binaries as part of the build phase 2024-12-18T00:07:55.1734901Z # Used for GPU_FLAG since that doesn't play nice 2024-12-18T00:07:55.1735399Z # shellcheck disable=SC2086,SC2090 2024-12-18T00:07:55.1735800Z container_name=$(docker run \ 2024-12-18T00:07:55.1736155Z  ${GPU_FLAG:-} \ 2024-12-18T00:07:55.1736513Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2024-12-18T00:07:55.1736922Z  -e BUILD_ENVIRONMENT \ 2024-12-18T00:07:55.1737278Z  -e PR_NUMBER \ 2024-12-18T00:07:55.1737604Z  -e GITHUB_ACTIONS \ 2024-12-18T00:07:55.1737940Z  -e GITHUB_REPOSITORY \ 2024-12-18T00:07:55.1738298Z  -e GITHUB_WORKFLOW \ 2024-12-18T00:07:55.1738725Z  -e GITHUB_JOB \ 2024-12-18T00:07:55.1739225Z  -e GITHUB_RUN_ID \ 2024-12-18T00:07:55.1739571Z  -e GITHUB_RUN_NUMBER \ 2024-12-18T00:07:55.1739923Z  -e GITHUB_RUN_ATTEMPT \ 2024-12-18T00:07:55.1740283Z  -e JOB_ID \ 2024-12-18T00:07:55.1740604Z  -e JOB_NAME \ 2024-12-18T00:07:55.1740925Z  -e BASE_SHA \ 2024-12-18T00:07:55.1741396Z  -e BRANCH \ 2024-12-18T00:07:55.1741696Z  -e SHA1 \ 2024-12-18T00:07:55.1742005Z  -e AWS_DEFAULT_REGION \ 2024-12-18T00:07:55.1742370Z  -e IN_WHEEL_TEST \ 2024-12-18T00:07:55.1742708Z  -e SHARD_NUMBER \ 2024-12-18T00:07:55.1743043Z  -e TEST_CONFIG \ 2024-12-18T00:07:55.1743369Z  -e NUM_TEST_SHARDS \ 2024-12-18T00:07:55.1743728Z  -e REENABLED_ISSUES \ 2024-12-18T00:07:55.1744099Z  -e CONTINUE_THROUGH_ERROR \ 2024-12-18T00:07:55.1744481Z  -e VERBOSE_TEST_LOGS \ 2024-12-18T00:07:55.1744844Z  -e TEST_SHOWLOCALS \ 2024-12-18T00:07:55.1745186Z  -e NO_TEST_TIMEOUT \ 2024-12-18T00:07:55.1745533Z  -e NO_TD \ 2024-12-18T00:07:55.1745844Z  -e TD_DISTRIBUTED \ 2024-12-18T00:07:55.1746192Z  -e PR_LABELS \ 2024-12-18T00:07:55.1746556Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-12-18T00:07:55.1746953Z  -e SCCACHE_BUCKET \ 2024-12-18T00:07:55.1747305Z  -e SCCACHE_REGION \ 2024-12-18T00:07:55.1747661Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-12-18T00:07:55.1748029Z  -e XLA_CUDA \ 2024-12-18T00:07:55.1748383Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-12-18T00:07:55.1748822Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-12-18T00:07:55.1749264Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-12-18T00:07:55.1749711Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-12-18T00:07:55.1750238Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-12-18T00:07:55.1750629Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-12-18T00:07:55.1751095Z  -e DASHBOARD_TAG \ 2024-12-18T00:07:55.1751389Z  -e IS_A100_RUNNER \ 2024-12-18T00:07:55.1751705Z  -e ARTIFACTS_FILE_SUFFIX \ 2024-12-18T00:07:55.1752103Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-12-18T00:07:55.1752530Z  --security-opt seccomp=unconfined \ 2024-12-18T00:07:55.1753034Z  --cap-add=SYS_PTRACE \ 2024-12-18T00:07:55.1753346Z  --ipc=host \ 2024-12-18T00:07:55.1753643Z  --shm-size="${SHM_SIZE}" \ 2024-12-18T00:07:55.1753964Z  --tty \ 2024-12-18T00:07:55.1754228Z  --detach \ 2024-12-18T00:07:55.1754524Z  --name="${container_name}" \ 2024-12-18T00:07:55.1754851Z  --user jenkins \ 2024-12-18T00:07:55.1755235Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-12-18T00:07:55.1755667Z  -w /var/lib/jenkins/workspace \ 2024-12-18T00:07:55.1756016Z  "${DOCKER_IMAGE}" 2024-12-18T00:07:55.1756306Z ) 2024-12-18T00:07:55.1756633Z # Propagate download.pytorch.org IP to container 2024-12-18T00:07:55.1757335Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-12-18T00:07:55.1758100Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-12-18T00:07:55.1758855Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-12-18T00:07:55.1764186Z shell: /usr/bin/bash -e {0} 2024-12-18T00:07:55.1764484Z env: 2024-12-18T00:07:55.1764721Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:55.1765070Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:55.1765529Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-12-18T00:07:55.1765916Z PR_NUMBER: 2024-12-18T00:07:55.1766190Z GITHUB_REPOSITORY: pytorch/pytorch 2024-12-18T00:07:55.1766529Z GITHUB_WORKFLOW: pull 2024-12-18T00:07:55.1766795Z GITHUB_JOB: test 2024-12-18T00:07:55.1767062Z GITHUB_RUN_ID: 12383255652 2024-12-18T00:07:55.1767362Z GITHUB_RUN_NUMBER: 276594 2024-12-18T00:07:55.1767662Z GITHUB_RUN_ATTEMPT: 1 2024-12-18T00:07:55.1767934Z JOB_ID: 34566462667 2024-12-18T00:07:55.1768446Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:07:55.1769115Z BRANCH: release/2.6 2024-12-18T00:07:55.1769428Z SHA1: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:07:55.1769834Z BASE_SHA: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:07:55.1770211Z TEST_CONFIG: distributed 2024-12-18T00:07:55.1770499Z SHARD_NUMBER: 1 2024-12-18T00:07:55.1770744Z NUM_TEST_SHARDS: 3 2024-12-18T00:07:55.1771009Z REENABLED_ISSUES: 2024-12-18T00:07:55.1771288Z CONTINUE_THROUGH_ERROR: False 2024-12-18T00:07:55.1771600Z VERBOSE_TEST_LOGS: False 2024-12-18T00:07:55.1771898Z TEST_SHOWLOCALS: False 2024-12-18T00:07:55.1772176Z NO_TEST_TIMEOUT: False 2024-12-18T00:07:55.1772447Z NO_TD: False 2024-12-18T00:07:55.1772697Z TD_DISTRIBUTED: False 2024-12-18T00:07:55.1773032Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-12-18T00:07:55.1773418Z SCCACHE_REGION: us-east-1 2024-12-18T00:07:55.1773706Z SCCACHE_S3_KEY_PREFIX: pull 2024-12-18T00:07:55.1774005Z SHM_SIZE: 2g 2024-12-18T00:07:55.1774753Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:07:55.1775564Z XLA_CUDA: 2024-12-18T00:07:55.1775949Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-12-18T00:07:55.1776426Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-12-18T00:07:55.1776776Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-12-18T00:07:55.1777104Z DASHBOARD_TAG: 2024-12-18T00:07:55.1777564Z HUGGING_FACE_HUB_TOKEN: *** 2024-12-18T00:07:55.1778010Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2024-12-18T00:07:55.1778333Z IS_A100_RUNNER: 0 2024-12-18T00:07:55.1779068Z ARTIFACTS_FILE_SUFFIX: test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_34566462667 2024-12-18T00:07:55.1779734Z ##[endgroup] 2024-12-18T00:07:55.1803973Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-12-18T00:07:55.1804456Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-12-18T00:07:55.1804892Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-12-18T00:07:55.1813675Z +++ nproc --ignore=2 2024-12-18T00:07:55.1844248Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=46 -e SCCACHE_BUCKET -e SCCACHE_REGION -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e IS_A100_RUNNER -e ARTIFACTS_FILE_SUFFIX --env-file=/tmp/github_env_12383255652 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:08:06.6454947Z + container_name=04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T00:08:06.6457094Z + grep download.pytorch.org /etc/hosts 2024-12-18T00:08:06.6458771Z + docker exec -i 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b sudo bash -c '/bin/cat >> /etc/hosts' 2024-12-18T00:08:06.7652073Z + echo DOCKER_CONTAINER_ID=04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T00:08:06.7656535Z ++ echo dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl 2024-12-18T00:08:06.7659202Z + docker exec -t 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b sh -c 'python3 -m pip install dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-12-18T00:08:07.1542620Z Processing ./dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl (from torch==2.6.0a0+git0cdf8b1) 2024-12-18T00:08:08.0399117Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.16.1) 2024-12-18T00:08:08.0400734Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (4.12.2) 2024-12-18T00:08:08.0402287Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (2.8.8) 2024-12-18T00:08:08.0405356Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.1.4) 2024-12-18T00:08:08.0408538Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (2024.10.0) 2024-12-18T00:08:08.0414017Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.13.1) 2024-12-18T00:08:08.0433854Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.3.0) 2024-12-18T00:08:08.0451779Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.3.0) 2024-12-18T00:08:08.0467735Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.22.4) 2024-12-18T00:08:08.0861232Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.0.2) 2024-12-18T00:08:08.4430934Z Installing collected packages: torch 2024-12-18T00:08:19.5457348Z Successfully installed torch-2.6.0a0+git0cdf8b1 2024-12-18T00:08:19.6179753Z + export TERM=vt100 2024-12-18T00:08:19.6180108Z + TERM=vt100 2024-12-18T00:08:19.6180416Z ++ dirname .ci/pytorch/test.sh 2024-12-18T00:08:19.6188774Z + source .ci/pytorch/common.sh 2024-12-18T00:08:19.6190541Z +++ dirname .ci/pytorch/common.sh 2024-12-18T00:08:19.6203449Z ++ source .ci/pytorch/common_utils.sh 2024-12-18T00:08:19.6203879Z +++ declare -f -t trap_add 2024-12-18T00:08:19.6208977Z ++ set -ex 2024-12-18T00:08:19.6209310Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-12-18T00:08:19.6209737Z ++ BUILD_TEST_LIBTORCH=0 2024-12-18T00:08:19.6210106Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-12-18T00:08:19.6210568Z + [[ -d /var/lib/jenkins/workspace ]] 2024-12-18T00:08:19.6212471Z ++ stat -c %u /var/lib/jenkins/workspace 2024-12-18T00:08:19.6226960Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-12-18T00:08:19.6227371Z + trap_add cleanup_workspace EXIT 2024-12-18T00:08:19.6227750Z + trap_add_cmd=cleanup_workspace 2024-12-18T00:08:19.6228093Z + shift 2024-12-18T00:08:19.6228353Z + for trap_add_name in "$@" 2024-12-18T00:08:19.6235814Z +++ trap -p EXIT 2024-12-18T00:08:19.6238339Z ++ eval 'extract_trap_cmd ' 2024-12-18T00:08:19.6238678Z +++ extract_trap_cmd 2024-12-18T00:08:19.6238983Z +++ printf '%s\n' '' 2024-12-18T00:08:19.6239310Z ++ printf '%s\n' cleanup_workspace 2024-12-18T00:08:19.6240710Z + trap -- ' 2024-12-18T00:08:19.6241002Z cleanup_workspace' EXIT 2024-12-18T00:08:19.6251456Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-12-18T00:08:20.1109789Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-12-18T00:08:20.1133351Z + echo 'Environment variables:' 2024-12-18T00:08:20.1134026Z Environment variables: 2024-12-18T00:08:20.1134324Z + env 2024-12-18T00:08:20.1139267Z INSTALLED_DB=yes 2024-12-18T00:08:20.1139777Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-12-18T00:08:20.1140148Z NVIDIA_VISIBLE_DEVICES=all 2024-12-18T00:08:20.1140700Z NV_NVML_DEV_VERSION=11.8.86-1 2024-12-18T00:08:20.1141250Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T00:08:20.1141794Z CONTINUE_THROUGH_ERROR=False 2024-12-18T00:08:20.1142215Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-12-18T00:08:20.1142676Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-12-18T00:08:20.1143133Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-12-18T00:08:20.1143567Z HOSTNAME=04018ea456d8 2024-12-18T00:08:20.1144357Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_61c5da34-5695-4c33-8a53-0e5394a1a3c0 2024-12-18T00:08:20.1145088Z GITHUB_ACTION=__self 2024-12-18T00:08:20.1145408Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-12-18T00:08:20.1147260Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-12-18T00:08:20.1149189Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-12-18T00:08:20.1149626Z NV_NVTX_VERSION=11.8.86-1 2024-12-18T00:08:20.1149934Z GITHUB_RUN_NUMBER=276594 2024-12-18T00:08:20.1150252Z TEST_CONFIG=distributed 2024-12-18T00:08:20.1150575Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-12-18T00:08:20.1150962Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-12-18T00:08:20.1151330Z IS_A100_RUNNER=0 2024-12-18T00:08:20.1151612Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-12-18T00:08:20.1151980Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-12-18T00:08:20.1152579Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2024-12-18T00:08:20.1152986Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-12-18T00:08:20.1153519Z GITHUB_TRIGGERING_ACTOR=malfet 2024-12-18T00:08:20.1153923Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-12-18T00:08:20.1154345Z GITHUB_REF_TYPE=branch 2024-12-18T00:08:20.1154646Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-12-18T00:08:20.1154973Z NCCL_VERSION=2.16.2-1 2024-12-18T00:08:20.1155323Z BASE_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:20.1155717Z XLA_CUDA= 2024-12-18T00:08:20.1156104Z HUGGING_FACE_HUB_TOKEN=*** 2024-12-18T00:08:20.1157838Z *** 2024-12-18T00:08:20.1158099Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-12-18T00:08:20.1158464Z GITHUB_REPOSITORY_ID=65600975 2024-12-18T00:08:20.1158806Z GITHUB_ACTIONS=true 2024-12-18T00:08:20.1159115Z NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:08:20.1159491Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-12-18T00:08:20.1159923Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-12-18T00:08:20.1160335Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:20.1160754Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-12-18T00:08:20.1161196Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:20.1161820Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/heads/release/2.6 2024-12-18T00:08:20.1162379Z UCC_HOME=/usr 2024-12-18T00:08:20.1162668Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-12-18T00:08:20.1163026Z VERBOSE_TEST_LOGS=False 2024-12-18T00:08:20.1163336Z NVIDIA_PRODUCT_NAME=CUDA 2024-12-18T00:08:20.1163701Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-12-18T00:08:20.1164106Z GITHUB_REF=refs/heads/release/2.6 2024-12-18T00:08:20.1164470Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-12-18T00:08:20.1164806Z SHARD_NUMBER=1 2024-12-18T00:08:20.1165088Z GITHUB_REF_PROTECTED=true 2024-12-18T00:08:20.1165393Z HOME=/var/lib/jenkins 2024-12-18T00:08:20.1165726Z GITHUB_API_URL=https://api.github.com 2024-12-18T00:08:20.1166119Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-12-18T00:08:20.1166534Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-12-18T00:08:20.1167060Z SCCACHE_S3_KEY_PREFIX=pull 2024-12-18T00:08:20.1167373Z CUDA_VERSION=11.8.0 2024-12-18T00:08:20.1167704Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-12-18T00:08:20.1168099Z NUM_TEST_SHARDS=3 2024-12-18T00:08:20.1168377Z UCX_HOME=/usr 2024-12-18T00:08:20.1168781Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-12-18T00:08:20.1169636Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_61c5da34-5695-4c33-8a53-0e5394a1a3c0 2024-12-18T00:08:20.1170639Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:08:20.1171622Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_61c5da34-5695-4c33-8a53-0e5394a1a3c0 2024-12-18T00:08:20.1172544Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-12-18T00:08:20.1173129Z GITHUB_EVENT_NAME=push 2024-12-18T00:08:20.1173434Z DASHBOARD_TAG= 2024-12-18T00:08:20.1173703Z GITHUB_RUN_ID=12383255652 2024-12-18T00:08:20.1174077Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-12-18T00:08:20.1174517Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-12-18T00:08:20.1175311Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_61c5da34-5695-4c33-8a53-0e5394a1a3c0 2024-12-18T00:08:20.1176087Z GITHUB_ACTOR=malfet 2024-12-18T00:08:20.1176375Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-12-18T00:08:20.1176716Z PR_NUMBER= 2024-12-18T00:08:20.1176985Z GITHUB_RUN_ATTEMPT=1 2024-12-18T00:08:20.1177295Z ANACONDA_PYTHON_VERSION=3.10 2024-12-18T00:08:20.1177694Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-12-18T00:08:20.1178093Z TERM=vt100 2024-12-18T00:08:20.1178471Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-12-18T00:08:20.1179024Z INSTALLED_VISION=yes 2024-12-18T00:08:20.1179334Z BRANCH=release/2.6 2024-12-18T00:08:20.1179640Z SCCACHE_REGION=us-east-1 2024-12-18T00:08:20.1179963Z OPENSSL_ROOT_DIR=/opt/openssl 2024-12-18T00:08:20.1180342Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-12-18T00:08:20.1180832Z CUDA_PATH=/usr/local/cuda 2024-12-18T00:08:20.1181465Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-12-18T00:08:20.1182166Z GITHUB_SERVER_URL=https://github.com 2024-12-18T00:08:20.1182593Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-12-18T00:08:20.1183019Z REENABLED_ISSUES= 2024-12-18T00:08:20.1183305Z SHLVL=1 2024-12-18T00:08:20.1183559Z MAX_JOBS=46 2024-12-18T00:08:20.1183837Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-12-18T00:08:20.1184156Z NVARCH=x86_64 2024-12-18T00:08:20.1184435Z GITHUB_ACTOR_ID=2453524 2024-12-18T00:08:20.1184848Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:20.1185326Z GITHUB_REF_NAME=release/2.6 2024-12-18T00:08:20.1185691Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-12-18T00:08:20.1186216Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-12-18T00:08:20.1186742Z GITHUB_JOB=test 2024-12-18T00:08:20.1187066Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-12-18T00:08:20.1187565Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-12-18T00:08:20.1188032Z NO_TEST_TIMEOUT=False 2024-12-18T00:08:20.1188328Z TD_DISTRIBUTED=False 2024-12-18T00:08:20.1188661Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-12-18T00:08:20.1189058Z GITHUB_REPOSITORY=pytorch/pytorch 2024-12-18T00:08:20.1189424Z NV_NVPROF_VERSION=11.8.87-1 2024-12-18T00:08:20.1189760Z GITHUB_RETENTION_DAYS=90 2024-12-18T00:08:20.1190071Z OPENSSL_DIR=/opt/openssl 2024-12-18T00:08:20.1190399Z GITHUB_ACTION_REPOSITORY= 2024-12-18T00:08:20.1191413Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:08:20.1192337Z GITHUB_BASE_REF= 2024-12-18T00:08:20.1192858Z ARTIFACTS_FILE_SUFFIX=test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_34566462667 2024-12-18T00:08:20.1193526Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-12-18T00:08:20.1193857Z CI=true 2024-12-18T00:08:20.1194133Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-12-18T00:08:20.1194505Z GITHUB_REPOSITORY_OWNER=pytorch 2024-12-18T00:08:20.1194838Z JOB_ID=34566462667 2024-12-18T00:08:20.1195114Z INSTALLED_PROTOBUF=yes 2024-12-18T00:08:20.1195415Z GITHUB_HEAD_REF= 2024-12-18T00:08:20.1195693Z GITHUB_ACTION_REF= 2024-12-18T00:08:20.1196039Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-12-18T00:08:20.1196457Z TEST_SHOWLOCALS=False 2024-12-18T00:08:20.1196746Z GITHUB_WORKFLOW=pull 2024-12-18T00:08:20.1197060Z DEBIAN_FRONTEND=noninteractive 2024-12-18T00:08:20.1197781Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_61c5da34-5695-4c33-8a53-0e5394a1a3c0 2024-12-18T00:08:20.1198516Z NO_TD=False 2024-12-18T00:08:20.1198797Z SKIP_SCCACHE_INITIALIZATION=1 2024-12-18T00:08:20.1199117Z _=/usr/bin/env 2024-12-18T00:08:20.1199497Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-12-18T00:08:20.1384682Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-12-18T00:08:20.1385444Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T00:08:20.1386244Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-12-18T00:08:20.1387091Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-12-18T00:08:20.1387627Z + BUILD_DIR=build 2024-12-18T00:08:20.1387937Z + BUILD_RENAMED_DIR=build_renamed 2024-12-18T00:08:20.1388297Z + BUILD_BIN_DIR=build/bin 2024-12-18T00:08:20.1388614Z + SHARD_NUMBER=1 2024-12-18T00:08:20.1388900Z + NUM_TEST_SHARDS=3 2024-12-18T00:08:20.1389179Z + export VALGRIND=ON 2024-12-18T00:08:20.1389477Z + VALGRIND=ON 2024-12-18T00:08:20.1389823Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-12-18T00:08:20.1390298Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-12-18T00:08:20.1390703Z + [[ 0 == \1 ]] 2024-12-18T00:08:20.1391078Z + [[ False == \1 ]] 2024-12-18T00:08:20.1391655Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-12-18T00:08:20.1392107Z ++ realpath build/custom_test_artifacts 2024-12-18T00:08:20.1400836Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-12-18T00:08:20.1401426Z + [[ -n '' ]] 2024-12-18T00:08:20.1401715Z + echo 'Environment variables' 2024-12-18T00:08:20.1402044Z Environment variables 2024-12-18T00:08:20.1402333Z + env 2024-12-18T00:08:20.1409448Z INSTALLED_DB=yes 2024-12-18T00:08:20.1409945Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-12-18T00:08:20.1410506Z NVIDIA_VISIBLE_DEVICES=all 2024-12-18T00:08:20.1410834Z NV_NVML_DEV_VERSION=11.8.86-1 2024-12-18T00:08:20.1411408Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T00:08:20.1411954Z CONTINUE_THROUGH_ERROR=False 2024-12-18T00:08:20.1412345Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-12-18T00:08:20.1412781Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-12-18T00:08:20.1413225Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-12-18T00:08:20.1413638Z HOSTNAME=04018ea456d8 2024-12-18T00:08:20.1414304Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_61c5da34-5695-4c33-8a53-0e5394a1a3c0 2024-12-18T00:08:20.1415030Z GITHUB_ACTION=__self 2024-12-18T00:08:20.1415350Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-12-18T00:08:20.1417191Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-12-18T00:08:20.1419376Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-12-18T00:08:20.1419822Z NV_NVTX_VERSION=11.8.86-1 2024-12-18T00:08:20.1420134Z GITHUB_RUN_NUMBER=276594 2024-12-18T00:08:20.1420600Z TEST_CONFIG=distributed 2024-12-18T00:08:20.1420959Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-12-18T00:08:20.1421359Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-12-18T00:08:20.1421725Z IS_A100_RUNNER=0 2024-12-18T00:08:20.1422030Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-12-18T00:08:20.1422413Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-12-18T00:08:20.1423001Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2024-12-18T00:08:20.1423364Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-12-18T00:08:20.1423726Z GITHUB_TRIGGERING_ACTOR=malfet 2024-12-18T00:08:20.1424119Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-12-18T00:08:20.1424555Z GITHUB_REF_TYPE=branch 2024-12-18T00:08:20.1424879Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-12-18T00:08:20.1425218Z NCCL_VERSION=2.16.2-1 2024-12-18T00:08:20.1425580Z BASE_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:20.1425975Z XLA_CUDA= 2024-12-18T00:08:20.1426372Z HUGGING_FACE_HUB_TOKEN=*** 2024-12-18T00:08:20.1426936Z *** 2024-12-18T00:08:20.1427210Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-12-18T00:08:20.1427587Z GITHUB_REPOSITORY_ID=65600975 2024-12-18T00:08:20.1427922Z GITHUB_ACTIONS=true 2024-12-18T00:08:20.1428236Z NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:08:20.1428639Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-12-18T00:08:20.1429079Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-12-18T00:08:20.1429503Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:20.1429911Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-12-18T00:08:20.1430341Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:20.1430981Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/heads/release/2.6 2024-12-18T00:08:20.1431755Z UCC_HOME=/usr 2024-12-18T00:08:20.1432055Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-12-18T00:08:20.1432426Z VERBOSE_TEST_LOGS=False 2024-12-18T00:08:20.1432734Z NVIDIA_PRODUCT_NAME=CUDA 2024-12-18T00:08:20.1433112Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-12-18T00:08:20.1433549Z GITHUB_REF=refs/heads/release/2.6 2024-12-18T00:08:20.1433922Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-12-18T00:08:20.1434408Z SHARD_NUMBER=1 2024-12-18T00:08:20.1434702Z GITHUB_REF_PROTECTED=true 2024-12-18T00:08:20.1435026Z HOME=/var/lib/jenkins 2024-12-18T00:08:20.1435367Z GITHUB_API_URL=https://api.github.com 2024-12-18T00:08:20.1435771Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-12-18T00:08:20.1436180Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-12-18T00:08:20.1436610Z SCCACHE_S3_KEY_PREFIX=pull 2024-12-18T00:08:20.1436941Z CUDA_VERSION=11.8.0 2024-12-18T00:08:20.1437276Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-12-18T00:08:20.1437680Z NUM_TEST_SHARDS=3 2024-12-18T00:08:20.1437951Z UCX_HOME=/usr 2024-12-18T00:08:20.1438364Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-12-18T00:08:20.1439257Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_61c5da34-5695-4c33-8a53-0e5394a1a3c0 2024-12-18T00:08:20.1440293Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:08:20.1441315Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_61c5da34-5695-4c33-8a53-0e5394a1a3c0 2024-12-18T00:08:20.1442273Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-12-18T00:08:20.1442863Z GITHUB_EVENT_NAME=push 2024-12-18T00:08:20.1443174Z DASHBOARD_TAG= 2024-12-18T00:08:20.1443576Z GITHUB_RUN_ID=12383255652 2024-12-18T00:08:20.1443940Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-12-18T00:08:20.1444373Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-12-18T00:08:20.1445152Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_61c5da34-5695-4c33-8a53-0e5394a1a3c0 2024-12-18T00:08:20.1445922Z GITHUB_ACTOR=malfet 2024-12-18T00:08:20.1446222Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-12-18T00:08:20.1446560Z PR_NUMBER= 2024-12-18T00:08:20.1446823Z GITHUB_RUN_ATTEMPT=1 2024-12-18T00:08:20.1447100Z VALGRIND=ON 2024-12-18T00:08:20.1447458Z ANACONDA_PYTHON_VERSION=3.10 2024-12-18T00:08:20.1447864Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-12-18T00:08:20.1448278Z TERM=vt100 2024-12-18T00:08:20.1448564Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-12-18T00:08:20.1448915Z INSTALLED_VISION=yes 2024-12-18T00:08:20.1449213Z BRANCH=release/2.6 2024-12-18T00:08:20.1449505Z SCCACHE_REGION=us-east-1 2024-12-18T00:08:20.1449826Z OPENSSL_ROOT_DIR=/opt/openssl 2024-12-18T00:08:20.1450180Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-12-18T00:08:20.1450544Z CUDA_PATH=/usr/local/cuda 2024-12-18T00:08:20.1451145Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-12-18T00:08:20.1451816Z GITHUB_SERVER_URL=https://github.com 2024-12-18T00:08:20.1452238Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-12-18T00:08:20.1452646Z REENABLED_ISSUES= 2024-12-18T00:08:20.1452910Z SHLVL=1 2024-12-18T00:08:20.1453149Z MAX_JOBS=46 2024-12-18T00:08:20.1453418Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-12-18T00:08:20.1453741Z NVARCH=x86_64 2024-12-18T00:08:20.1454018Z GITHUB_ACTOR_ID=2453524 2024-12-18T00:08:20.1454406Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:20.1454874Z GITHUB_REF_NAME=release/2.6 2024-12-18T00:08:20.1455224Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-12-18T00:08:20.1455747Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-12-18T00:08:20.1456246Z GITHUB_JOB=test 2024-12-18T00:08:20.1456543Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-12-18T00:08:20.1457023Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-12-18T00:08:20.1457472Z NO_TEST_TIMEOUT=False 2024-12-18T00:08:20.1457769Z TD_DISTRIBUTED=False 2024-12-18T00:08:20.1458087Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-12-18T00:08:20.1458552Z GITHUB_REPOSITORY=pytorch/pytorch 2024-12-18T00:08:20.1459091Z NV_NVPROF_VERSION=11.8.87-1 2024-12-18T00:08:20.1459435Z GITHUB_RETENTION_DAYS=90 2024-12-18T00:08:20.1459766Z OPENSSL_DIR=/opt/openssl 2024-12-18T00:08:20.1460089Z GITHUB_ACTION_REPOSITORY= 2024-12-18T00:08:20.1461107Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:08:20.1462063Z GITHUB_BASE_REF= 2024-12-18T00:08:20.1462600Z ARTIFACTS_FILE_SUFFIX=test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_34566462667 2024-12-18T00:08:20.1463234Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-12-18T00:08:20.1463582Z CI=true 2024-12-18T00:08:20.1463846Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-12-18T00:08:20.1464230Z GITHUB_REPOSITORY_OWNER=pytorch 2024-12-18T00:08:20.1464578Z JOB_ID=34566462667 2024-12-18T00:08:20.1464880Z INSTALLED_PROTOBUF=yes 2024-12-18T00:08:20.1465192Z GITHUB_HEAD_REF= 2024-12-18T00:08:20.1465468Z GITHUB_ACTION_REF= 2024-12-18T00:08:20.1465831Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-12-18T00:08:20.1466259Z TEST_SHOWLOCALS=False 2024-12-18T00:08:20.1466586Z GITHUB_WORKFLOW=pull 2024-12-18T00:08:20.1466913Z DEBIAN_FRONTEND=noninteractive 2024-12-18T00:08:20.1467645Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_61c5da34-5695-4c33-8a53-0e5394a1a3c0 2024-12-18T00:08:20.1468403Z NO_TD=False 2024-12-18T00:08:20.1468690Z SKIP_SCCACHE_INITIALIZATION=1 2024-12-18T00:08:20.1469030Z _=/usr/bin/env 2024-12-18T00:08:20.1469314Z + echo 'Testing pytorch' 2024-12-18T00:08:20.1469621Z Testing pytorch 2024-12-18T00:08:20.1469917Z + export LANG=C.UTF-8 2024-12-18T00:08:20.1470213Z + LANG=C.UTF-8 2024-12-18T00:08:20.1470483Z + PR_NUMBER= 2024-12-18T00:08:20.1470884Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-12-18T00:08:20.1471254Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-12-18T00:08:20.1471682Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-12-18T00:08:20.1472093Z + [[ distributed == \s\l\o\w ]] 2024-12-18T00:08:20.1472515Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-12-18T00:08:20.1473006Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-12-18T00:08:20.1473508Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-12-18T00:08:20.1473920Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-12-18T00:08:20.1474305Z + [[ distributed == *crossref* ]] 2024-12-18T00:08:20.1474694Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-12-18T00:08:20.1475137Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-12-18T00:08:20.1475583Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-12-18T00:08:20.1476014Z + pip_install --user ninja==1.10.2 2024-12-18T00:08:20.1476471Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2024-12-18T00:08:20.1477039Z + python3 -m pip install --progress-bar off --user ninja==1.10.2 2024-12-18T00:08:22.0902848Z Collecting ninja==1.10.2 2024-12-18T00:08:22.1123855Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-12-18T00:08:22.1242843Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-12-18T00:08:22.4828490Z Installing collected packages: ninja 2024-12-18T00:08:22.4907736Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-12-18T00:08:22.4908902Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-12-18T00:08:22.4965977Z Successfully installed ninja-1.10.2 2024-12-18T00:08:22.5636272Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:08:22.5638098Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:08:22.5639250Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-12-18T00:08:22.5639671Z + install_tlparse 2024-12-18T00:08:22.5640012Z + pip_install --user tlparse==0.3.25 2024-12-18T00:08:22.5640737Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2024-12-18T00:08:22.5641337Z + python3 -m pip install --progress-bar off --user tlparse==0.3.25 2024-12-18T00:08:22.9581782Z Collecting tlparse==0.3.25 2024-12-18T00:08:22.9885876Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-12-18T00:08:23.0030806Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-12-18T00:08:23.3819871Z Installing collected packages: tlparse 2024-12-18T00:08:23.4192309Z Successfully installed tlparse-0.3.25 2024-12-18T00:08:23.4867090Z ++ python -m site --user-base 2024-12-18T00:08:23.5120059Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:08:23.5121339Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-12-18T00:08:23.5121838Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-12-18T00:08:23.5122532Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-12-18T00:08:23.5123193Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-12-18T00:08:23.5123990Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-12-18T00:08:23.5124554Z + cd test 2024-12-18T00:08:23.5124950Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-12-18T00:08:25.2374732Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-12-18T00:08:25.2375239Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-12-18T00:08:25.2377252Z + DYNAMO_BENCHMARK_FLAGS=() 2024-12-18T00:08:25.2379092Z + [[ distributed == *pr_time_benchmarks* ]] 2024-12-18T00:08:25.2379637Z + [[ distributed == *dynamo_eager* ]] 2024-12-18T00:08:25.2380026Z + [[ distributed == *aot_eager* ]] 2024-12-18T00:08:25.2380705Z + [[ distributed == *aot_inductor* ]] 2024-12-18T00:08:25.2381112Z + [[ distributed == *inductor* ]] 2024-12-18T00:08:25.2381500Z + [[ distributed == *dynamic* ]] 2024-12-18T00:08:25.2381842Z + [[ distributed == *cpu* ]] 2024-12-18T00:08:25.2382234Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-12-18T00:08:25.2418157Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-12-18T00:08:25.2419025Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-12-18T00:08:25.2419531Z + cd test 2024-12-18T00:08:25.2419887Z + python -c 'import torch; print(torch.__config__.show())' 2024-12-18T00:08:27.2425196Z PyTorch built with: 2024-12-18T00:08:27.2425563Z - GCC 9.4 2024-12-18T00:08:27.2425844Z - C++ Version: 201703 2024-12-18T00:08:27.2426512Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-12-18T00:08:27.2427386Z - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-12-18T00:08:27.2427961Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-12-18T00:08:27.2428395Z - LAPACK is enabled (usually provided by MKL) 2024-12-18T00:08:27.2428806Z - NNPACK is enabled 2024-12-18T00:08:27.2429121Z - CPU capability usage: AVX512 2024-12-18T00:08:27.2429476Z - CUDA Runtime 11.8 2024-12-18T00:08:27.2429901Z - NVCC architecture flags: -gencode;arch=compute_75,code=sm_75 2024-12-18T00:08:27.2430377Z - CuDNN 90.1 2024-12-18T00:08:27.2430772Z - Magma 2.6.1 2024-12-18T00:08:27.2437091Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-12-18T00:08:27.2443043Z 2024-12-18T00:08:27.6800708Z + cd test 2024-12-18T00:08:27.6801210Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-12-18T00:08:29.2201679Z ATen/Parallel: 2024-12-18T00:08:29.2202083Z at::get_num_threads() : 24 2024-12-18T00:08:29.2202517Z at::get_num_interop_threads() : 24 2024-12-18T00:08:29.2202911Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-12-18T00:08:29.2203286Z omp_get_max_threads() : 24 2024-12-18T00:08:29.2203935Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-12-18T00:08:29.2204637Z mkl_get_max_threads() : 24 2024-12-18T00:08:29.2205099Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-12-18T00:08:29.2205631Z std::thread::hardware_concurrency() : 48 2024-12-18T00:08:29.2206018Z Environment variables: 2024-12-18T00:08:29.2206322Z OMP_NUM_THREADS : [not set] 2024-12-18T00:08:29.2206657Z MKL_NUM_THREADS : [not set] 2024-12-18T00:08:29.2206997Z ATen parallel backend: OpenMP 2024-12-18T00:08:29.2207218Z 2024-12-18T00:08:29.4588071Z + [[ distributed == *numpy_2* ]] 2024-12-18T00:08:29.4588584Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-12-18T00:08:29.4589047Z + [[ distributed == *backward* ]] 2024-12-18T00:08:29.4589849Z + [[ distributed == *xla* ]] 2024-12-18T00:08:29.4590211Z + [[ distributed == *executorch* ]] 2024-12-18T00:08:29.4590582Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-12-18T00:08:29.4591138Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-12-18T00:08:29.4591567Z + [[ distributed == distributed ]] 2024-12-18T00:08:29.4591910Z + test_distributed 2024-12-18T00:08:29.4592233Z + echo 'Testing distributed python tests' 2024-12-18T00:08:29.4592614Z Testing distributed python tests 2024-12-18T00:08:29.4593099Z + python test/run_test.py --distributed-tests --shard 1 3 --verbose 2024-12-18T00:08:29.5655949Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:08:29.5656961Z import pkg_resources 2024-12-18T00:08:34.6292663Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:08:34.7037801Z Ignoring disabled issues: [''] 2024-12-18T00:08:34.7205195Z Found test times from artifacts 2024-12-18T00:08:34.7852106Z Found test times from artifacts 2024-12-18T00:08:34.7868926Z Running all tests 2024-12-18T00:08:34.7987947Z Running parallel tests on 2 processes 2024-12-18T00:08:34.7991223Z Name: tests to run (est. time: 110.28min) 2024-12-18T00:08:34.7991611Z Serial tests (68): 2024-12-18T00:08:34.7991976Z distributed/test_dynamo_distributed 1/1 2024-12-18T00:08:34.7992398Z distributed/_tensor/test_dtensor_ops 1/1 2024-12-18T00:08:34.7992813Z distributed/_tensor/test_random_ops 1/1 2024-12-18T00:08:34.7993199Z distributed/launcher/test_run 1/1 2024-12-18T00:08:34.7993633Z distributed/_shard/sharded_tensor/ops/test_init 1/1 2024-12-18T00:08:34.7994187Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 2024-12-18T00:08:34.7994706Z distributed/_tensor/test_view_ops 1/1 2024-12-18T00:08:34.7995427Z distributed/rpc/cuda/test_tensorpipe_agent 1/2 2024-12-18T00:08:34.7995950Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 2024-12-18T00:08:34.7996505Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 2024-12-18T00:08:34.7996992Z distributed/test_c10d_ops_nccl 1/1 2024-12-18T00:08:34.7997415Z distributed/_tensor/debug/test_op_coverage 1/1 2024-12-18T00:08:34.7997866Z distributed/test_compute_comm_reordering 1/1 2024-12-18T00:08:34.7998293Z distributed/_tensor/test_dtensor 1/1 2024-12-18T00:08:34.7998685Z distributed/pipelining/test_unflatten 1/1 2024-12-18T00:08:34.7999128Z distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 2024-12-18T00:08:34.7999601Z distributed/_tensor/experimental/test_local_map 1/1 2024-12-18T00:08:34.8000052Z distributed/test_distributed_spawn 1/9 2024-12-18T00:08:34.8000457Z distributed/test_distributed_spawn 4/9 2024-12-18T00:08:34.8000846Z distributed/test_distributed_spawn 7/9 2024-12-18T00:08:34.8001266Z distributed/_tensor/test_redistribute 1/1 2024-12-18T00:08:34.8001701Z distributed/_tools/test_fsdp2_mem_tracker 1/1 2024-12-18T00:08:34.8002143Z distributed/checkpoint/test_format_utils 1/1 2024-12-18T00:08:34.8002620Z distributed/_tensor/debug/test_comm_mode_features 1/1 2024-12-18T00:08:34.8003097Z distributed/checkpoint/e2e/test_fine_tuning 1/1 2024-12-18T00:08:34.8003600Z distributed/_tensor/experimental/test_tp_transform 1/1 2024-12-18T00:08:34.8004366Z distributed/checkpoint/e2e/test_pipeline 1/1 2024-12-18T00:08:34.8004805Z distributed/checkpoint/test_traverse 1/1 2024-12-18T00:08:34.8005271Z distributed/tensor/parallel/test_tp_random_state 1/1 2024-12-18T00:08:34.8005889Z distributed/checkpoint/test_dtensor_checkpoint 1/1 2024-12-18T00:08:34.8006381Z distributed/_composable/test_replicate_with_compiler 1/1 2024-12-18T00:08:34.8006844Z distributed/test_collective_utils 1/1 2024-12-18T00:08:34.8007223Z distributed/test_c10d_logger 1/1 2024-12-18T00:08:34.8007727Z distributed/fsdp/test_fsdp_state_dict 1/2 2024-12-18T00:08:34.8008111Z distributed/test_c10d_nccl 2/3 2024-12-18T00:08:34.8008470Z distributed/pipelining/test_schedule 1/1 2024-12-18T00:08:34.8008871Z distributed/_tensor/test_op_strategy 1/1 2024-12-18T00:08:34.8009299Z distributed/checkpoint/test_fsdp_model_state 1/1 2024-12-18T00:08:34.8009724Z distributed/rpc/test_share_memory 1/1 2024-12-18T00:08:34.8010146Z distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 2024-12-18T00:08:34.8010578Z distributed/checkpoint/test_checkpoint 1/1 2024-12-18T00:08:34.8010983Z distributed/fsdp/test_fsdp_memory 1/1 2024-12-18T00:08:34.8011608Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 2024-12-18T00:08:34.8012133Z distributed/tensor/parallel/test_parallelize_api 1/1 2024-12-18T00:08:34.8012596Z distributed/fsdp/test_fsdp_traversal 1/1 2024-12-18T00:08:34.8013000Z distributed/checkpoint/test_state_dict 1/1 2024-12-18T00:08:34.8013421Z distributed/test_multi_threaded_pg 1/1 2024-12-18T00:08:34.8013825Z distributed/fsdp/test_fsdp_uneven 1/1 2024-12-18T00:08:34.8014220Z distributed/fsdp/test_fsdp_input 1/1 2024-12-18T00:08:34.8014603Z distributed/test_c10d_ucc 1/1 2024-12-18T00:08:34.8015043Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 2024-12-18T00:08:34.8015531Z distributed/test_pg_wrapper 1/1 2024-12-18T00:08:34.8015991Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-12-18T00:08:34.8016485Z distributed/rpc/test_tensorpipe_agent 1/1 2024-12-18T00:08:34.8016965Z distributed/_shard/sharding_plan/test_sharding_plan 1/1 2024-12-18T00:08:34.8017455Z distributed/fsdp/test_fsdp_multiple_forward 1/1 2024-12-18T00:08:34.8017934Z distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 2024-12-18T00:08:34.8018456Z distributed/fsdp/test_fsdp_meta 1/1 2024-12-18T00:08:34.8019090Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 2024-12-18T00:08:34.8019747Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 2024-12-18T00:08:34.8020358Z distributed/test_launcher 1/1 2024-12-18T00:08:34.8020807Z distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 2024-12-18T00:08:34.8021305Z distributed/elastic/events/lib_test 1/1 2024-12-18T00:08:34.8021732Z distributed/elastic/metrics/api_test 1/1 2024-12-18T00:08:34.8022196Z distributed/elastic/multiprocessing/api_test 1/1 2024-12-18T00:08:34.8022682Z distributed/elastic/timer/local_timer_test 1/1 2024-12-18T00:08:34.8023125Z distributed/elastic/utils/logging_test 1/1 2024-12-18T00:08:34.8023550Z distributed/elastic/utils/util_test 1/1 2024-12-18T00:08:34.8023939Z Parallel tests (0): 2024-12-18T00:08:34.8024265Z Name: excluded (est. time: 0.0min) 2024-12-18T00:08:34.8024621Z Serial tests (0): 2024-12-18T00:08:34.8024905Z Parallel tests (0): 2024-12-18T00:08:34.8058751Z Running distributed/test_dynamo_distributed 1/1 ... [2024-12-18 00:08:34.805259] 2024-12-18T00:08:34.8059553Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:08:34.8060889Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:08:34.805597] 2024-12-18T00:13:52.6924567Z 2024-12-18T00:13:52.6926312Z distributed/test_dynamo_distributed 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_dynamo_distributed_1.1_7f2977431673f942_.log 2024-12-18T00:13:52.6971829Z Running 58 items in this shard: test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_call_method_forward, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_ddp_optimizer_inductor_strides_dont_specialize, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_issue90375, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_symbol_splitting, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_direct, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_indirect, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_no_binding, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_torture_multi, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation_with_fx_cache, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_scalar, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_speculation_divergence, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_tensor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_dim_mismatch, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_graph_break_empty_graph_still_collective, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_missing_source, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_scalar_missing_source, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_type_mismatch, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_baseline_aot_eager_multiprocess, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_setattr, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_unspecialized_forced_getattr_inline, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_unspecialized_forced_getattr_no_inline, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_get_pg_attr, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_aot_autograd, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_async_subclass_no_specialize, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_compiled_flex_attention_full_model_ddp, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_compiled_flex_attention_local_ddp, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_custom_layer, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_aot_eager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_empty_graph_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_diff_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_same_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_orig_params_assert, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_guards, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_register_attr_or_module, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_staticmethod, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_ctx_manager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_inference, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_training, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_transpose, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_higher_order_op, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ignored_parameters, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_no_split 2024-12-18T00:13:52.7014531Z 2024-12-18T00:13:52.7015010Z Running distributed/_tensor/test_dtensor_ops 1/1 ... [2024-12-18 00:13:52.692268] 2024-12-18T00:13:52.7015836Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:13:52.7017689Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:13:52.692635] 2024-12-18T00:13:57.7588527Z 2024-12-18T00:13:57.7589668Z distributed/_tensor/test_dtensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_ops_1.1_33d90eef123d3c83_.log 2024-12-18T00:13:57.7590759Z Running 0 items in this shard: 2024-12-18T00:13:57.7591027Z 2024-12-18T00:13:57.7593228Z Running distributed/_tensor/test_random_ops 1/1 ... [2024-12-18 00:13:57.759150] 2024-12-18T00:13:57.7593806Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:13:57.7596818Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_random_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:13:57.759497] 2024-12-18T00:15:18.2290046Z 2024-12-18T00:15:18.2291252Z distributed/_tensor/test_random_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_random_ops_1.1_b72a2e9902170a0f_.log 2024-12-18T00:15:18.2297110Z Running 11 items in this shard: test/distributed/_tensor/test_random_ops.py::DistTensorRandomInitTest::test_init_ops, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_dropout_1d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_rand_1d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_uniform_2d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_fsdp_tp_model_meta_init, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_manual_seed, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_manual_seed_submesh, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_meta_tensor_init, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_pipeline_parallel_manual_seed, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_rng_tracker_init, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_tp_model_meta_init 2024-12-18T00:15:18.2302773Z 2024-12-18T00:15:18.2303067Z Running distributed/launcher/test_run 1/1 ... [2024-12-18 00:15:18.228967] 2024-12-18T00:15:18.2303982Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:15:18.2305455Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/launcher/test_run.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:15:18.229329] 2024-12-18T00:16:04.4754405Z 2024-12-18T00:16:04.4755535Z distributed/launcher/test_run 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.launcher.test_run_1.1_5dda03571d7a3f14_.log 2024-12-18T00:16:04.4767699Z Running 24 items in this shard: test/distributed/launcher/test_run.py::ElasticLaunchTest::test_capture_logs_using_default_logs_specs, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_init_method_env_with_torchelastic, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_init_method_tcp_with_torchelastic, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_is_not_torchelastic_launched, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_is_torchelastic_launched, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_is_torchelastic_launched_with_logs_spec_defined, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_elastic, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_elastic_agent_raise_exception, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_elastic_multiple_agents, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_elastic_worker_raise_exception, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_run_path, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_shutdown, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_standalone, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_user_script_bash, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_user_script_default_nproc, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_user_script_python, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_user_script_python_caffe2_bc, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_with_env_vars, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_logs_logs_spec_entrypoint_must_be_defined, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_min_max_nodes_parse, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_nproc_gpu_launch_configurations, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_nproc_launch_auto_configurations, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_nproc_launch_number_configurations, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_nproc_launch_unknown_configurations 2024-12-18T00:16:04.4778972Z 2024-12-18T00:16:04.4779505Z Running distributed/_shard/sharded_tensor/ops/test_init 1/1 ... [2024-12-18 00:16:04.475738] 2024-12-18T00:16:04.4780149Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:16:04.4781489Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:16:04.476103] 2024-12-18T00:16:40.1270781Z 2024-12-18T00:16:40.1272089Z distributed/_shard/sharded_tensor/ops/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_init_1.1_aab645d3723ef400_.log 2024-12-18T00:16:40.1274920Z Running 3 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_kaiming_uniform, test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_normal, test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_uniform 2024-12-18T00:16:40.1277293Z 2024-12-18T00:16:40.1277736Z Running distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 ... [2024-12-18 00:16:40.127280] 2024-12-18T00:16:40.1278402Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:16:40.1279772Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:16:40.127644] 2024-12-18T00:17:03.2013371Z 2024-12-18T00:17:03.2018683Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_clip_grad_norm__1.1_b4c392db59610145_.log 2024-12-18T00:17:03.2022437Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py::TestClipGradNormWorldSize2::test_clip_grad_norm_1d, test/distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py::TestClipGradNormWorldSize4::test_clip_grad_norm_2d 2024-12-18T00:17:03.2023820Z 2024-12-18T00:17:03.2024126Z Running distributed/_tensor/test_view_ops 1/1 ... [2024-12-18 00:17:03.201331] 2024-12-18T00:17:03.2024697Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:17:03.2025983Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_view_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:17:03.201700] 2024-12-18T00:17:39.6474095Z 2024-12-18T00:17:39.6475658Z distributed/_tensor/test_view_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_view_ops_1.1_50492ad2f4dedc26_.log 2024-12-18T00:17:39.6479078Z Running 4 items in this shard: test/distributed/_tensor/test_view_ops.py::TestViewOps::test_complex_view_ops, test/distributed/_tensor/test_view_ops.py::TestViewOps::test_dtensor_view_op_uneven, test/distributed/_tensor/test_view_ops.py::TestViewOps::test_view_groups, test/distributed/_tensor/test_view_ops.py::TestViewOps::test_view_ops 2024-12-18T00:17:39.6481347Z 2024-12-18T00:17:39.6481704Z Running distributed/rpc/cuda/test_tensorpipe_agent 1/2 ... [2024-12-18 00:17:39.647503] 2024-12-18T00:17:39.6482308Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:17:39.6483673Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '--shard-id=1', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:17:39.647854] 2024-12-18T00:27:28.8598326Z 2024-12-18T00:27:28.8599609Z distributed/rpc/cuda/test_tensorpipe_agent 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.cuda.test_tensorpipe_agent_1.2_629243349af919f6_.log 2024-12-18T00:27:28.8630346Z Running 47 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRpcTest::test_profiler_remote_cuda, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_simple, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_to_cpu_continuation, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_to_cpu_continuation_gpu_root, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_invalid_devices, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_valid_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDdpComparisonTest::test_ddp_dist_autograd_local_vs_remote_gpu, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_async_execution_with_cuda_future, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_cuda_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_list_with_cuda_sparse_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_list_with_cuda_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_not_cuda, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_nested, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_default_to_non_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_5, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_6, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_5, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_6, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_8, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_non_default_to_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_to_cpu_non_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_in_options, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_min_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_remote, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_response_loop, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_one_to_many, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_return_to_gpu, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_return_to_gpu_self, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_wrong_worker_name, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_mismatch, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_devices_option_mismatch_reverse, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_dist_autograd_sync_streams, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_gradients_synchronizations 2024-12-18T00:27:28.8660802Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRpcTest::test_profiler_remote_cuda 2024-12-18T00:27:28.8662076Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_simple 2024-12-18T00:27:28.8663424Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_to_cpu_continuation 2024-12-18T00:27:28.8664862Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_to_cpu_continuation_gpu_root 2024-12-18T00:27:28.8666267Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_invalid_devices 2024-12-18T00:27:28.8667553Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_valid_device 2024-12-18T00:27:28.8668962Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDdpComparisonTest::test_ddp_dist_autograd_local_vs_remote_gpu 2024-12-18T00:27:28.8670652Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_async_execution_with_cuda_future 2024-12-18T00:27:28.8672352Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_cuda_tensor 2024-12-18T00:27:28.8673980Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_list_with_cuda_sparse_tensor 2024-12-18T00:27:28.8675650Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_list_with_cuda_tensor 2024-12-18T00:27:28.8677206Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_not_cuda 2024-12-18T00:27:28.8678634Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_nested 2024-12-18T00:27:28.8680068Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu 2024-12-18T00:27:28.8681447Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_default 2024-12-18T00:27:28.8683057Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_default_to_non_default 2024-12-18T00:27:28.8684505Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_1 2024-12-18T00:27:28.8685888Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_2 2024-12-18T00:27:28.8687269Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_3 2024-12-18T00:27:28.8690544Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_4 2024-12-18T00:27:28.8691978Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_5 2024-12-18T00:27:28.8693380Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_6 2024-12-18T00:27:28.8694829Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_2 2024-12-18T00:27:28.8696310Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_3 2024-12-18T00:27:28.8697797Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_4 2024-12-18T00:27:28.8699525Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_5 2024-12-18T00:27:28.8701048Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_6 2024-12-18T00:27:28.8702563Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_8 2024-12-18T00:27:28.8704133Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_non_default_to_default 2024-12-18T00:27:28.8705805Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_to_cpu_non_default 2024-12-18T00:27:28.8707330Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_in_options 2024-12-18T00:27:28.8708843Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_min_device 2024-12-18T00:27:28.8710407Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_remote 2024-12-18T00:27:28.8712046Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_response_loop 2024-12-18T00:27:28.8713437Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_one_to_many 2024-12-18T00:27:28.8714768Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_return_to_gpu 2024-12-18T00:27:28.8716128Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_return_to_gpu_self 2024-12-18T00:27:28.8717501Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_wrong_worker_name 2024-12-18T00:27:28.8718791Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_mismatch 2024-12-18T00:27:28.8720313Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_devices_option_mismatch_reverse 2024-12-18T00:27:28.8721812Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization1 2024-12-18T00:27:28.8723385Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization3 2024-12-18T00:27:28.8724866Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization1 2024-12-18T00:27:28.8726321Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization2 2024-12-18T00:27:28.8727776Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization3 2024-12-18T00:27:28.8729236Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_dist_autograd_sync_streams 2024-12-18T00:27:28.8730715Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_gradients_synchronizations 2024-12-18T00:27:28.8731669Z 2024-12-18T00:27:28.8732220Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 ... [2024-12-18 00:27:28.860942] 2024-12-18T00:27:28.8732893Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:27:28.8734289Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:27:28.861299] 2024-12-18T00:28:07.2252570Z 2024-12-18T00:28:07.2256317Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_tensor_ops_1.1_6c6a467ca9118a06_.log 2024-12-18T00:28:07.2260861Z Running 5 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_clone, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_deep_copy, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_detach, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_inplace_copy, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_set_requires_grad 2024-12-18T00:28:07.2263389Z 2024-12-18T00:28:07.2263784Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 ... [2024-12-18 00:28:07.225310] 2024-12-18T00:28:07.2264455Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:28:07.2265810Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:28:07.225691] 2024-12-18T00:28:25.9347945Z 2024-12-18T00:28:25.9349351Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_embedding_bag_1.1_a416816ab23249c0_.log 2024-12-18T00:28:25.9351929Z Running 2 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_embedding_bag.py::TestShardedEmbeddingBag::test_sharded_embedding_bag_colwise, test/distributed/_shard/sharded_tensor/ops/test_embedding_bag.py::TestShardedEmbeddingBag::test_sharded_embedding_bag_rowwise 2024-12-18T00:28:25.9353391Z 2024-12-18T00:28:25.9353666Z Running distributed/test_c10d_ops_nccl 1/1 ... [2024-12-18 00:28:25.935219] 2024-12-18T00:28:25.9354206Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:28:25.9357578Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_ops_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:28:25.935571] 2024-12-18T00:28:41.9418164Z 2024-12-18T00:28:41.9419595Z distributed/test_c10d_ops_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_ops_nccl_1.1_6652867542679a86_.log 2024-12-18T00:28:41.9431424Z Running 25 items in this shard: test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_in_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_alltoall_ops_with_cudafree_race, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_barrier, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_broadcast_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_empty_tensors, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_nccl_watchdog_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_complex, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_object_list, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_sparse_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_tensor_register_hook 2024-12-18T00:28:41.9452559Z Running 25 items in this shard: test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_in_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_alltoall_ops_with_cudafree_race, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_barrier, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_broadcast_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_empty_tensors, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_nccl_watchdog_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_complex, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_object_list, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_sparse_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_tensor_register_hook 2024-12-18T00:28:41.9473399Z Running 25 items in this shard: test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_in_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_alltoall_ops_with_cudafree_race, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_barrier, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_broadcast_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_empty_tensors, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_nccl_watchdog_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_complex, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_object_list, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_sparse_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_tensor_register_hook 2024-12-18T00:28:41.9495533Z Running 25 items in this shard: test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_in_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_alltoall_ops_with_cudafree_race, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_barrier, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_broadcast_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_empty_tensors, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_nccl_watchdog_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_complex, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_object_list, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_sparse_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_tensor_register_hook 2024-12-18T00:28:41.9506393Z 2024-12-18T00:28:41.9506743Z Running distributed/_tensor/debug/test_op_coverage 1/1 ... [2024-12-18 00:28:41.942139] 2024-12-18T00:28:41.9507343Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:28:41.9508669Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/debug/test_op_coverage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:28:41.942506] 2024-12-18T00:28:46.6691618Z 2024-12-18T00:28:46.6692931Z distributed/_tensor/debug/test_op_coverage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_f5cf6091ee2e9fc6_.log 2024-12-18T00:28:46.6694780Z Running 1 items in this shard: test/distributed/_tensor/debug/test_op_coverage.py::TestOpCoverage::test_trace_with_inductor_decomp 2024-12-18T00:28:46.6695458Z 2024-12-18T00:28:46.6695798Z Running distributed/test_compute_comm_reordering 1/1 ... [2024-12-18 00:28:46.669264] 2024-12-18T00:28:46.6696369Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:28:46.6699027Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_compute_comm_reordering.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:28:46.669617] 2024-12-18T00:29:51.6470169Z 2024-12-18T00:29:51.6471509Z distributed/test_compute_comm_reordering 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_compute_comm_reordering_1.1_752fbf28fccb92eb_.log 2024-12-18T00:29:51.6476589Z Running 7 items in this shard: test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_grouped_scheduler_node, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_nccl_heuristics, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_raise_comms, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap_custom_runtime_estimation, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits_raise_comms 2024-12-18T00:29:51.6480785Z 2024-12-18T00:29:51.6481091Z Running distributed/_tensor/test_dtensor 1/1 ... [2024-12-18 00:29:51.647347] 2024-12-18T00:29:51.6481631Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:29:51.6482887Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:29:51.647686] 2024-12-18T00:33:51.9405291Z 2024-12-18T00:33:51.9406489Z distributed/_tensor/test_dtensor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_1.1_6fc3edc5e3eed925_.log 2024-12-18T00:33:51.9422688Z Running 36 items in this shard: test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_async_output, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_constructor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_new_empty_strided, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_properties, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_save_load, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_save_load_import, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_hash, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_read_only_after_set, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_stride, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_negative_dim, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_then_to_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding_raise_error, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_grad_hint, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_sync, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_meta_dtensor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_modules_w_meta_dtensor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_shard_tensor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_shard_tensor_2d, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local_grad_hint, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_auto_implicit_replication, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_default_value_sub_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_device_mesh_nd, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_2d_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_api_device_mesh_context_manager, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_device_mesh_device_conversion, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_spec_local_shard_offset, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_from_local_sub_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_implicit_replication, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_implicit_replication_for_foreach_ops, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_metadata_consistency_check, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_redistribute_sub_mesh, test/distributed/_tensor/test_dtensor.py::TestDTensorPlacementTypes::test_split_tensor_1D, test/distributed/_tensor/test_dtensor.py::DTensorLogTest::test_dtensor_log 2024-12-18T00:33:51.9437774Z 2024-12-18T00:33:51.9438108Z Running distributed/pipelining/test_unflatten 1/1 ... [2024-12-18 00:33:51.940870] 2024-12-18T00:33:51.9438921Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:33:51.9440176Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_unflatten.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:33:51.941246] 2024-12-18T00:33:56.6180601Z 2024-12-18T00:33:56.6181934Z distributed/pipelining/test_unflatten 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_unflatten_1.1_a8d0b05490f8261f_.log 2024-12-18T00:33:56.6183449Z Running 1 items in this shard: test/distributed/pipelining/test_unflatten.py::UnflattenTests::test_unflatten 2024-12-18T00:33:56.6184054Z 2024-12-18T00:33:56.6184402Z Running distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 ... [2024-12-18 00:33:56.618064] 2024-12-18T00:33:56.6185036Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:33:56.6186460Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/fsdp/test_fsdp_dsd.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:33:56.618434] 2024-12-18T00:34:46.7558807Z 2024-12-18T00:34:46.7560213Z distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.fsdp.test_fsdp_dsd_1.1_a2961e367e3f034f_.log 2024-12-18T00:34:46.7565270Z Running 6 items in this shard: test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_1d_fsdp_cpu_offload_full_model_state_dict, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_1d_fsdp_get_model_state_dict, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp1_and_load_with_fsdp2, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp1_and_load_with_fsdp2_tp, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp2_tp_and_load_with_tp, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_tp_and_load_with_fsdp2_tp 2024-12-18T00:34:46.7569526Z 2024-12-18T00:34:46.7570220Z Running distributed/_tensor/experimental/test_local_map 1/1 ... [2024-12-18 00:34:46.755837] 2024-12-18T00:34:46.7570947Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:34:46.7572232Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/experimental/test_local_map.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:34:46.756192] 2024-12-18T00:35:14.2905737Z 2024-12-18T00:35:14.2907846Z distributed/_tensor/experimental/test_local_map 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.experimental.test_local_map_1.1_20abb985a7e2c6e1_.log 2024-12-18T00:35:14.2910881Z Running 4 items in this shard: test/distributed/_tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_correctness, test/distributed/_tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_in_placements, test/distributed/_tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_out_placements, test/distributed/_tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_redistribute 2024-12-18T00:35:14.2912957Z 2024-12-18T00:35:14.2913445Z Running distributed/test_distributed_spawn 1/9 ... [2024-12-18 00:35:14.291030] 2024-12-18T00:35:14.2925989Z /usr/bin/mpiexec 2024-12-18T00:35:14.2926398Z MPI not available -- MPI backend tests will be skipped 2024-12-18T00:35:14.2926963Z Running distributed tests for the test backend with env init_method 2024-12-18T00:35:14.2927549Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:35:14.2932358Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:35:14.293007] 2024-12-18T00:35:18.4816997Z 2024-12-18T00:35:18.4818226Z distributed/test_distributed_spawn 1/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.9_7e394755e43cf579_.log 2024-12-18T00:35:18.4819665Z Running 0 items in this shard: 2024-12-18T00:35:18.4819916Z 2024-12-18T00:35:18.4823396Z Running distributed tests for the test backend with file init_method 2024-12-18T00:35:18.4825219Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:35:18.4829590Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:35:18.482765] 2024-12-18T00:35:22.6822384Z 2024-12-18T00:35:22.6823586Z distributed/test_distributed_spawn 1/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.9_a4a4edc02eaeae1b_.log 2024-12-18T00:35:22.6824707Z Running 0 items in this shard: 2024-12-18T00:35:22.6824935Z 2024-12-18T00:35:22.6831039Z Running distributed tests for the nccl backend with env init_method 2024-12-18T00:35:22.6833405Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:35:22.6837638Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:35:22.683571] 2024-12-18T00:41:25.6071351Z 2024-12-18T00:41:25.6072533Z distributed/test_distributed_spawn 1/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.9_9e1a74ad1a397900_.log 2024-12-18T00:41:25.6092430Z Running 35 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-12-18T00:41:25.6111841Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2024-12-18T00:41:25.6113133Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2024-12-18T00:41:25.6114478Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2024-12-18T00:41:25.6115902Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync 2024-12-18T00:41:25.6117150Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2024-12-18T00:41:25.6118394Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg 2024-12-18T00:41:25.6119616Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2024-12-18T00:41:25.6120867Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2024-12-18T00:41:25.6122190Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2024-12-18T00:41:25.6123489Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max 2024-12-18T00:41:25.6124689Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min 2024-12-18T00:41:25.6125827Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2024-12-18T00:41:25.6126941Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async 2024-12-18T00:41:25.6128108Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2024-12-18T00:41:25.6129308Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split 2024-12-18T00:41:25.6130610Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2024-12-18T00:41:25.6132424Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group 2024-12-18T00:41:25.6133824Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group 2024-12-18T00:41:25.6135161Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2024-12-18T00:41:25.6136475Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping 2024-12-18T00:41:25.6137765Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2024-12-18T00:41:25.6139031Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2024-12-18T00:41:25.6140325Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params 2024-12-18T00:41:25.6141678Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace 2024-12-18T00:41:25.6142898Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2024-12-18T00:41:25.6144053Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group 2024-12-18T00:41:25.6145148Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv 2024-12-18T00:41:25.6146323Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup 2024-12-18T00:41:25.6147646Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups 2024-12-18T00:41:25.6148942Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size 2024-12-18T00:41:25.6150357Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2024-12-18T00:41:25.6151618Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2024-12-18T00:41:25.6152671Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2024-12-18T00:41:25.6153751Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2024-12-18T00:41:25.6154955Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-12-18T00:41:25.6155664Z 2024-12-18T00:41:25.6156044Z Running distributed tests for the nccl backend with file init_method 2024-12-18T00:41:25.6156519Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:41:25.6157718Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:41:25.609758] 2024-12-18T00:47:28.7569856Z 2024-12-18T00:47:28.7571016Z distributed/test_distributed_spawn 1/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.9_1d78ff1119b015a5_.log 2024-12-18T00:47:28.7590596Z Running 35 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-12-18T00:47:28.7609167Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2024-12-18T00:47:28.7610459Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2024-12-18T00:47:28.7611804Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2024-12-18T00:47:28.7613229Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync 2024-12-18T00:47:28.7614470Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2024-12-18T00:47:28.7615711Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg 2024-12-18T00:47:28.7616961Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2024-12-18T00:47:28.7618214Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2024-12-18T00:47:28.7619797Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2024-12-18T00:47:28.7621149Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max 2024-12-18T00:47:28.7622398Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min 2024-12-18T00:47:28.7623582Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2024-12-18T00:47:28.7624862Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async 2024-12-18T00:47:28.7626128Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2024-12-18T00:47:28.7627310Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split 2024-12-18T00:47:28.7628637Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2024-12-18T00:47:28.7629956Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group 2024-12-18T00:47:28.7631269Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group 2024-12-18T00:47:28.7632906Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2024-12-18T00:47:28.7634223Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping 2024-12-18T00:47:28.7635501Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2024-12-18T00:47:28.7636711Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2024-12-18T00:47:28.7638010Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params 2024-12-18T00:47:28.7639361Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace 2024-12-18T00:47:28.7640445Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2024-12-18T00:47:28.7641469Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group 2024-12-18T00:47:28.7642446Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv 2024-12-18T00:47:28.7643492Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup 2024-12-18T00:47:28.7644865Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups 2024-12-18T00:47:28.7646091Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size 2024-12-18T00:47:28.7647420Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2024-12-18T00:47:28.7648568Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2024-12-18T00:47:28.7649622Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2024-12-18T00:47:28.7650701Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2024-12-18T00:47:28.7651910Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-12-18T00:47:28.7652617Z 2024-12-18T00:47:28.7652857Z Running distributed tests for the gloo backend with env init_method 2024-12-18T00:47:28.7653354Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:47:28.7654637Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:47:28.759149] 2024-12-18T00:54:32.1926237Z 2024-12-18T00:54:32.1929660Z distributed/test_distributed_spawn 1/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.9_6ef26a58ab32fcc3_.log 2024-12-18T00:54:32.1949450Z Running 35 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-12-18T00:54:32.1968272Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2024-12-18T00:54:32.1969508Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2024-12-18T00:54:32.1970881Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2024-12-18T00:54:32.1972203Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync 2024-12-18T00:54:32.1973410Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2024-12-18T00:54:32.1974605Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg 2024-12-18T00:54:32.1975808Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2024-12-18T00:54:32.1977036Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2024-12-18T00:54:32.1978320Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2024-12-18T00:54:32.1979850Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max 2024-12-18T00:54:32.1981089Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min 2024-12-18T00:54:32.1982264Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2024-12-18T00:54:32.1983412Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async 2024-12-18T00:54:32.1984602Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2024-12-18T00:54:32.1985856Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split 2024-12-18T00:54:32.1987265Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2024-12-18T00:54:32.1988665Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group 2024-12-18T00:54:32.1990071Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group 2024-12-18T00:54:32.1991408Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2024-12-18T00:54:32.1992756Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping 2024-12-18T00:54:32.1993969Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2024-12-18T00:54:32.1995092Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2024-12-18T00:54:32.1996309Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params 2024-12-18T00:54:32.1997574Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace 2024-12-18T00:54:32.1998825Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2024-12-18T00:54:32.1999847Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group 2024-12-18T00:54:32.2000823Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv 2024-12-18T00:54:32.2001929Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup 2024-12-18T00:54:32.2003011Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups 2024-12-18T00:54:32.2004158Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size 2024-12-18T00:54:32.2005415Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2024-12-18T00:54:32.2006505Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2024-12-18T00:54:32.2007501Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2024-12-18T00:54:32.2008734Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2024-12-18T00:54:32.2009953Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-12-18T00:54:32.2010666Z 2024-12-18T00:54:32.2010925Z Running distributed tests for the gloo backend with file init_method 2024-12-18T00:54:32.2011414Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:54:32.2012699Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:54:32.194880] 2024-12-18T01:01:35.0416202Z 2024-12-18T01:01:35.0417371Z distributed/test_distributed_spawn 1/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.9_e1ef8b9db8abbc5f_.log 2024-12-18T01:01:35.0437368Z Running 35 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-12-18T01:01:35.0456338Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2024-12-18T01:01:35.0457619Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2024-12-18T01:01:35.0459030Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2024-12-18T01:01:35.0460673Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync 2024-12-18T01:01:35.0461967Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda 2024-12-18T01:01:35.0463252Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg 2024-12-18T01:01:35.0464529Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2024-12-18T01:01:35.0465823Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2024-12-18T01:01:35.0467193Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2024-12-18T01:01:35.0468532Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max 2024-12-18T01:01:35.0469755Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min 2024-12-18T01:01:35.0471032Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max 2024-12-18T01:01:35.0472154Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async 2024-12-18T01:01:35.0473323Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2024-12-18T01:01:35.0474541Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split 2024-12-18T01:01:35.0475912Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2024-12-18T01:01:35.0477271Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group 2024-12-18T01:01:35.0478628Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group 2024-12-18T01:01:35.0479905Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2024-12-18T01:01:35.0481180Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping 2024-12-18T01:01:35.0482437Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2024-12-18T01:01:35.0483612Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu 2024-12-18T01:01:35.0484864Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params 2024-12-18T01:01:35.0486177Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace 2024-12-18T01:01:35.0487355Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2024-12-18T01:01:35.0488485Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group 2024-12-18T01:01:35.0489543Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv 2024-12-18T01:01:35.0490690Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup 2024-12-18T01:01:35.0492011Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups 2024-12-18T01:01:35.0493247Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size 2024-12-18T01:01:35.0494580Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2024-12-18T01:01:35.0495742Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2024-12-18T01:01:35.0496797Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl 2024-12-18T01:01:35.0497880Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2024-12-18T01:01:35.0499148Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger 2024-12-18T01:01:35.0500079Z 2024-12-18T01:01:35.0500379Z Running distributed/test_distributed_spawn 4/9 ... [2024-12-18 01:01:35.043671] 2024-12-18T01:01:35.0500934Z /usr/bin/mpiexec 2024-12-18T01:01:35.0501316Z MPI not available -- MPI backend tests will be skipped 2024-12-18T01:01:35.0501903Z Running distributed tests for the test backend with env init_method 2024-12-18T01:01:35.0502436Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:01:35.0503788Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:01:35.045442] 2024-12-18T01:01:39.2342311Z 2024-12-18T01:01:39.2343789Z distributed/test_distributed_spawn 4/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.9_8f89234d0f67fd40_.log 2024-12-18T01:01:39.2344913Z Running 0 items in this shard: 2024-12-18T01:01:39.2345155Z 2024-12-18T01:01:39.2351199Z Running distributed tests for the test backend with file init_method 2024-12-18T01:01:39.2352953Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:01:39.2357020Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:01:39.235517] 2024-12-18T01:01:43.4261291Z 2024-12-18T01:01:43.4262488Z distributed/test_distributed_spawn 4/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.9_8dd5100d1e74bd3e_.log 2024-12-18T01:01:43.4263631Z Running 0 items in this shard: 2024-12-18T01:01:43.4263879Z 2024-12-18T01:01:43.4267894Z Running distributed tests for the nccl backend with env init_method 2024-12-18T01:01:43.4269755Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:01:43.4274101Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:01:43.427216] 2024-12-18T01:06:05.0800621Z 2024-12-18T01:06:05.0801829Z distributed/test_distributed_spawn 4/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.9_a6d493d9d7e6b2af_.log 2024-12-18T01:06:05.0819174Z Running 31 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2024-12-18T01:06:05.0838615Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2024-12-18T01:06:05.0840218Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2024-12-18T01:06:05.0841799Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value 2024-12-18T01:06:05.0843461Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2024-12-18T01:06:05.0844829Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup 2024-12-18T01:06:05.0846155Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2024-12-18T01:06:05.0847411Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2024-12-18T01:06:05.0848622Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2024-12-18T01:06:05.0849775Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min 2024-12-18T01:06:05.0850904Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2024-12-18T01:06:05.0852006Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2024-12-18T01:06:05.0853085Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum 2024-12-18T01:06:05.0854159Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2024-12-18T01:06:05.0855350Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda 2024-12-18T01:06:05.0856593Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2024-12-18T01:06:05.0857763Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2024-12-18T01:06:05.0858938Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2024-12-18T01:06:05.0860213Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2024-12-18T01:06:05.0861643Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2024-12-18T01:06:05.0862956Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2024-12-18T01:06:05.0864151Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook 2024-12-18T01:06:05.0865382Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2024-12-18T01:06:05.0866605Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg 2024-12-18T01:06:05.0867860Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params 2024-12-18T01:06:05.0869130Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states 2024-12-18T01:06:05.0870377Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2024-12-18T01:06:05.0871748Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2024-12-18T01:06:05.0872924Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo 2024-12-18T01:06:05.0874009Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product 2024-12-18T01:06:05.0875058Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2024-12-18T01:06:05.0876068Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2024-12-18T01:06:05.0876690Z 2024-12-18T01:06:05.0876927Z Running distributed tests for the nccl backend with file init_method 2024-12-18T01:06:05.0877400Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:06:05.0878618Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:06:05.081443] 2024-12-18T01:10:26.2023798Z 2024-12-18T01:10:26.2024955Z distributed/test_distributed_spawn 4/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.9_d634cfba6ecdc720_.log 2024-12-18T01:10:26.2056212Z Running 31 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2024-12-18T01:10:26.2084984Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2024-12-18T01:10:26.2086606Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2024-12-18T01:10:26.2088265Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value 2024-12-18T01:10:26.2089778Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2024-12-18T01:10:26.2091068Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup 2024-12-18T01:10:26.2092309Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2024-12-18T01:10:26.2093610Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2024-12-18T01:10:26.2094852Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2024-12-18T01:10:26.2096037Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min 2024-12-18T01:10:26.2097200Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2024-12-18T01:10:26.2098327Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2024-12-18T01:10:26.2099671Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum 2024-12-18T01:10:26.2100872Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2024-12-18T01:10:26.2102147Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda 2024-12-18T01:10:26.2103456Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2024-12-18T01:10:26.2104699Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2024-12-18T01:10:26.2105878Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2024-12-18T01:10:26.2107003Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2024-12-18T01:10:26.2108326Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2024-12-18T01:10:26.2109651Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2024-12-18T01:10:26.2110863Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook 2024-12-18T01:10:26.2112186Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2024-12-18T01:10:26.2113339Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg 2024-12-18T01:10:26.2114507Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params 2024-12-18T01:10:26.2115702Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states 2024-12-18T01:10:26.2116949Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2024-12-18T01:10:26.2118057Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2024-12-18T01:10:26.2119130Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo 2024-12-18T01:10:26.2120281Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product 2024-12-18T01:10:26.2121397Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2024-12-18T01:10:26.2122467Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2024-12-18T01:10:26.2123064Z 2024-12-18T01:10:26.2123313Z Running distributed tests for the gloo backend with env init_method 2024-12-18T01:10:26.2123817Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:10:26.2125093Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:10:26.204293] 2024-12-18T01:15:19.7193497Z 2024-12-18T01:15:19.7194660Z distributed/test_distributed_spawn 4/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.9_2c4f089c130ee9e8_.log 2024-12-18T01:15:19.7211479Z Running 31 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2024-12-18T01:15:19.7228698Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2024-12-18T01:15:19.7230289Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2024-12-18T01:15:19.7232084Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value 2024-12-18T01:15:19.7233650Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2024-12-18T01:15:19.7234982Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup 2024-12-18T01:15:19.7236273Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2024-12-18T01:15:19.7237720Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2024-12-18T01:15:19.7239011Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2024-12-18T01:15:19.7240238Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min 2024-12-18T01:15:19.7241442Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2024-12-18T01:15:19.7242614Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2024-12-18T01:15:19.7243868Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum 2024-12-18T01:15:19.7244977Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2024-12-18T01:15:19.7246205Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda 2024-12-18T01:15:19.7247478Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2024-12-18T01:15:19.7248683Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2024-12-18T01:15:19.7249822Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2024-12-18T01:15:19.7250905Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2024-12-18T01:15:19.7252189Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2024-12-18T01:15:19.7253584Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2024-12-18T01:15:19.7254740Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook 2024-12-18T01:15:19.7255934Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2024-12-18T01:15:19.7257122Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg 2024-12-18T01:15:19.7258342Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params 2024-12-18T01:15:19.7259819Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states 2024-12-18T01:15:19.7261071Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2024-12-18T01:15:19.7262245Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2024-12-18T01:15:19.7263383Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo 2024-12-18T01:15:19.7264606Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product 2024-12-18T01:15:19.7265782Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2024-12-18T01:15:19.7266918Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2024-12-18T01:15:19.7267556Z 2024-12-18T01:15:19.7267818Z Running distributed tests for the gloo backend with file init_method 2024-12-18T01:15:19.7268418Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:15:19.7269783Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:15:19.721129] 2024-12-18T01:20:13.7856686Z 2024-12-18T01:20:13.7857938Z distributed/test_distributed_spawn 4/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.9_b5c578dacbc86ace_.log 2024-12-18T01:20:13.7875484Z Running 31 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2024-12-18T01:20:13.7893337Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2024-12-18T01:20:13.7894909Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2024-12-18T01:20:13.7896445Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value 2024-12-18T01:20:13.7897972Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2024-12-18T01:20:13.7899517Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup 2024-12-18T01:20:13.7900820Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2024-12-18T01:20:13.7902159Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2024-12-18T01:20:13.7903447Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max 2024-12-18T01:20:13.7904676Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min 2024-12-18T01:20:13.7905883Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2024-12-18T01:20:13.7907058Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2024-12-18T01:20:13.7908195Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum 2024-12-18T01:20:13.7909330Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2024-12-18T01:20:13.7910657Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda 2024-12-18T01:20:13.7912080Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2024-12-18T01:20:13.7913287Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2024-12-18T01:20:13.7914430Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast 2024-12-18T01:20:13.7915521Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2024-12-18T01:20:13.7916806Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2024-12-18T01:20:13.7918098Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2024-12-18T01:20:13.7919256Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook 2024-12-18T01:20:13.7920450Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2024-12-18T01:20:13.7921633Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg 2024-12-18T01:20:13.7922855Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params 2024-12-18T01:20:13.7924081Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states 2024-12-18T01:20:13.7925346Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2024-12-18T01:20:13.7926492Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2024-12-18T01:20:13.7927596Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo 2024-12-18T01:20:13.7928780Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product 2024-12-18T01:20:13.7929921Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2024-12-18T01:20:13.7931024Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2024-12-18T01:20:13.7931820Z 2024-12-18T01:20:13.7932290Z Running distributed/test_distributed_spawn 7/9 ... [2024-12-18 01:20:13.787050] 2024-12-18T01:20:13.7932850Z /usr/bin/mpiexec 2024-12-18T01:20:13.7933230Z MPI not available -- MPI backend tests will be skipped 2024-12-18T01:20:13.7933816Z Running distributed tests for the test backend with env init_method 2024-12-18T01:20:13.7934332Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:20:13.7935694Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:20:13.788894] 2024-12-18T01:20:17.9948972Z 2024-12-18T01:20:17.9950202Z distributed/test_distributed_spawn 7/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.9_73d1bdd0fc2cbc90_.log 2024-12-18T01:20:17.9951376Z Running 0 items in this shard: 2024-12-18T01:20:17.9951596Z 2024-12-18T01:20:17.9957688Z Running distributed tests for the test backend with file init_method 2024-12-18T01:20:17.9959463Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:20:17.9963420Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:20:17.996160] 2024-12-18T01:20:22.1973973Z 2024-12-18T01:20:22.1975124Z distributed/test_distributed_spawn 7/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.9_e0ae957bf88ce43d_.log 2024-12-18T01:20:22.1976189Z Running 0 items in this shard: 2024-12-18T01:20:22.1976427Z 2024-12-18T01:20:22.1980176Z Running distributed tests for the nccl backend with env init_method 2024-12-18T01:20:22.1982076Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:20:22.1986317Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:20:22.198433] 2024-12-18T01:25:11.7945072Z 2024-12-18T01:25:11.7946574Z distributed/test_distributed_spawn 7/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.9_ca4ba1909aa32dd1_.log 2024-12-18T01:25:11.7965611Z Running 32 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2024-12-18T01:25:11.7984498Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2024-12-18T01:25:11.7985912Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2024-12-18T01:25:11.7987354Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2024-12-18T01:25:11.7988760Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2024-12-18T01:25:11.7990112Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2024-12-18T01:25:11.7991535Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2024-12-18T01:25:11.7992814Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2024-12-18T01:25:11.7994111Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group 2024-12-18T01:25:11.7995269Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2024-12-18T01:25:11.7996607Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda 2024-12-18T01:25:11.7997785Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier 2024-12-18T01:25:11.7998804Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2024-12-18T01:25:11.7999872Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group 2024-12-18T01:25:11.8001072Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err 2024-12-18T01:25:11.8002274Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2024-12-18T01:25:11.8003412Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2024-12-18T01:25:11.8004718Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2024-12-18T01:25:11.8006020Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged 2024-12-18T01:25:11.8007245Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn 2024-12-18T01:25:11.8008427Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group 2024-12-18T01:25:11.8009586Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params 2024-12-18T01:25:11.8010733Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future 2024-12-18T01:25:11.8011804Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2024-12-18T01:25:11.8012931Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler 2024-12-18T01:25:11.8014120Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order 2024-12-18T01:25:11.8015347Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2024-12-18T01:25:11.8016457Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2024-12-18T01:25:11.8017485Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda 2024-12-18T01:25:11.8018608Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2024-12-18T01:25:11.8020102Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2024-12-18T01:25:11.8021377Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2024-12-18T01:25:11.8022665Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2024-12-18T01:25:11.8023415Z 2024-12-18T01:25:11.8023678Z Running distributed tests for the nccl backend with file init_method 2024-12-18T01:25:11.8024223Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:25:11.8025848Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:25:11.796321] 2024-12-18T01:30:00.9503832Z 2024-12-18T01:30:00.9507360Z distributed/test_distributed_spawn 7/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.9_f1acca7c7cb7b082_.log 2024-12-18T01:30:00.9524779Z Running 32 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2024-12-18T01:30:00.9542594Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2024-12-18T01:30:00.9545758Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2024-12-18T01:30:00.9547245Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2024-12-18T01:30:00.9548651Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2024-12-18T01:30:00.9549929Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2024-12-18T01:30:00.9551355Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2024-12-18T01:30:00.9552649Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2024-12-18T01:30:00.9553869Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group 2024-12-18T01:30:00.9555017Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2024-12-18T01:30:00.9556277Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda 2024-12-18T01:30:00.9557457Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier 2024-12-18T01:30:00.9558664Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2024-12-18T01:30:00.9559768Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group 2024-12-18T01:30:00.9561111Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err 2024-12-18T01:30:00.9562343Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2024-12-18T01:30:00.9563519Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2024-12-18T01:30:00.9564871Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2024-12-18T01:30:00.9566219Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged 2024-12-18T01:30:00.9567486Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn 2024-12-18T01:30:00.9568688Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group 2024-12-18T01:30:00.9569821Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params 2024-12-18T01:30:00.9570937Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future 2024-12-18T01:30:00.9572037Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2024-12-18T01:30:00.9573200Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler 2024-12-18T01:30:00.9574424Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order 2024-12-18T01:30:00.9575671Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2024-12-18T01:30:00.9576884Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2024-12-18T01:30:00.9577932Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda 2024-12-18T01:30:00.9579347Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2024-12-18T01:30:00.9580651Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2024-12-18T01:30:00.9581907Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2024-12-18T01:30:00.9583199Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2024-12-18T01:30:00.9583943Z 2024-12-18T01:30:00.9584232Z Running distributed tests for the gloo backend with env init_method 2024-12-18T01:30:00.9584760Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:30:00.9586122Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:30:00.952225] 2024-12-18T01:34:33.7110689Z 2024-12-18T01:34:33.7111995Z distributed/test_distributed_spawn 7/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.9_5dcb700ebef37f01_.log 2024-12-18T01:34:33.7129146Z Running 32 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2024-12-18T01:34:33.7146935Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2024-12-18T01:34:33.7148268Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2024-12-18T01:34:33.7149697Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2024-12-18T01:34:33.7151293Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2024-12-18T01:34:33.7152494Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2024-12-18T01:34:33.7153647Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2024-12-18T01:34:33.7155043Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2024-12-18T01:34:33.7156432Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group 2024-12-18T01:34:33.7157612Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2024-12-18T01:34:33.7158908Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda 2024-12-18T01:34:33.7160118Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier 2024-12-18T01:34:33.7161171Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2024-12-18T01:34:33.7162297Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group 2024-12-18T01:34:33.7163520Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err 2024-12-18T01:34:33.7164745Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2024-12-18T01:34:33.7165928Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2024-12-18T01:34:33.7167274Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2024-12-18T01:34:33.7168885Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged 2024-12-18T01:34:33.7170230Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn 2024-12-18T01:34:33.7171449Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group 2024-12-18T01:34:33.7172681Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params 2024-12-18T01:34:33.7173754Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future 2024-12-18T01:34:33.7174995Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2024-12-18T01:34:33.7176177Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler 2024-12-18T01:34:33.7177390Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order 2024-12-18T01:34:33.7178720Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2024-12-18T01:34:33.7180060Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2024-12-18T01:34:33.7181152Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda 2024-12-18T01:34:33.7182359Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2024-12-18T01:34:33.7183661Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2024-12-18T01:34:33.7184916Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2024-12-18T01:34:33.7186274Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2024-12-18T01:34:33.7187014Z 2024-12-18T01:34:33.7187279Z Running distributed tests for the gloo backend with file init_method 2024-12-18T01:34:33.7187813Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:34:33.7189186Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:34:33.712898] 2024-12-18T01:39:05.9417608Z 2024-12-18T01:39:05.9418776Z distributed/test_distributed_spawn 7/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.9_5bdb35fd530124a2_.log 2024-12-18T01:39:05.9437050Z Running 32 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2024-12-18T01:39:05.9454252Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2024-12-18T01:39:05.9455536Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2024-12-18T01:39:05.9456918Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad 2024-12-18T01:39:05.9458272Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook 2024-12-18T01:39:05.9459761Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda 2024-12-18T01:39:05.9461004Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2024-12-18T01:39:05.9462361Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2024-12-18T01:39:05.9463651Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group 2024-12-18T01:39:05.9464877Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2024-12-18T01:39:05.9466224Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda 2024-12-18T01:39:05.9467473Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier 2024-12-18T01:39:05.9468632Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda 2024-12-18T01:39:05.9469786Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group 2024-12-18T01:39:05.9471041Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err 2024-12-18T01:39:05.9472387Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2024-12-18T01:39:05.9473563Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2024-12-18T01:39:05.9475001Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2024-12-18T01:39:05.9476307Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged 2024-12-18T01:39:05.9477537Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn 2024-12-18T01:39:05.9478710Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group 2024-12-18T01:39:05.9479804Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params 2024-12-18T01:39:05.9480881Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future 2024-12-18T01:39:05.9481951Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2024-12-18T01:39:05.9483077Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler 2024-12-18T01:39:05.9484335Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order 2024-12-18T01:39:05.9485538Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2024-12-18T01:39:05.9486643Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2024-12-18T01:39:05.9487673Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda 2024-12-18T01:39:05.9488793Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2024-12-18T01:39:05.9490015Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2024-12-18T01:39:05.9491280Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2024-12-18T01:39:05.9492427Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2024-12-18T01:39:05.9493098Z 2024-12-18T01:39:05.9493373Z Running distributed/_tensor/test_redistribute 1/1 ... [2024-12-18 01:39:05.943216] 2024-12-18T01:39:05.9493895Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:39:05.9495061Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_redistribute.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:39:05.943567] 2024-12-18T01:40:40.8577459Z 2024-12-18T01:40:40.8578444Z distributed/_tensor/test_redistribute 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_redistribute_1.1_31a688f553ef4281_.log 2024-12-18T01:40:40.8586680Z Running 13 items in this shard: test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_partial_to_replicate_forward_backward, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_partial_to_shard, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_redistribute_negative_shard_dim, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_redistribute_shard_dim_change, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_redistribute_uneven_sharding, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_local_partial_grad, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_partial, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_replicate_forward_backward, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_shard_forward_backward, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_shard_dim_alltoall, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_shard_to_replicate_forward_backward, test/distributed/_tensor/test_redistribute.py::MultiDimRedistributeTest::test_multi_dim_mesh, test/distributed/_tensor/test_redistribute.py::MultiDimRedistributeTest::test_redistribute_shard_dim_multi_dim_mesh 2024-12-18T01:40:40.8593532Z 2024-12-18T01:40:40.8593859Z Running distributed/_tools/test_fsdp2_mem_tracker 1/1 ... [2024-12-18 01:40:40.858573] 2024-12-18T01:40:40.8594432Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:40:40.8595715Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_fsdp2_mem_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:40:40.858939] 2024-12-18T01:41:14.7599000Z 2024-12-18T01:41:14.7600353Z distributed/_tools/test_fsdp2_mem_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_fsdp2_mem_tracker_1.1_bee6ec382cb18ebe_.log 2024-12-18T01:41:14.7603732Z Running 3 items in this shard: test/distributed/_tools/test_fsdp2_mem_tracker.py::TestTrackerFullyShard1DTrainingCore::test_tracker_multi_group_eager, test/distributed/_tools/test_fsdp2_mem_tracker.py::TestTrackerFullyShard1DTrainingCore::test_tracker_non_root_forward_backward, test/distributed/_tools/test_fsdp2_mem_tracker.py::TestTrackerFullyShard1DTrainingCompose::test_tracker_with_activation_checkpointing 2024-12-18T01:41:14.7605767Z 2024-12-18T01:41:14.7606115Z Running distributed/checkpoint/test_format_utils 1/1 ... [2024-12-18 01:41:14.759923] 2024-12-18T01:41:14.7606718Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:41:14.7608040Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_format_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:41:14.760302] 2024-12-18T01:41:37.5338041Z 2024-12-18T01:41:37.5339559Z distributed/checkpoint/test_format_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_format_utils_1.1_be21b584631f774b_.log 2024-12-18T01:41:37.5342074Z Running 3 items in this shard: test/distributed/checkpoint/test_format_utils.py::TestFormatUtils::test_dcp_to_torch_save, test/distributed/checkpoint/test_format_utils.py::TestFormatUtils::test_online_torch_save_to_dcp, test/distributed/checkpoint/test_format_utils.py::TestFormatUtils::test_torch_save_to_dcp 2024-12-18T01:41:37.5343653Z 2024-12-18T01:41:37.5344032Z Running distributed/_tensor/debug/test_comm_mode_features 1/1 ... [2024-12-18 01:41:37.534219] 2024-12-18T01:41:37.5344783Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:41:37.5348740Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/debug/test_comm_mode_features.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:41:37.534650] 2024-12-18T01:42:09.2316568Z 2024-12-18T01:42:09.2321040Z distributed/_tensor/debug/test_comm_mode_features 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_comm_mode_features_1.1_2d5ca764175a67b1_.log 2024-12-18T01:42:09.2324340Z Running 4 items in this shard: test/distributed/_tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_MLPStacked_distributed_sharding_display, test/distributed/_tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_MLP_distributed_sharding_display, test/distributed/_tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_MLP_module_tracing, test/distributed/_tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_transformer_module_tracing 2024-12-18T01:42:09.2326746Z 2024-12-18T01:42:09.2327111Z Running distributed/checkpoint/e2e/test_fine_tuning 1/1 ... [2024-12-18 01:42:09.232027] 2024-12-18T01:42:09.2327719Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:42:09.2329025Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_fine_tuning.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:42:09.232411] 2024-12-18T01:42:20.8777050Z 2024-12-18T01:42:20.8778272Z distributed/checkpoint/e2e/test_fine_tuning 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_fine_tuning_1.1_0c15d68a9a2f340c_.log 2024-12-18T01:42:20.8780223Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_fine_tuning.py::TestFineTuning::test_fine_tuning 2024-12-18T01:42:20.8780865Z 2024-12-18T01:42:20.8781253Z Running distributed/_tensor/experimental/test_tp_transform 1/1 ... [2024-12-18 01:42:20.877781] 2024-12-18T01:42:20.8782237Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:42:20.8784078Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/experimental/test_tp_transform.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:42:20.878181] 2024-12-18T01:42:47.9645124Z 2024-12-18T01:42:47.9646445Z distributed/_tensor/experimental/test_tp_transform 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.experimental.test_tp_transform_1.1_ec717b929b1f810e_.log 2024-12-18T01:42:47.9649399Z Running 3 items in this shard: test/distributed/_tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_e2e, test/distributed/_tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_no_bias, test/distributed/_tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_with_uncovered_op 2024-12-18T01:42:47.9651275Z 2024-12-18T01:42:47.9651622Z Running distributed/checkpoint/e2e/test_pipeline 1/1 ... [2024-12-18 01:42:47.964503] 2024-12-18T01:42:47.9652225Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:42:47.9653523Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_pipeline.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:42:47.964859] 2024-12-18T01:42:59.1090104Z 2024-12-18T01:42:59.1091432Z distributed/checkpoint/e2e/test_pipeline 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_pipeline_1.1_b00ce770f3777b1d_.log 2024-12-18T01:42:59.1092892Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_pipeline.py::TestPipeline::test_pipeline 2024-12-18T01:42:59.1093504Z 2024-12-18T01:42:59.1095663Z Running distributed/checkpoint/test_traverse 1/1 ... [2024-12-18 01:42:59.109112] 2024-12-18T01:42:59.1096623Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:42:59.1099000Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_traverse.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:42:59.109565] 2024-12-18T01:43:03.4352479Z 2024-12-18T01:43:03.4353725Z distributed/checkpoint/test_traverse 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_traverse_1.1_f48fa04480dc491f_.log 2024-12-18T01:43:03.4357772Z Running 7 items in this shard: test/distributed/checkpoint/test_traverse.py::TestTraverse::test_get_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_set_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_doesnt_ignore_intermediate_collections, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_dict, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_list, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_shallow, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_with_ordered_dict 2024-12-18T01:43:03.4360961Z 2024-12-18T01:43:03.4361342Z Running distributed/tensor/parallel/test_tp_random_state 1/1 ... [2024-12-18 01:43:03.435271] 2024-12-18T01:43:03.4361980Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:43:03.4363317Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_random_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:43:03.435686] 2024-12-18T01:43:15.0802839Z 2024-12-18T01:43:15.0804608Z distributed/tensor/parallel/test_tp_random_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_random_state_1.1_1043dfe5b2b4bc5e_.log 2024-12-18T01:43:15.0806315Z Running 1 items in this shard: test/distributed/tensor/parallel/test_tp_random_state.py::TensorParallelRandomStateTests::test_model_init 2024-12-18T01:43:15.0807021Z 2024-12-18T01:43:15.0808428Z Running distributed/checkpoint/test_dtensor_checkpoint 1/1 ... [2024-12-18 01:43:15.080449] 2024-12-18T01:43:15.0809048Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:43:15.0810379Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dtensor_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:43:15.080807] 2024-12-18T01:43:26.2244512Z 2024-12-18T01:43:26.2245832Z distributed/checkpoint/test_dtensor_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dtensor_checkpoint_1.1_dbd250dbb5f09c33_.log 2024-12-18T01:43:26.2247542Z Running 1 items in this shard: test/distributed/checkpoint/test_dtensor_checkpoint.py::DTensorPlanner::test_distributed_tensor_planner 2024-12-18T01:43:26.2248235Z 2024-12-18T01:43:26.2248628Z Running distributed/_composable/test_replicate_with_compiler 1/1 ... [2024-12-18 01:43:26.224420] 2024-12-18T01:43:26.2249263Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:43:26.2250611Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_replicate_with_compiler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:43:26.224785] 2024-12-18T01:45:54.8597550Z 2024-12-18T01:45:54.8599255Z distributed/_composable/test_replicate_with_compiler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_replicate_with_compiler_1.1_11a7bf26a24fcdfe_.log 2024-12-18T01:45:54.8605096Z Running 10 items in this shard: test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_bucketing_coalesced_op, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_bucketing_concat_op, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_backward_only, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_bf16, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_cpu, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_cpu_no_sync, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_fp16, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_gpu, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_gpu_ac, test/distributed/_composable/test_replicate_with_compiler.py::DDP_TP_Test::test_ddp_tp 2024-12-18T01:45:54.8609763Z 2024-12-18T01:45:54.8610081Z Running distributed/test_collective_utils 1/1 ... [2024-12-18 01:45:54.859856] 2024-12-18T01:45:54.8610627Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:45:54.8611882Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_collective_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:45:54.860206] 2024-12-18T01:45:58.7433215Z 2024-12-18T01:45:58.7434365Z distributed/test_collective_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_collective_utils_1.1_d7c50749f4c551f6_.log 2024-12-18T01:45:58.7435283Z 2024-12-18T01:45:58.7437585Z Running distributed/test_c10d_logger 1/1 ... [2024-12-18 01:45:58.743637] 2024-12-18T01:45:58.7438555Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:45:58.7441880Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:45:58.744008] 2024-12-18T01:46:14.6003530Z 2024-12-18T01:46:14.6004639Z distributed/test_c10d_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_logger_1.1_e661757fab846e4d_.log 2024-12-18T01:46:14.6006352Z Running 2 items in this shard: test/distributed/test_c10d_logger.py::C10dErrorLoggerTest::test_exception_logger, test/distributed/test_c10d_logger.py::C10dErrorLoggerTest::test_get_or_create_logger 2024-12-18T01:46:14.6007308Z 2024-12-18T01:46:14.6007640Z Running distributed/fsdp/test_fsdp_state_dict 1/2 ... [2024-12-18 01:46:14.600300] 2024-12-18T01:46:14.6008256Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:46:14.6009536Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '--shard-id=1', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:46:14.600657] 2024-12-18T01:53:45.4189560Z 2024-12-18T01:53:45.4192382Z distributed/fsdp/test_fsdp_state_dict 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_state_dict_1.2_3c83763df6c2269c_.log 2024-12-18T01:53:45.4262721Z Running 78 items in this shard: test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_full_state_dict_missing_unexpected_keys_cleaned, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_rank0_offload_save_load_flow_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_type, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_torch_save_load, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict4GPUs::test_local_state_dict_reshard 2024-12-18T01:53:45.4329399Z 2024-12-18T01:53:45.4329638Z Running distributed/test_c10d_nccl 2/3 ... [2024-12-18 01:53:45.420156] 2024-12-18T01:53:45.4330119Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:53:45.4331461Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=2', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:53:45.420521] 2024-12-18T02:07:39.7927793Z 2024-12-18T02:07:39.7928888Z distributed/test_c10d_nccl 2/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_2.3_898ada9fd3d2861e_.log 2024-12-18T02:07:39.7964758Z Running 74 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e4m3fn, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_rank_filter, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_p2p, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off, test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_unwaited, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_recursive_split_group 2024-12-18T02:07:39.7999666Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl 2024-12-18T02:07:39.8000658Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg 2024-12-18T02:07:39.8001691Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup 2024-12-18T02:07:39.8002777Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg 2024-12-18T02:07:39.8003985Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend0 2024-12-18T02:07:39.8005008Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16 2024-12-18T02:07:39.8005970Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64 2024-12-18T02:07:39.8006961Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e4m3fn 2024-12-18T02:07:39.8007920Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_check 2024-12-18T02:07:39.8008836Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_rank_filter 2024-12-18T02:07:39.8009934Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_False 2024-12-18T02:07:39.8010938Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_p2p 2024-12-18T02:07:39.8011929Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0 2024-12-18T02:07:39.8012962Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc 2024-12-18T02:07:39.8013998Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_True 2024-12-18T02:07:39.8015148Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view 2024-12-18T02:07:39.8016322Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value 2024-12-18T02:07:39.8017412Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl 2024-12-18T02:07:39.8018770Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view 2024-12-18T02:07:39.8020274Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2024-12-18T02:07:39.8021589Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2024-12-18T02:07:39.8022988Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False 2024-12-18T02:07:39.8024413Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view 2024-12-18T02:07:39.8025876Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph 2024-12-18T02:07:39.8027226Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl 2024-12-18T02:07:39.8028498Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config 2024-12-18T02:07:39.8029673Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing 2024-12-18T02:07:39.8030831Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl 2024-12-18T02:07:39.8032315Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail 2024-12-18T02:07:39.8033575Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule 2024-12-18T02:07:39.8034740Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state 2024-12-18T02:07:39.8035959Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward 2024-12-18T02:07:39.8037298Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list 2024-12-18T02:07:39.8038721Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list 2024-12-18T02:07:39.8040146Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None 2024-12-18T02:07:39.8041504Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg 2024-12-18T02:07:39.8042715Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view 2024-12-18T02:07:39.8043982Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops 2024-12-18T02:07:39.8044970Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit 2024-12-18T02:07:39.8045997Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill 2024-12-18T02:07:39.8046982Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking 2024-12-18T02:07:39.8047932Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl 2024-12-18T02:07:39.8048835Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl 2024-12-18T02:07:39.8049704Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl 2024-12-18T02:07:39.8050503Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier 2024-12-18T02:07:39.8051309Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids 2024-12-18T02:07:39.8052192Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off 2024-12-18T02:07:39.8053070Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership 2024-12-18T02:07:39.8053971Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream 2024-12-18T02:07:39.8054971Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k 2024-12-18T02:07:39.8055843Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k_float8_errors 2024-12-18T02:07:39.8056680Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_unwaited 2024-12-18T02:07:39.8057625Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives 2024-12-18T02:07:39.8058725Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_False 2024-12-18T02:07:39.8060144Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_True 2024-12-18T02:07:39.8061376Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_False 2024-12-18T02:07:39.8062501Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_True 2024-12-18T02:07:39.8063632Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_False 2024-12-18T02:07:39.8064740Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_False 2024-12-18T02:07:39.8065771Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync 2024-12-18T02:07:39.8066865Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_True 2024-12-18T02:07:39.8068131Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_True 2024-12-18T02:07:39.8069514Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_True 2024-12-18T02:07:39.8070800Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_True 2024-12-18T02:07:39.8072177Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False 2024-12-18T02:07:39.8073493Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False 2024-12-18T02:07:39.8074800Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True 2024-12-18T02:07:39.8076057Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False 2024-12-18T02:07:39.8077288Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True 2024-12-18T02:07:39.8078599Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_True 2024-12-18T02:07:39.8079990Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_False 2024-12-18T02:07:39.8081257Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_True 2024-12-18T02:07:39.8082634Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False 2024-12-18T02:07:39.8083832Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_recursive_split_group 2024-12-18T02:07:39.8084537Z 2024-12-18T02:07:39.8084855Z Running distributed/pipelining/test_schedule 1/1 ... [2024-12-18 02:07:39.794327] 2024-12-18T02:07:39.8085401Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:07:39.8086644Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_schedule.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:07:39.794694] 2024-12-18T02:08:27.5182424Z 2024-12-18T02:08:27.5184031Z distributed/pipelining/test_schedule 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_schedule_1.1_45c09618d6ffe387_.log 2024-12-18T02:08:27.5201691Z Running 28 items in this shard: test/distributed/pipelining/test_schedule.py::ScheduleTest::test_get_schedule_class, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass0, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass1, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass2, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass3, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass4, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_ScheduleClass0, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_ScheduleClass1, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_flex_and_zero_bubble_ScheduleClass0, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_flex_and_zero_bubble_ScheduleClass1, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref0, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref1, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref2, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref3, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref4, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref5, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref6, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref7, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_csv_csv_name_zb1p_2rank_2stagep, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_grad_with_split_b_w, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_grad_with_v_schedule, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_merge_bw_test_info0, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_send_recv_test_info0, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_send_recv_test_info1, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_unshard_reshard_test_info0, test/distributed/pipelining/test_schedule.py::TestValidateSchedule::test_invalid_schedule_missing_action, test/distributed/pipelining/test_schedule.py::TestValidateSchedule::test_invalid_schedule_missing_rank, test/distributed/pipelining/test_schedule.py::TestValidateSchedule::test_valid_schedule 2024-12-18T02:08:27.5215997Z 2024-12-18T02:08:27.5216314Z Running distributed/_tensor/test_op_strategy 1/1 ... [2024-12-18 02:08:27.518390] 2024-12-18T02:08:27.5216886Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:08:27.5218145Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_op_strategy.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:08:27.518861] 2024-12-18T02:08:32.0952039Z 2024-12-18T02:08:32.0953589Z distributed/_tensor/test_op_strategy 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_op_strategy_1.1_2b108bfcc1b26b28_.log 2024-12-18T02:08:32.0960603Z Running 15 items in this shard: test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_batch_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_bmm_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_free_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_mm_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_bmm_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_bmm_2d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_linearity_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_mm_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_mm_2d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_pointwise_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_bmm_strategies, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_mm_strategies, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_redistribute_cost_latency, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_redistribute_cost_mesh_1d, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_redistribute_cost_mesh_2d 2024-12-18T02:08:32.0966836Z 2024-12-18T02:08:32.0967187Z Running distributed/checkpoint/test_fsdp_model_state 1/1 ... [2024-12-18 02:08:32.095161] 2024-12-18T02:08:32.0967800Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:08:32.0969417Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_model_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:08:32.095527] 2024-12-18T02:08:50.5069828Z 2024-12-18T02:08:50.5071131Z distributed/checkpoint/test_fsdp_model_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_model_state_1.1_14c464eb9723dcad_.log 2024-12-18T02:08:50.5073502Z Running 2 items in this shard: test/distributed/checkpoint/test_fsdp_model_state.py::FsdpModelStateCheckpoint::test_fsdp_model_state_no_resharding, test/distributed/checkpoint/test_fsdp_model_state.py::FsdpModelStateCheckpoint::test_fsdp_model_state_with_resharding 2024-12-18T02:08:50.5074864Z 2024-12-18T02:08:50.5075793Z Running distributed/rpc/test_share_memory 1/1 ... [2024-12-18 02:08:50.507473] 2024-12-18T02:08:50.5076346Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:08:50.5080109Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/rpc/test_share_memory.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:08:50.507828] 2024-12-18T02:09:03.5133936Z 2024-12-18T02:09:03.5135152Z distributed/rpc/test_share_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.test_share_memory_1.1_a5856aa9a2422d00_.log 2024-12-18T02:09:03.5136508Z Running 1 items in this shard: test/distributed/rpc/test_share_memory.py::TestRPCPickler::test_case 2024-12-18T02:09:03.5137501Z Running 1 items in this shard: test/distributed/rpc/test_share_memory.py::TestRPCPickler::test_case 2024-12-18T02:09:03.5138085Z 2024-12-18T02:09:03.5138441Z Running distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 ... [2024-12-18 02:09:03.513640] 2024-12-18T02:09:03.5139184Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:09:03.5142679Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_dtensor_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:09:03.514032] 2024-12-18T02:11:42.1832417Z 2024-12-18T02:11:42.1833703Z distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_dtensor_state_dict_1.1_64fea7bb21c59bcd_.log 2024-12-18T02:11:42.1847905Z Running 15 items in this shard: test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_False_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_False_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_True_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_True_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_False_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_False_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_True_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_True_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_False_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_False_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_True_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_True_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_fsdp_init_with_device_mesh_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_fsdp_init_with_device_mesh_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_raises_warning_or_errors_cuda 2024-12-18T02:11:42.1861172Z 2024-12-18T02:11:42.1861528Z Running distributed/checkpoint/test_checkpoint 1/1 ... [2024-12-18 02:11:42.183509] 2024-12-18T02:11:42.1862142Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:11:42.1863489Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:11:42.183873] 2024-12-18T02:12:34.9877059Z 2024-12-18T02:12:34.9878299Z distributed/checkpoint/test_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_checkpoint_1.1_b75de875062f8f79_.log 2024-12-18T02:12:34.9883864Z Running 8 items in this shard: test/distributed/checkpoint/test_checkpoint.py::TestDistributedCheckpointing::test_default_metadata, test/distributed/checkpoint/test_checkpoint.py::TestDistributedCheckpointing::test_tensor_metadata_with_missing_rank_spec, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_dummy_reader_works, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_dummy_writer_works, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_load_error_handling, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_load_error_handling_no_dist, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_save_error_handling, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_save_error_handling_no_dist 2024-12-18T02:12:34.9888111Z 2024-12-18T02:12:34.9888415Z Running distributed/fsdp/test_fsdp_memory 1/1 ... [2024-12-18 02:12:34.987999] 2024-12-18T02:12:34.9888985Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:12:34.9890249Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:12:34.988381] 2024-12-18T02:12:53.0990963Z 2024-12-18T02:12:53.0992173Z distributed/fsdp/test_fsdp_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_memory_1.1_c04dba40b7dc4db9_.log 2024-12-18T02:12:53.0994067Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_memory.py::TestFSDPMemory::test_fsdp_memory_ckpt_ckpt, test/distributed/fsdp/test_fsdp_memory.py::TestFSDPMemory::test_fsdp_memory_ckpt_no_ckpt 2024-12-18T02:12:53.0995071Z 2024-12-18T02:12:53.0995467Z Running distributed/_composable/fsdp/test_fully_shard_frozen 1/1 ... [2024-12-18 02:12:53.099171] 2024-12-18T02:12:53.0996524Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:12:53.0997901Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_frozen.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:12:53.099523] 2024-12-18T02:13:26.4503190Z 2024-12-18T02:13:26.4504524Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_f516fde209b526bf_.log 2024-12-18T02:13:26.4507621Z Running 3 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_multi_forward_mixed_requires_grad, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_across_groups, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_per_group 2024-12-18T02:13:26.4509809Z 2024-12-18T02:13:26.4510200Z Running distributed/tensor/parallel/test_parallelize_api 1/1 ... [2024-12-18 02:13:26.450661] 2024-12-18T02:13:26.4510911Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:13:26.4512372Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_parallelize_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:13:26.451024] 2024-12-18T02:14:52.5376656Z 2024-12-18T02:14:52.5378000Z distributed/tensor/parallel/test_parallelize_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_parallelize_api_1.1_df650fe5ac7b691d_.log 2024-12-18T02:14:52.5386583Z Running 12 items in this shard: test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_empty_plan, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_linear_col_wise_parallel, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_linear_row_wise_parallel, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_mlp_with_module_api, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_mlp_with_module_api_nested, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_module_multi_wildcard, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_module_with_digit, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_module_with_question, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_module_with_star, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_prepare_module_input, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_prepare_module_output, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_under_devicemesh_context 2024-12-18T02:14:52.5394202Z 2024-12-18T02:14:52.5394529Z Running distributed/fsdp/test_fsdp_traversal 1/1 ... [2024-12-18 02:14:52.537726] 2024-12-18T02:14:52.5395085Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:14:52.5396351Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:14:52.538131] 2024-12-18T02:15:04.6831742Z 2024-12-18T02:15:04.6833560Z distributed/fsdp/test_fsdp_traversal 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_traversal_1.1_d2aa15fbf48b5d69_.log 2024-12-18T02:15:04.6835097Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_traversal.py::TestTraversalCUDA::test_fsdp_modules_cuda 2024-12-18T02:15:04.6835694Z 2024-12-18T02:15:04.6836028Z Running distributed/checkpoint/test_state_dict 1/1 ... [2024-12-18 02:15:04.683020] 2024-12-18T02:15:04.6836619Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:15:04.6837907Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:15:04.683414] 2024-12-18T02:17:58.9484506Z 2024-12-18T02:17:58.9486413Z distributed/checkpoint/test_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_state_dict_1.1_325f5010e1535b47_.log 2024-12-18T02:17:58.9507936Z Running 23 items in this shard: test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_activation_ckpt_fqns_ddp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_activation_ckpt_fqns_fsdp1, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_broadcast_from_rank0, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_broadcast_from_rank0_hsdp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_compiled_fsdp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_cpu_offload_full_state_dict, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_ddp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_deprecate_fsdp_api, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_deprecate_partial, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_extra_state, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_flattened_osd, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_fsdp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_fsdp2, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_fsdp_ddp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_fsdp_root_not_initialized, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_non_persistent_buffers, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_optim_state_dict_param_matching, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_setting_meta_device_model, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_setting_meta_device_model_broadcasting, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_shared_weight, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_single_gpu, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_strict, test/distributed/checkpoint/test_state_dict.py::TestNoComm::test_no_dist 2024-12-18T02:17:58.9527613Z 2024-12-18T02:17:58.9532423Z Running distributed/test_multi_threaded_pg 1/1 ... [2024-12-18 02:17:58.953043] 2024-12-18T02:17:58.9533451Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:17:58.9539008Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_multi_threaded_pg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:17:58.953575] 2024-12-18T02:18:03.7302301Z 2024-12-18T02:18:03.7304274Z distributed/test_multi_threaded_pg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_multi_threaded_pg_1.1_b98190b55ff89008_.log 2024-12-18T02:18:03.7320712Z Running 22 items in this shard: test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_all_to_all_single_list, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_all_to_all_single_none, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_all_to_all_single_tensor, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_broadcast_object_list, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_non_zero, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_non_zero_all, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_zero, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_skip, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce_coalesced, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce_ops, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_to_all, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_allgather, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_assert_equal_on_rank, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_broadcast, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_broadcast_object_list, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_bwd_sees_fwd_pg, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_gather, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_reduce_scatter, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_scatter, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_subpg, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_using_pg_from_another_thread 2024-12-18T02:18:03.7331125Z 2024-12-18T02:18:03.7331431Z Running distributed/fsdp/test_fsdp_uneven 1/1 ... [2024-12-18 02:18:03.730499] 2024-12-18T02:18:03.7332482Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:18:03.7333737Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:18:03.731045] 2024-12-18T02:18:18.8345042Z 2024-12-18T02:18:18.8347066Z distributed/fsdp/test_fsdp_uneven 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_uneven_1.1_9e3c6300d8470ab0_.log 2024-12-18T02:18:18.8349723Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShardCUDA::test_one_iteration_cuda 2024-12-18T02:18:18.8350736Z 2024-12-18T02:18:18.8354969Z Running distributed/fsdp/test_fsdp_input 1/1 ... [2024-12-18 02:18:18.835056] 2024-12-18T02:18:18.8355852Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:18:18.8358124Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:18:18.835498] 2024-12-18T02:18:35.6921656Z 2024-12-18T02:18:35.6926599Z distributed/fsdp/test_fsdp_input 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_input_1.1_1614b5d15bcc8270_.log 2024-12-18T02:18:35.6929819Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda, test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda 2024-12-18T02:18:35.6930900Z 2024-12-18T02:18:35.6931167Z Running distributed/test_c10d_ucc 1/1 ... [2024-12-18 02:18:35.692149] 2024-12-18T02:18:35.6931875Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:18:35.6933539Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_ucc.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:18:35.692582] 2024-12-18T02:27:14.8425906Z 2024-12-18T02:27:14.8427614Z distributed/test_c10d_ucc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_ucc_1.1_9b584052081fcd60_.log 2024-12-18T02:27:14.8476129Z Running 62 items in this shard: test/distributed/test_c10d_ucc.py::RendezvousEnvTest::test_logging_init, test/distributed/test_c10d_ucc.py::TimeoutTest::test_default_store_timeout_ucc, test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_allgather_basics, test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_allreduce_basics, test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_barrier_implies_wait, test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_broadcast_basics, test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_empty_tensors, test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_reduce_basics, test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_reduce_scatter_base_basics, test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_send_recv_all_to_all, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_ucc, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_global_local_unused_params_grad, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ignored_output, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_save_load_checkpoint, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sparse_gradients, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_2gpu_module, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_4gpu_module, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_cpu_module, test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_cpu_module_grad_is_view, test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_incremented_ucc_default, test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_incremented_ucc_subgroup, test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_set_default_pg_ucc, test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_set_ucc_new_group, test/distributed/test_c10d_ucc.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_ucc.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_ucc.py::CommTest::test_ucc_barrier_device_ids, test/distributed/test_c10d_ucc.py::CommTest::test_ucc_rank_membership, test/distributed/test_c10d_ucc.py::CommTest::test_ucc_warn_not_in_group, test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_allgather_base, test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-12-18T02:27:14.8506118Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::RendezvousEnvTest::test_logging_init 2024-12-18T02:27:14.8507084Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::TimeoutTest::test_default_store_timeout_ucc 2024-12-18T02:27:14.8508067Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_allgather_basics 2024-12-18T02:27:14.8509048Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_allreduce_basics 2024-12-18T02:27:14.8510061Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_barrier_implies_wait 2024-12-18T02:27:14.8511248Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_broadcast_basics 2024-12-18T02:27:14.8512114Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_empty_tensors 2024-12-18T02:27:14.8512962Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_reduce_basics 2024-12-18T02:27:14.8513892Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_reduce_scatter_base_basics 2024-12-18T02:27:14.8514816Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_send_recv_all_to_all 2024-12-18T02:27:14.8515743Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_dataclass_output 2024-12-18T02:27:14.8518531Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_dataclass_output_unused_param 2024-12-18T02:27:14.8519722Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2024-12-18T02:27:14.8520923Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2024-12-18T02:27:14.8522185Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2024-12-18T02:27:14.8523445Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2024-12-18T02:27:14.8524766Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False 2024-12-18T02:27:14.8526150Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2024-12-18T02:27:14.8527477Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2024-12-18T02:27:14.8528847Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2024-12-18T02:27:14.8530019Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing 2024-12-18T02:27:14.8531218Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False 2024-12-18T02:27:14.8532908Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2024-12-18T02:27:14.8534457Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False 2024-12-18T02:27:14.8535880Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2024-12-18T02:27:14.8537217Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu 2024-12-18T02:27:14.8538482Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_ucc 2024-12-18T02:27:14.8539964Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once 2024-12-18T02:27:14.8541202Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients 2024-12-18T02:27:14.8542399Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init 2024-12-18T02:27:14.8543610Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type 2024-12-18T02:27:14.8544943Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty 2024-12-18T02:27:14.8546262Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_global_local_unused_params_grad 2024-12-18T02:27:14.8547573Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view 2024-12-18T02:27:14.8548964Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph 2024-12-18T02:27:14.8550281Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ignored_output 2024-12-18T02:27:14.8551618Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters 2024-12-18T02:27:14.8552741Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_invalid_powerSGD_state 2024-12-18T02:27:14.8553788Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_save_load_checkpoint 2024-12-18T02:27:14.8554809Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sparse_gradients 2024-12-18T02:27:14.8555877Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view 2024-12-18T02:27:14.8557006Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2024-12-18T02:27:14.8558271Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input 2024-12-18T02:27:14.8559439Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_1gpu_module_device_ids_integer_list 2024-12-18T02:27:14.8560664Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_1gpu_module_device_ids_torch_device_list 2024-12-18T02:27:14.8561804Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_2gpu_module 2024-12-18T02:27:14.8562816Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_4gpu_module 2024-12-18T02:27:14.8563820Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_cpu_module 2024-12-18T02:27:14.8564944Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_cpu_module_grad_is_view 2024-12-18T02:27:14.8565972Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_incremented_ucc_default 2024-12-18T02:27:14.8566901Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_incremented_ucc_subgroup 2024-12-18T02:27:14.8567814Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_set_default_pg_ucc 2024-12-18T02:27:14.8568690Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_set_ucc_new_group 2024-12-18T02:27:14.8569760Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_tensor_dtype_complex 2024-12-18T02:27:14.8570635Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_tensor_dtype_mismatch 2024-12-18T02:27:14.8571520Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_ucc_barrier_device_ids 2024-12-18T02:27:14.8572390Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_ucc_rank_membership 2024-12-18T02:27:14.8573260Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_ucc_warn_not_in_group 2024-12-18T02:27:14.8574292Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_allgather_base 2024-12-18T02:27:14.8575490Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_collectives 2024-12-18T02:27:14.8576781Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-12-18T02:27:14.8578197Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-12-18T02:27:14.8579296Z 2024-12-18T02:27:14.8579703Z Running distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 ... [2024-12-18 02:27:14.845400] 2024-12-18T02:27:14.8580378Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:27:14.8581766Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:27:14.845953] 2024-12-18T02:27:58.6743370Z 2024-12-18T02:27:58.6744787Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_dict_1.1_e60ebf6b17165b9c_.log 2024-12-18T02:27:58.6750024Z Running 6 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_2d_state_dict_correctness, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_dp_state_dict_cpu_offload, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_dp_state_dict_save_load, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_dp_tp_state_dict_save_load, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_hsdp_tp_state_dict_save_load, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiThread::test_rank0_offload_full_state_dict 2024-12-18T02:27:58.6754102Z 2024-12-18T02:27:58.6754377Z Running distributed/test_pg_wrapper 1/1 ... [2024-12-18 02:27:58.674628] 2024-12-18T02:27:58.6754903Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:27:58.6756190Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_pg_wrapper.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:27:58.674985] 2024-12-18T02:30:54.4101473Z 2024-12-18T02:30:54.4102622Z distributed/test_pg_wrapper 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_pg_wrapper_1.1_b79301e4e7822c25_.log 2024-12-18T02:30:54.4112677Z Running 17 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_coalescing_manager_debug_mode_detail, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_hang, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_detail, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_off, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_debug_level_detail_no_gloo, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_new_group_no_gloo, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_hang, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode_off, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_debug_mode 2024-12-18T02:30:54.4122296Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_coalescing_manager_debug_mode_detail 2024-12-18T02:30:54.4123477Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_hang 2024-12-18T02:30:54.4124667Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_detail 2024-12-18T02:30:54.4125994Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_off 2024-12-18T02:30:54.4127224Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch 2024-12-18T02:30:54.4128442Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch_debug_mode 2024-12-18T02:30:54.4129667Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_debug_level_detail_no_gloo 2024-12-18T02:30:54.4130792Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_new_group_no_gloo 2024-12-18T02:30:54.4132245Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_hang 2024-12-18T02:30:54.4133424Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda 2024-12-18T02:30:54.4134749Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda_debug_mode 2024-12-18T02:30:54.4136292Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode 2024-12-18T02:30:54.4137633Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode_off 2024-12-18T02:30:54.4138983Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch 2024-12-18T02:30:54.4140203Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda 2024-12-18T02:30:54.4141493Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda_debug_mode 2024-12-18T02:30:54.4142816Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_debug_mode 2024-12-18T02:30:54.4143531Z 2024-12-18T02:30:54.4143949Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-12-18 02:30:54.410445] 2024-12-18T02:30:54.4144613Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:30:54.4146001Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:30:54.410814] 2024-12-18T02:37:12.5559072Z 2024-12-18T02:37:12.5560421Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_61c430d99580d69c_.log 2024-12-18T02:37:12.5700865Z Running 24 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::test_compiled_fsdp, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardForwardInputs::test_root_move_forward_input_to_device, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardRegisteredParams::test_param_registration_after_backward, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardRegisteredParams::test_param_registration_after_forward, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardCastAfterInit::test_to_float64_after_init, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_explicit_prefetching, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_multi_forward_module, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_non_root_forward_backward, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_post_optim_event, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group_cpu_offload_eager, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group_unshard_async_op, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_single_group_shard_dim0, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_single_group_shard_largest_dim, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCompose::test_train_parity_with_activation_checkpointing, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardShardPlacementFnMultiProcess::test_train_parity_shard_placement_fn_shard_largest_dim, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardShardPlacementFnMultiThread::test_shard_placement_fn_contiguous_params_grads, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardSharedParams::test_train_parity_with_shared_params, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardGradientAccumulation::test_1f1b_microbatching, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardGradientAccumulation::test_gradient_accumulation, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardNDTraining::test_2d_mlp_with_nd_mesh, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardHSDP3DTraining::test_3d_mlp_with_nd_mesh, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardHSDPTraining::test_train_parity_hsdp, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardCustomForwardMethod::test_register_fsdp_forward_method 2024-12-18T02:37:12.6229738Z 2024-12-18T02:37:12.6239354Z Running distributed/rpc/test_tensorpipe_agent 1/1 ... [2024-12-18 02:37:12.623795] 2024-12-18T02:37:12.6239980Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:37:12.6243726Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/rpc/test_tensorpipe_agent.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:37:12.624170] 2024-12-18T02:37:16.9323066Z 2024-12-18T02:37:16.9324251Z distributed/rpc/test_tensorpipe_agent 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.test_tensorpipe_agent_1.1_43db6207905a34e1_.log 2024-12-18T02:37:16.9325352Z Running 0 items in this shard: 2024-12-18T02:37:16.9325591Z 2024-12-18T02:37:16.9325970Z Running distributed/_shard/sharding_plan/test_sharding_plan 1/1 ... [2024-12-18 02:37:16.932330] 2024-12-18T02:37:16.9326604Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:37:16.9329380Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:37:16.932683] 2024-12-18T02:37:41.4619668Z 2024-12-18T02:37:41.4622684Z distributed/_shard/sharding_plan/test_sharding_plan 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharding_plan.test_sharding_plan_1.1_3c5bc94401dbff11_.log 2024-12-18T02:37:41.4625558Z Running 3 items in this shard: test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_custom_sharding_planner, test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_shard_module_sub_process_group, test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_sharding_plan_errors 2024-12-18T02:37:41.4627349Z 2024-12-18T02:37:41.4627690Z Running distributed/fsdp/test_fsdp_multiple_forward 1/1 ... [2024-12-18 02:37:41.461891] 2024-12-18T02:37:41.4628359Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:37:41.4629705Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:37:41.462272] 2024-12-18T02:37:45.8379082Z 2024-12-18T02:37:45.8380872Z distributed/fsdp/test_fsdp_multiple_forward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_multiple_forward_1.1_d9bb5da17296518f_.log 2024-12-18T02:37:45.8382025Z Running 0 items in this shard: 2024-12-18T02:37:45.8382272Z 2024-12-18T02:37:45.8382633Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 ... [2024-12-18 02:37:45.837882] 2024-12-18T02:37:45.8383265Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:37:45.8384922Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:37:45.838280] 2024-12-18T02:40:01.3954072Z 2024-12-18T02:40:01.3955408Z distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_sharded_grad_scaler_1.1_c9a8c18052d0670e_.log 2024-12-18T02:40:01.3972057Z Running 20 items in this shard: test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_grad_scaling, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_inf_gradients_skip_optim_step, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_scaling_unscaling_sparse, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_sharded_grad_scaler_found_inf 2024-12-18T02:40:01.3987935Z 2024-12-18T02:40:01.3988229Z Running distributed/fsdp/test_fsdp_meta 1/1 ... [2024-12-18 02:40:01.395562] 2024-12-18T02:40:01.3988763Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:40:01.3989963Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:40:01.395923] 2024-12-18T02:40:51.3325437Z 2024-12-18T02:40:51.3326674Z distributed/fsdp/test_fsdp_meta 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_meta_1.1_3de85f30b1f9093a_.log 2024-12-18T02:40:51.3337619Z Running 15 items in this shard: test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_bad_arg_meta, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_bad_arg_torchdistx, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_meta_device_with_mixed_precision, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_default_init_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_default_init_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_reset_params_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_reset_params_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_default_init_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_default_init_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_init_fn_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_init_fn_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_meta_device_default_init, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_meta_device_reset_params, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_torchdistX_default_init, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_torchdistX_init_fn 2024-12-18T02:40:51.3347510Z 2024-12-18T02:40:51.3347883Z Running distributed/_shard/sharding_spec/test_sharding_spec 1/1 ... [2024-12-18 02:40:51.332598] 2024-12-18T02:40:51.3348570Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:40:51.3349992Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:40:51.332959] 2024-12-18T02:41:21.9263658Z 2024-12-18T02:41:21.9265153Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharding_spec.test_sharding_spec_1.1_8f7a44c1b4ab7eaa_.log 2024-12-18T02:41:21.9272038Z Running 11 items in this shard: test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_check_overlapping, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_chunked_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_device_placement, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_enumerable_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_chunk_sharding_params, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_chunked_dim_size, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_split_size, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_infer_sharding_spec_from_shards_metadata, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec_shard_tensor, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec_tensor_ctor 2024-12-18T02:41:21.9278559Z 2024-12-18T02:41:21.9278980Z Running distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 ... [2024-12-18 02:41:21.926683] 2024-12-18T02:41:21.9279630Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:41:21.9280966Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:41:21.927049] 2024-12-18T02:41:39.6364591Z 2024-12-18T02:41:39.6366094Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_reshard_1.1_476e9ada9fb215ff_.log 2024-12-18T02:41:39.6368429Z Running 2 items in this shard: test/distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py::TestReshard::test_sharded_tensor_reshard, test/distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py::TestReshard::test_sharded_tensor_reshard_errors 2024-12-18T02:41:39.6369686Z 2024-12-18T02:41:39.6369948Z Running distributed/test_launcher 1/1 ... [2024-12-18 02:41:39.636522] 2024-12-18T02:41:39.6370463Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:41:39.6371681Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_launcher.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:41:39.636883] 2024-12-18T02:41:44.9656591Z 2024-12-18T02:41:44.9657989Z distributed/test_launcher 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_launcher_1.1_683a3b9ed0b0675c_.log 2024-12-18T02:41:44.9659537Z Running 1 items in this shard: test/distributed/test_launcher.py::TestDistributedLaunch::test_launch_user_script 2024-12-18T02:41:44.9660151Z 2024-12-18T02:41:44.9660547Z Running distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 ... [2024-12-18 02:41:44.965792] 2024-12-18T02:41:44.9661208Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:41:44.9663833Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:41:44.966184] 2024-12-18T02:51:29.6350110Z 2024-12-18T02:51:29.6352261Z distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_1.1_e1f7a6721f856577_.log 2024-12-18T02:51:29.6394531Z Running 69 items in this shard: test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorMetadata::test_serialize_and_deserialize, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestCreateTensorFromParams::test_empty, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardParameter::test_shard_parameter, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardParameter::test_shard_parameter_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardTensor::test_shard_tensor, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardTensor::test_shard_tensor_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardTensor::test_shard_tensor_with_empty_shard, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestModuleHookApi::test_collect_local_shard, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestModuleHookApi::test_reshard_output, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestLocalTensor::test_local_tensor, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestLocalTensor::test_local_tensor_error, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_cleanup, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_complete_world_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_like, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_full, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_ones, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_rand, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_zeros, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_gather_even, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_gather_uneven, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_insufficient_sharding_dims, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_invalid_pg_rpc_ranks, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_invalid_sharding, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_load_state_dict_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_multiple_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_partial_world_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_sharded_tensor_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_sharded_tensor_sizes, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_sharding_columns, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_state_dict, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_state_dict_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_state_dict_no_sharded_tensors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_create_sharded_tensor_with_ones, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_gather_even, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_gather_uneven, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_grid_sharding, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_multiple_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_partial_world_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_device, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_to_cpu, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_to_cuda, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_to_test, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_uneven_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_with_rpc_names, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalTensor::test_init_from_local_tensor, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalTensor::test_init_from_local_tensor_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_and_global_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_and_global_metadata_invalid_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_pin_memory, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_property_cross_ranks, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_shards_gaps, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_shards_overlap, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_st_base_init_from_local_shards_and_global_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorCustomOps::test_custom_op, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorCustomOps::test_custom_op_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorCustomOps::test_custom_op_override, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardMetadata::test_create_shard_with_no_placement, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardMetadata::test_shard_metadata_init, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorSubGroupInit::test_sub_process_group_placement_validation, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorSubGroupInit::test_sub_process_group_sharded_tensor_init, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestCreateTensorNoProcessGroupMode::test_init_from_local_shards_and_global_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestCreateTensorNoProcessGroupMode::test_non_contiguous_local_shards 2024-12-18T02:51:29.6434640Z 2024-12-18T02:51:29.6434950Z Running distributed/elastic/events/lib_test 1/1 ... [2024-12-18 02:51:29.636548] 2024-12-18T02:51:29.6435523Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:51:29.6436803Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/events/lib_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:51:29.636908] 2024-12-18T02:51:34.0128305Z 2024-12-18T02:51:34.0130412Z distributed/elastic/events/lib_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.events.lib_test_1.1_7f3ec8b7d8dff807_.log 2024-12-18T02:51:34.0135811Z Running 8 items in this shard: test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_created, test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser, test/distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str 2024-12-18T02:51:34.0139659Z 2024-12-18T02:51:34.0139996Z Running distributed/elastic/metrics/api_test 1/1 ... [2024-12-18 02:51:34.012787] 2024-12-18T02:51:34.0140602Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:51:34.0141925Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:51:34.013209] 2024-12-18T02:51:38.2887082Z 2024-12-18T02:51:38.2888276Z distributed/elastic/metrics/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.metrics.api_test_1.1_4fc1641e259f9755_.log 2024-12-18T02:51:38.2890491Z Running 3 items in this shard: test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_get_metric_name, test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_inheritance, test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_profile 2024-12-18T02:51:38.2891880Z 2024-12-18T02:51:38.2892251Z Running distributed/elastic/multiprocessing/api_test 1/1 ... [2024-12-18 02:51:38.288662] 2024-12-18T02:51:38.2893250Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:51:38.2894681Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:51:38.289024] 2024-12-18T02:52:16.4512326Z 2024-12-18T02:52:16.4513725Z distributed/elastic/multiprocessing/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.multiprocessing.api_test_1.1_4063b1564aa229b1_.log 2024-12-18T02:52:16.4529262Z Running 25 items in this shard: test/distributed/elastic/multiprocessing/api_test.py::RunProcResultsTest::test_get_failures, test/distributed/elastic/multiprocessing/api_test.py::RunProcResultsTest::test_is_failed, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_str_bad_input, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_value, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_value_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_args_env_len_mismatch, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_function_large_ret_val, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_function_raise, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_function_with_tensor, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_invalid_log_dir, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_multiprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_multiprocessing_context_poll_raises_exception, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_pcontext_wait, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_pcontext_wait_on_a_child_thread, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_to_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_void_function, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_wait_for_all_child_procs_to_exit, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_binary_exit, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_binary_incorrect_entrypoint, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_binary_raises, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_subprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_validate_full_rank, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsFuncTest::test_function, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsBinaryTest::test_binary, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsBinaryTest::test_binary_redirect_and_tee 2024-12-18T02:52:16.4543974Z 2024-12-18T02:52:16.4544453Z Running distributed/elastic/timer/local_timer_test 1/1 ... [2024-12-18 02:52:16.450992] 2024-12-18T02:52:16.4545064Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:52:16.4546371Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/timer/local_timer_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:52:16.451353] 2024-12-18T02:52:26.4424477Z 2024-12-18T02:52:26.4425948Z distributed/elastic/timer/local_timer_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.local_timer_test_1.1_0759355597f56ff3_.log 2024-12-18T02:52:26.4434327Z Running 14 items in this shard: test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_client_interaction, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_exception_propagation, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_get_timer_recursive, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_happy_path, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_no_client, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_timer, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get_less_than_size, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get_size, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_acquire_release, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_expired_timers, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_valid_timers, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_watchdog_call_count, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_watchdog_empty_queue 2024-12-18T02:52:26.4441585Z 2024-12-18T02:52:26.4441913Z Running distributed/elastic/utils/logging_test 1/1 ... [2024-12-18 02:52:26.442660] 2024-12-18T02:52:26.4442512Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:52:26.4443880Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:52:26.443018] 2024-12-18T02:52:30.8190969Z 2024-12-18T02:52:30.8192434Z distributed/elastic/utils/logging_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.logging_test_1.1_07147dee7fcfe789_.log 2024-12-18T02:52:30.8196318Z Running 2 items in this shard: test/distributed/elastic/utils/logging_test.py::LoggingTest::test_derive_module_name, test/distributed/elastic/utils/logging_test.py::LoggingTest::test_logger_name 2024-12-18T02:52:30.8197297Z 2024-12-18T02:52:30.8197605Z Running distributed/elastic/utils/util_test 1/1 ... [2024-12-18 02:52:30.818955] 2024-12-18T02:52:30.8198173Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:52:30.8199445Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/utils/util_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:52:30.819313] 2024-12-18T02:52:35.2954484Z 2024-12-18T02:52:35.2955918Z distributed/elastic/utils/util_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.util_test_1.1_9f848f2c1fc1943c_.log 2024-12-18T02:52:35.2961952Z Running 12 items in this shard: test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier_hash_store, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier_timeout_operations, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier_timeout_rank_tracing, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_get_all_rank_0, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_get_all_rank_n, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_synchronize, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_synchronize_hash_store, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_custom_name, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_different, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_none 2024-12-18T02:52:35.2967022Z 2024-12-18T02:52:36.0434688Z Running test batch 'tests to run' cost 9841.24 seconds 2024-12-18T02:52:36.8466319Z 2024-12-18T02:52:36.8466804Z real 164m7.388s 2024-12-18T02:52:36.8467158Z user 388m3.775s 2024-12-18T02:52:36.8467445Z sys 170m11.379s 2024-12-18T02:52:36.8467718Z + assert_git_not_dirty 2024-12-18T02:52:36.8468109Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-12-18T02:52:36.8468589Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-12-18T02:52:36.8471489Z ++ git status --porcelain 2024-12-18T02:52:36.8472539Z ++ grep -v '?? third_party' 2024-12-18T02:52:39.4545239Z ++ true 2024-12-18T02:52:39.4545555Z + git_status= 2024-12-18T02:52:39.4545854Z + [[ -n '' ]] 2024-12-18T02:52:39.4546184Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-12-18T02:52:39.4546619Z + [[ 1 == 1 ]] 2024-12-18T02:52:39.4546919Z + echo 'Testing distributed C++ tests' 2024-12-18T02:52:39.4547360Z Testing distributed C++ tests 2024-12-18T02:52:39.4550737Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T02:52:39.4566982Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T02:52:39.4576344Z + export CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T02:52:39.4577136Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T02:52:39.4577781Z + python test/run_test.py --cpp --verbose -i cpp/FileStoreTest 2024-12-18T02:52:39.5623731Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T02:52:39.5624764Z import pkg_resources 2024-12-18T02:52:44.5934180Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T02:52:44.6098887Z Found test times from artifacts 2024-12-18T02:52:44.6742097Z Found test times from artifacts 2024-12-18T02:52:44.6766266Z Running all tests 2024-12-18T02:52:44.6766727Z Running parallel tests on 2 processes 2024-12-18T02:52:44.6767152Z Name: tests to run (est. time: 0.0min) 2024-12-18T02:52:44.6767519Z Serial tests (1): 2024-12-18T02:52:44.6767823Z cpp/FileStoreTest 1/1 2024-12-18T02:52:44.6768148Z Parallel tests (0): 2024-12-18T02:52:44.6768455Z Name: excluded (est. time: 0.0min) 2024-12-18T02:52:44.6768801Z Serial tests (0): 2024-12-18T02:52:44.6769091Z Parallel tests (0): 2024-12-18T02:52:44.6822867Z Running cpp/FileStoreTest 1/1 ... [2024-12-18 02:52:44.682063] 2024-12-18T02:52:44.6823399Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:52:44.6828140Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/FileStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-94a644091be1b736.xml', '-x', '--reruns=2'] ... [2024-12-18 02:52:44.682613] 2024-12-18T02:52:47.9556857Z 2024-12-18T02:52:47.9558144Z cpp/FileStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.FileStoreTest_1.1_5bb18050eeb37473_.log 2024-12-18T02:52:47.9558960Z 2024-12-18T02:52:49.1853672Z Running test batch 'tests to run' cost 4.51 seconds 2024-12-18T02:52:49.9761611Z + python test/run_test.py --cpp --verbose -i cpp/HashStoreTest 2024-12-18T02:52:50.0821368Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T02:52:50.0822393Z import pkg_resources 2024-12-18T02:52:55.1066941Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T02:52:55.1230052Z Found test times from artifacts 2024-12-18T02:52:55.1892743Z Found test times from artifacts 2024-12-18T02:52:55.1915835Z Running all tests 2024-12-18T02:52:55.1916200Z Running parallel tests on 2 processes 2024-12-18T02:52:55.1916663Z Name: tests to run (est. time: 0.0min) 2024-12-18T02:52:55.1917031Z Serial tests (1): 2024-12-18T02:52:55.1917335Z cpp/HashStoreTest 1/1 2024-12-18T02:52:55.1917647Z Parallel tests (0): 2024-12-18T02:52:55.1917963Z Name: excluded (est. time: 0.0min) 2024-12-18T02:52:55.1918307Z Serial tests (0): 2024-12-18T02:52:55.1918593Z Parallel tests (0): 2024-12-18T02:52:55.1974197Z Running cpp/HashStoreTest 1/1 ... [2024-12-18 02:52:55.196807] 2024-12-18T02:52:55.1974746Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:52:55.1976141Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/HashStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-411d9464b2ee0b30.xml', '-x', '--reruns=2'] ... [2024-12-18 02:52:55.197288] 2024-12-18T02:52:58.2199762Z 2024-12-18T02:52:58.2200861Z cpp/HashStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.HashStoreTest_1.1_755c1b147766363d_.log 2024-12-18T02:52:58.2203200Z 2024-12-18T02:52:59.6736614Z Running test batch 'tests to run' cost 4.48 seconds 2024-12-18T02:53:00.4615998Z + python test/run_test.py --cpp --verbose -i cpp/TCPStoreTest 2024-12-18T02:53:00.5680947Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T02:53:00.5681974Z import pkg_resources 2024-12-18T02:53:05.5776859Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T02:53:05.5937330Z Found test times from artifacts 2024-12-18T02:53:05.6582185Z Found test times from artifacts 2024-12-18T02:53:05.6606604Z Running all tests 2024-12-18T02:53:05.6607113Z Running parallel tests on 2 processes 2024-12-18T02:53:05.6607726Z Name: tests to run (est. time: 0.0min) 2024-12-18T02:53:05.6608343Z Serial tests (1): 2024-12-18T02:53:05.6608809Z cpp/TCPStoreTest 1/1 2024-12-18T02:53:05.6609289Z Parallel tests (0): 2024-12-18T02:53:05.6609782Z Name: excluded (est. time: 0.0min) 2024-12-18T02:53:05.6610319Z Serial tests (0): 2024-12-18T02:53:05.6610765Z Parallel tests (0): 2024-12-18T02:53:05.6663615Z Running cpp/TCPStoreTest 1/1 ... [2024-12-18 02:53:05.666136] 2024-12-18T02:53:05.6664400Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:53:05.6670658Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/TCPStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-d7d7cf3339845ea1.xml', '-x', '--reruns=2'] ... [2024-12-18 02:53:05.666726] 2024-12-18T02:53:20.2356888Z 2024-12-18T02:53:20.2357923Z cpp/TCPStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.TCPStoreTest_1.1_03bfdc91bd30b668_.log 2024-12-18T02:53:20.2358694Z 2024-12-18T02:53:20.9935759Z Running test batch 'tests to run' cost 15.33 seconds 2024-12-18T02:53:21.7702866Z + echo 'Testing multi-GPU linalg tests' 2024-12-18T02:53:21.7703354Z Testing multi-GPU linalg tests 2024-12-18T02:53:21.7703949Z + python test/run_test.py -i test_linalg.py -k test_matmul_offline_mgpu_tunable --verbose 2024-12-18T02:53:21.8762149Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T02:53:21.8763146Z import pkg_resources 2024-12-18T02:53:26.9083749Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T02:53:26.9250253Z Found test times from artifacts 2024-12-18T02:53:26.9897745Z Found test times from artifacts 2024-12-18T02:53:26.9919550Z Running all tests 2024-12-18T02:53:26.9919942Z Running parallel tests on 2 processes 2024-12-18T02:53:26.9920376Z Name: tests to run (est. time: 0.01min) 2024-12-18T02:53:26.9920767Z Serial tests (1): 2024-12-18T02:53:26.9921046Z test_linalg 1/1 2024-12-18T02:53:26.9921333Z Parallel tests (0): 2024-12-18T02:53:26.9921646Z Name: excluded (est. time: 0.0min) 2024-12-18T02:53:26.9921989Z Serial tests (0): 2024-12-18T02:53:26.9922262Z Parallel tests (0): 2024-12-18T02:53:26.9976812Z Running test_linalg 1/1 ... [2024-12-18 02:53:26.997053] 2024-12-18T02:53:26.9977287Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:53:26.9978596Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_linalg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_matmul_offline_mgpu_tunable', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:53:26.997403] 2024-12-18T02:53:33.4285125Z 2024-12-18T02:53:33.4286157Z test_linalg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_linalg_1.1_dfddd869d1cd41ca_.log 2024-12-18T02:53:33.4287736Z Running 1 items in this shard: test/test_linalg.py::TestLinalgCUDA::test_matmul_offline_mgpu_tunableop_cuda_float32 2024-12-18T02:53:33.4288324Z 2024-12-18T02:53:34.1618206Z Running test batch 'tests to run' cost 7.17 seconds 2024-12-18T02:53:34.8962034Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-12-18T02:53:34.8962521Z ++ command -v mpiexec 2024-12-18T02:53:34.8964227Z + MPIEXEC=/usr/bin/mpiexec 2024-12-18T02:53:34.8964576Z + [[ -n /usr/bin/mpiexec ]] 2024-12-18T02:53:34.8965281Z + MPICMD='/usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest' 2024-12-18T02:53:34.8966336Z + eval '/usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest' 2024-12-18T02:53:34.8967331Z ++ /usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest 2024-12-18T02:53:37.0048083Z Test successful 2024-12-18T02:53:37.0095738Z Test successful 2024-12-18T02:53:37.1548901Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupGlooTest 2024-12-18T02:53:37.2603529Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T02:53:37.2604559Z import pkg_resources 2024-12-18T02:53:42.3484143Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T02:53:42.3642980Z Found test times from artifacts 2024-12-18T02:53:42.4293645Z Found test times from artifacts 2024-12-18T02:53:42.4316461Z Running all tests 2024-12-18T02:53:42.4316903Z Running parallel tests on 2 processes 2024-12-18T02:53:42.4317311Z Name: tests to run (est. time: 0.0min) 2024-12-18T02:53:42.4317675Z Serial tests (1): 2024-12-18T02:53:42.4317995Z cpp/ProcessGroupGlooTest 1/1 2024-12-18T02:53:42.4318379Z Parallel tests (0): 2024-12-18T02:53:42.4318968Z Name: excluded (est. time: 0.0min) 2024-12-18T02:53:42.4319326Z Serial tests (0): 2024-12-18T02:53:42.4319617Z Parallel tests (0): 2024-12-18T02:53:42.4374440Z Running cpp/ProcessGroupGlooTest 1/1 ... [2024-12-18 02:53:42.436783] 2024-12-18T02:53:42.4375002Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:53:42.4376453Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupGlooTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-9cbaaf0430380c7f.xml', '-x', '--reruns=2'] ... [2024-12-18 02:53:42.437266] 2024-12-18T02:53:50.6240034Z 2024-12-18T02:53:50.6241146Z cpp/ProcessGroupGlooTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupGlooTest_1.1_028ec5f62f95aca9_.log 2024-12-18T02:53:50.6241974Z 2024-12-18T02:53:51.3765303Z Running test batch 'tests to run' cost 8.94 seconds 2024-12-18T02:53:52.1740210Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupNCCLTest 2024-12-18T02:53:52.2798578Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T02:53:52.2799592Z import pkg_resources 2024-12-18T02:53:57.3363766Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T02:53:57.3526250Z Found test times from artifacts 2024-12-18T02:53:57.4171738Z Found test times from artifacts 2024-12-18T02:53:57.4189869Z Running all tests 2024-12-18T02:53:57.4192664Z Running parallel tests on 2 processes 2024-12-18T02:53:57.4193546Z Name: tests to run (est. time: 0.0min) 2024-12-18T02:53:57.4193992Z Serial tests (1): 2024-12-18T02:53:57.4194319Z cpp/ProcessGroupNCCLTest 1/1 2024-12-18T02:53:57.4194679Z Parallel tests (0): 2024-12-18T02:53:57.4195349Z Name: excluded (est. time: 0.0min) 2024-12-18T02:53:57.4195708Z Serial tests (0): 2024-12-18T02:53:57.4196002Z Parallel tests (0): 2024-12-18T02:53:57.4251258Z Running cpp/ProcessGroupNCCLTest 1/1 ... [2024-12-18 02:53:57.424520] 2024-12-18T02:53:57.4251813Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:53:57.4253264Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-d59ce55faeb414d5.xml', '-x', '--reruns=2'] ... [2024-12-18 02:53:57.424996] 2024-12-18T02:54:17.5902467Z 2024-12-18T02:54:17.5905845Z cpp/ProcessGroupNCCLTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupNCCLTest_1.1_c642d57b62fef2a5_.log 2024-12-18T02:54:17.5906795Z 2024-12-18T02:54:18.3438381Z Running test batch 'tests to run' cost 20.92 seconds 2024-12-18T02:54:19.1140950Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupNCCLErrorsTest 2024-12-18T02:54:19.2205898Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T02:54:19.2206936Z import pkg_resources 2024-12-18T02:54:24.2645325Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T02:54:24.2808293Z Found test times from artifacts 2024-12-18T02:54:24.3453141Z Found test times from artifacts 2024-12-18T02:54:24.3475921Z Running all tests 2024-12-18T02:54:24.3476423Z Running parallel tests on 2 processes 2024-12-18T02:54:24.3477213Z Name: tests to run (est. time: 0.0min) 2024-12-18T02:54:24.3477651Z Serial tests (1): 2024-12-18T02:54:24.3477965Z cpp/ProcessGroupNCCLErrorsTest 1/1 2024-12-18T02:54:24.3478350Z Parallel tests (0): 2024-12-18T02:54:24.3478664Z Name: excluded (est. time: 0.0min) 2024-12-18T02:54:24.3479040Z Serial tests (0): 2024-12-18T02:54:24.3479592Z Parallel tests (0): 2024-12-18T02:54:24.3534135Z Running cpp/ProcessGroupNCCLErrorsTest 1/1 ... [2024-12-18 02:54:24.352893] 2024-12-18T02:54:24.3534739Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:54:24.3536237Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLErrorsTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-f129af15f9ce5703.xml', '-x', '--reruns=2'] ... [2024-12-18 02:54:24.353399] 2024-12-18T02:54:37.5054586Z 2024-12-18T02:54:37.5055753Z cpp/ProcessGroupNCCLErrorsTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupNCCLErrorsTest_1.1_8a0a9aea239df6ef_.log 2024-12-18T02:54:37.5056665Z 2024-12-18T02:54:38.2607839Z Running test batch 'tests to run' cost 13.91 seconds 2024-12-18T02:54:39.0656661Z + [[ 1 == 1 ]] 2024-12-18T02:54:39.0657051Z + test_rpc 2024-12-18T02:54:39.0657368Z + echo 'Testing RPC C++ tests' 2024-12-18T02:54:39.0657717Z Testing RPC C++ tests 2024-12-18T02:54:39.0661275Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T02:54:39.0676190Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T02:54:39.0688440Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T02:54:39.0689173Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc 2024-12-18T02:54:39.1742152Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T02:54:39.1743191Z import pkg_resources 2024-12-18T02:54:44.2161136Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T02:54:44.2324254Z Found test times from artifacts 2024-12-18T02:54:44.2968019Z Found test times from artifacts 2024-12-18T02:54:44.2986182Z Running all tests 2024-12-18T02:54:44.2988757Z Running parallel tests on 2 processes 2024-12-18T02:54:44.2989621Z Name: tests to run (est. time: 0.0min) 2024-12-18T02:54:44.2990071Z Serial tests (1): 2024-12-18T02:54:44.2990381Z cpp/test_cpp_rpc 1/1 2024-12-18T02:54:44.2990708Z Parallel tests (0): 2024-12-18T02:54:44.2991015Z Name: excluded (est. time: 0.0min) 2024-12-18T02:54:44.2991480Z Serial tests (0): 2024-12-18T02:54:44.2991767Z Parallel tests (0): 2024-12-18T02:54:44.3042396Z Running cpp/test_cpp_rpc 1/1 ... [2024-12-18 02:54:44.304037] 2024-12-18T02:54:44.3042895Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:54:44.3047237Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-7bdc5814439d9464.xml', '-x', '--reruns=2'] ... [2024-12-18 02:54:44.304520] 2024-12-18T02:54:50.9367827Z 2024-12-18T02:54:50.9368768Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_0a260d833ffb8314_.log 2024-12-18T02:54:50.9371997Z 2024-12-18T02:54:51.6949983Z Running test batch 'tests to run' cost 7.4 seconds 2024-12-18T02:54:52.4867852Z + cleanup_workspace 2024-12-18T02:54:52.4868572Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-12-18T02:54:52.4869506Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-12-18T02:54:52.4870261Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-12-18T02:54:52.4870809Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-12-18T02:54:52.4871569Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-12-18T02:54:52.4872256Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-12-18T02:54:52.4872814Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-12-18T02:54:52.9862212Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@release/2.6 2024-12-18T02:54:52.9862837Z with: 2024-12-18T02:54:52.9863139Z benchmark-results-dir: test/test-reports 2024-12-18T02:54:52.9863532Z dry-run: false 2024-12-18T02:54:52.9863807Z schema-version: v3 2024-12-18T02:54:52.9864326Z github-token: *** 2024-12-18T02:54:52.9864612Z env: 2024-12-18T02:54:52.9864874Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:54:52.9865278Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:54:52.9865923Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:54:52.9866510Z ##[endgroup] 2024-12-18T02:54:52.9906119Z ##[group]Run set -eux 2024-12-18T02:54:52.9906448Z set -eux 2024-12-18T02:54:52.9906781Z python3 -mpip install boto3==1.35.33 2024-12-18T02:54:52.9916289Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:54:52.9916683Z env: 2024-12-18T02:54:52.9916977Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:54:52.9917345Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:54:52.9918064Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:54:52.9918589Z ##[endgroup] 2024-12-18T02:54:52.9949415Z + python3 -mpip install boto3==1.35.33 2024-12-18T02:54:53.2528935Z Defaulting to user installation because normal site-packages is not writeable 2024-12-18T02:54:54.4157238Z Collecting boto3==1.35.33 2024-12-18T02:54:54.4655412Z Downloading boto3-1.35.33-py3-none-any.whl (139 kB) 2024-12-18T02:54:54.5340230Z Collecting s3transfer<0.11.0,>=0.10.0 2024-12-18T02:54:54.5404765Z Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB) 2024-12-18T02:54:55.8090483Z Collecting botocore<1.36.0,>=1.35.33 2024-12-18T02:54:55.8135834Z Downloading botocore-1.35.83-py3-none-any.whl (13.3 MB) 2024-12-18T02:54:56.0196999Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0) 2024-12-18T02:54:56.0262865Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10) 2024-12-18T02:54:56.0267262Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1) 2024-12-18T02:54:56.1835029Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0) 2024-12-18T02:54:56.2716464Z Installing collected packages: botocore, s3transfer, boto3 2024-12-18T02:54:56.8692665Z Successfully installed boto3-1.35.33 botocore-1.35.83 s3transfer-0.10.4 2024-12-18T02:54:56.9642078Z ##[group]Run set -eux 2024-12-18T02:54:56.9642412Z set -eux 2024-12-18T02:54:56.9642692Z  2024-12-18T02:54:56.9642985Z if [[ -z "${GITHUB_TOKEN}" ]]; then 2024-12-18T02:54:56.9643541Z  echo "Missing github-token input" 2024-12-18T02:54:56.9643895Z  exit 1 2024-12-18T02:54:56.9644268Z fi 2024-12-18T02:54:56.9651620Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:54:56.9652036Z env: 2024-12-18T02:54:56.9652281Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:54:56.9652645Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:54:56.9653238Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:54:56.9654038Z GITHUB_TOKEN: *** 2024-12-18T02:54:56.9654289Z ##[endgroup] 2024-12-18T02:54:56.9682999Z + [[ -z *** ]] 2024-12-18T02:54:56.9744224Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main 2024-12-18T02:54:56.9744745Z with: 2024-12-18T02:54:56.9745165Z github-token: *** 2024-12-18T02:54:56.9745452Z env: 2024-12-18T02:54:56.9745713Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:54:56.9746109Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:54:56.9746991Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:54:56.9747596Z ##[endgroup] 2024-12-18T02:54:56.9774552Z ##[group]Run set -eux 2024-12-18T02:54:56.9774843Z set -eux 2024-12-18T02:54:56.9775092Z  2024-12-18T02:54:56.9775591Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-12-18T02:54:56.9782009Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:54:56.9782465Z env: 2024-12-18T02:54:56.9782730Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:54:56.9783133Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:54:56.9783791Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:54:56.9784619Z GITHUB_TOKEN: *** 2024-12-18T02:54:56.9784898Z ##[endgroup] 2024-12-18T02:54:56.9809956Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 12383255652 i-0d4c5ae1b54bc839d 2024-12-18T02:55:00.4814086Z setting job-id=34566462667 2024-12-18T02:55:00.4815027Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T02:55:00.4923515Z ##[group]Run set -eux 2024-12-18T02:55:00.4923843Z set -eux 2024-12-18T02:55:00.4924107Z  2024-12-18T02:55:00.4924536Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \ 2024-12-18T02:55:00.4925267Z  --schema-version "${SCHEMA_VERSION}" \ 2024-12-18T02:55:00.4925670Z  --repo "${REPO}" \ 2024-12-18T02:55:00.4926023Z  --head-branch "${HEAD_BRANCH}" \ 2024-12-18T02:55:00.4926407Z  --head-sha "${HEAD_SHA}" \ 2024-12-18T02:55:00.4926791Z  --workflow-id "${WORKFLOW_RUN_ID}" \ 2024-12-18T02:55:00.4927184Z  --run-attempt "${RUN_ATTEMPT}" \ 2024-12-18T02:55:00.4927575Z  --job-id "${JOB_ID}" \ 2024-12-18T02:55:00.4927923Z  --job-name "${JOB_NAME}" 2024-12-18T02:55:00.4934179Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:55:00.4934623Z env: 2024-12-18T02:55:00.4934886Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:00.4935270Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:00.4935909Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:00.4936484Z SCHEMA_VERSION: v3 2024-12-18T02:55:00.4936782Z REPO: pytorch/pytorch 2024-12-18T02:55:00.4937106Z HEAD_BRANCH: refs/heads/release/2.6 2024-12-18T02:55:00.4937506Z HEAD_SHA: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T02:55:00.4937912Z WORKFLOW_RUN_ID: 12383255652 2024-12-18T02:55:00.4938235Z RUN_ATTEMPT: 1 2024-12-18T02:55:00.4938511Z JOB_ID: 34566462667 2024-12-18T02:55:00.4939330Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T02:55:00.4939979Z ##[endgroup] 2024-12-18T02:55:00.4969343Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/release/2.6/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/release/2.6 --head-sha 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 --workflow-id 12383255652 --run-attempt 1 --job-id 34566462667 --job-name 'linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu)' 2024-12-18T02:55:00.5325074Z ##[group]Run set -eux 2024-12-18T02:55:00.5325374Z set -eux 2024-12-18T02:55:00.5325613Z  2024-12-18T02:55:00.5325888Z # TODO (huydhn): Implement this part 2024-12-18T02:55:00.5326454Z echo "runners=[]" >> "${GITHUB_OUTPUT}" 2024-12-18T02:55:00.5332382Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:55:00.5332892Z env: 2024-12-18T02:55:00.5333147Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:00.5333706Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:00.5334362Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:00.5334931Z ##[endgroup] 2024-12-18T02:55:00.5358414Z + echo 'runners=[]' 2024-12-18T02:55:00.5393279Z ##[group]Run set -eux 2024-12-18T02:55:00.5393591Z set -eux 2024-12-18T02:55:00.5393863Z  2024-12-18T02:55:00.5394149Z # TODO (huydhn): Implement this part 2024-12-18T02:55:00.5394599Z echo "dependencies={}" >> "${GITHUB_OUTPUT}" 2024-12-18T02:55:00.5400057Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:55:00.5400491Z env: 2024-12-18T02:55:00.5400748Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:00.5401135Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:00.5401760Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:00.5402320Z ##[endgroup] 2024-12-18T02:55:00.5424975Z + echo 'dependencies={}' 2024-12-18T02:55:00.5453027Z ##[group]Run set -eux 2024-12-18T02:55:00.5453482Z set -eux 2024-12-18T02:55:00.5453752Z  2024-12-18T02:55:00.5454065Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then 2024-12-18T02:55:00.5454562Z  echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping" 2024-12-18T02:55:00.5455106Z  # We don't want the job to fail if the directory doesn't exist 2024-12-18T02:55:00.5455556Z  exit 0 2024-12-18T02:55:00.5455820Z fi 2024-12-18T02:55:00.5456171Z  2024-12-18T02:55:00.5456433Z if [[ "${DRY_RUN}" == "true" ]]; then 2024-12-18T02:55:00.5456932Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2024-12-18T02:55:00.5457526Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2024-12-18T02:55:00.5457989Z  --metadata "${BENCHMARK_METADATA}" \ 2024-12-18T02:55:00.5458381Z  --runners "${RUNNER_INFO}" \ 2024-12-18T02:55:00.5458859Z  --dependencies "${DEPENDENCIES}" \ 2024-12-18T02:55:00.5459417Z  --dry-run 2024-12-18T02:55:00.5459784Z else 2024-12-18T02:55:00.5460241Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2024-12-18T02:55:00.5460899Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2024-12-18T02:55:00.5461413Z  --metadata "${BENCHMARK_METADATA}" \ 2024-12-18T02:55:00.5461841Z  --runners "${RUNNER_INFO}" \ 2024-12-18T02:55:00.5462258Z  --dependencies "${DEPENDENCIES}" 2024-12-18T02:55:00.5462635Z fi 2024-12-18T02:55:00.5468168Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:55:00.5468649Z env: 2024-12-18T02:55:00.5468918Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:00.5469326Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:00.5469980Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:00.5470613Z BENCHMARK_RESULTS_DIR: test/test-reports 2024-12-18T02:55:00.5471114Z DRY_RUN: false 2024-12-18T02:55:00.5472714Z BENCHMARK_METADATA: {"timestamp": 1734490500, "schema_version": "v3", "name": "linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/release/2.6", "head_sha": "0cdf8b1d09254cfda66191d1bd01e3041c3c76f7", "workflow_id": 12383255652, "run_attempt": 1, "job_id": 34566462667} 2024-12-18T02:55:00.5474244Z RUNNER_INFO: [] 2024-12-18T02:55:00.5474508Z DEPENDENCIES: {} 2024-12-18T02:55:00.5474770Z ##[endgroup] 2024-12-18T02:55:00.5496981Z + [[ ! -d test/test-reports ]] 2024-12-18T02:55:00.5497467Z + [[ false == \t\r\u\e ]] 2024-12-18T02:55:00.5500763Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/release/2.6/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1734490500, "schema_version": "v3", "name": "linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/release/2.6", "head_sha": "0cdf8b1d09254cfda66191d1bd01e3041c3c76f7", "workflow_id": 12383255652, "run_attempt": 1, "job_id": 34566462667}' --runners '[]' --dependencies '{}' 2024-12-18T02:55:00.7278198Z ##[group]Run cat test/**/*_toprint.log || true 2024-12-18T02:55:00.7278610Z cat test/**/*_toprint.log || true 2024-12-18T02:55:00.7284763Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:55:00.7285202Z env: 2024-12-18T02:55:00.7285458Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:00.7285839Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:00.7286460Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:00.7287004Z ##[endgroup] 2024-12-18T02:55:00.7361737Z cat: 'test/**/*_toprint.log': No such file or directory 2024-12-18T02:55:00.7420765Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-12-18T02:55:00.7421331Z kill "$MONITOR_SCRIPT_PID" 2024-12-18T02:55:00.7427146Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:55:00.7427553Z env: 2024-12-18T02:55:00.7427790Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:00.7428151Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:00.7428739Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:00.7429271Z MONITOR_SCRIPT_PID: 58998 2024-12-18T02:55:00.7429563Z ##[endgroup] 2024-12-18T02:55:00.7599533Z Prepare all required actions 2024-12-18T02:55:00.7599993Z Getting action download info 2024-12-18T02:55:00.9367407Z Download action repository 'actions/upload-artifact@v4' (SHA:6f51ac03b9356f520e9adb1b1b7802705f340c2b) 2024-12-18T02:55:01.2663544Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-12-18T02:55:01.2663977Z with: 2024-12-18T02:55:01.2664449Z file-suffix: test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_34566462667 2024-12-18T02:55:01.2665040Z s3-bucket: gha-artifacts 2024-12-18T02:55:01.2665358Z env: 2024-12-18T02:55:01.2665622Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:01.2666027Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:01.2666684Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:01.2667264Z ##[endgroup] 2024-12-18T02:55:01.2702868Z ##[group]Run # Remove any previous test jsons if they exist 2024-12-18T02:55:01.2703483Z # Remove any previous test jsons if they exist 2024-12-18T02:55:01.2703926Z rm -f test-jsons-*.zip 2024-12-18T02:55:01.2704445Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2024-12-18T02:55:01.2710755Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:55:01.2711213Z env: 2024-12-18T02:55:01.2711591Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:01.2711975Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:01.2712596Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:01.2713349Z FILE_SUFFIX: test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_34566462667 2024-12-18T02:55:01.2713886Z ##[endgroup] 2024-12-18T02:55:01.2857267Z adding: test/test-reports/td_exclusions-0b771886e57a976d305c.json (deflated 86%) 2024-12-18T02:55:01.2858247Z adding: test/test-reports/td_exclusions-a913d1b3c6de557a52bc.json (deflated 15%) 2024-12-18T02:55:01.2859125Z adding: test/test-reports/td_exclusions-c10059ff98441f78ef3b.json (deflated 14%) 2024-12-18T02:55:01.2860137Z adding: test/test-reports/td_exclusions-a8b0b759d947714a1c18.json (deflated 14%) 2024-12-18T02:55:01.2860881Z adding: test/test-reports/td_exclusions-f31242d60397c71fec1b.json (deflated 20%) 2024-12-18T02:55:01.2861669Z adding: test/test-reports/td_exclusions-70e3eddaa4dae2edad64.json (deflated 12%) 2024-12-18T02:55:01.2862472Z adding: test/test-reports/td_exclusions-c47b4673f223b9647a36.json (deflated 12%) 2024-12-18T02:55:01.2863243Z adding: test/test-reports/td_exclusions-0a9c402d274b4b0dc0ab.json (deflated 11%) 2024-12-18T02:55:01.2864016Z adding: test/test-reports/td_exclusions-9d8c91cb4a9d74245f1e.json (deflated 18%) 2024-12-18T02:55:01.2894390Z ##[group]Run # Remove any previous test reports if they exist 2024-12-18T02:55:01.2894934Z # Remove any previous test reports if they exist 2024-12-18T02:55:01.2895384Z rm -f test-reports-*.zip 2024-12-18T02:55:01.2896037Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2024-12-18T02:55:01.2902235Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:55:01.2902690Z env: 2024-12-18T02:55:01.2902956Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:01.2903358Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:01.2904019Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:01.2904957Z FILE_SUFFIX: test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_34566462667 2024-12-18T02:55:01.2905504Z ##[endgroup] 2024-12-18T02:55:01.3009644Z adding: test/test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-d03c917415630208.xml (deflated 92%) 2024-12-18T02:55:01.3011270Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_ops/distributed._tensor.test_dtensor_ops-4b5a3a823961eaf1.xml (deflated 29%) 2024-12-18T02:55:01.3012918Z adding: test/test-reports/python-pytest/distributed._tensor.test_random_ops/distributed._tensor.test_random_ops-ecb8a75d4211c88c.xml (deflated 78%) 2024-12-18T02:55:01.3014285Z adding: test/test-reports/python-pytest/distributed.launcher.test_run/distributed.launcher.test_run-1094c9b22ef79bd2.xml (deflated 84%) 2024-12-18T02:55:01.3015776Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_init/distributed._shard.sharded_tensor.ops.test_init-511335140843e167.xml (deflated 61%) 2024-12-18T02:55:01.3017542Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_clip_grad_norm_/distributed._composable.fsdp.test_fully_shard_clip_grad_norm_-9af753d789c0ac0b.xml (deflated 51%) 2024-12-18T02:55:01.3019445Z adding: test/test-reports/python-pytest/distributed._tensor.test_view_ops/distributed._tensor.test_view_ops-cc52130f1bbb11f6.xml (deflated 61%) 2024-12-18T02:55:01.3020972Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-19ce5c1df442cb35.xml (deflated 38%) 2024-12-18T02:55:01.3022592Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e8517940d63a0ddc.xml (deflated 37%) 2024-12-18T02:55:01.3024194Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-30e8b65f81ecf095.xml (deflated 37%) 2024-12-18T02:55:01.3025799Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-46d68f39c32503b1.xml (deflated 38%) 2024-12-18T02:55:01.3027397Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5c39066fb62a8226.xml (deflated 37%) 2024-12-18T02:55:01.3028995Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-15040751221059ab.xml (deflated 37%) 2024-12-18T02:55:01.3030587Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a50b66c567ebf3e6.xml (deflated 38%) 2024-12-18T02:55:01.3032478Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ae3dd06abc14426c.xml (deflated 38%) 2024-12-18T02:55:01.3034034Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-592c1623276e328d.xml (deflated 39%) 2024-12-18T02:55:01.3035599Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-edf518fce72fb4fb.xml (deflated 39%) 2024-12-18T02:55:01.3037156Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-cd7da290c80d71e2.xml (deflated 39%) 2024-12-18T02:55:01.3038712Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-405772d6ca6080c5.xml (deflated 39%) 2024-12-18T02:55:01.3040255Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-8284b0144441d360.xml (deflated 39%) 2024-12-18T02:55:01.3041801Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-97b7faaca67672c2.xml (deflated 39%) 2024-12-18T02:55:01.3043536Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e0ed15f2cd5d6ab8.xml (deflated 39%) 2024-12-18T02:55:01.3045094Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2c43bfb59562e7a7.xml (deflated 39%) 2024-12-18T02:55:01.3046761Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-053b945814918c9a.xml (deflated 38%) 2024-12-18T02:55:01.3048307Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5e3b531b794808bf.xml (deflated 38%) 2024-12-18T02:55:01.3049856Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ad36960aa9484441.xml (deflated 38%) 2024-12-18T02:55:01.3051412Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-75abff3093381f15.xml (deflated 38%) 2024-12-18T02:55:01.3052962Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ec5a5b7c491928fb.xml (deflated 38%) 2024-12-18T02:55:01.3054513Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a2e69d87820aff71.xml (deflated 38%) 2024-12-18T02:55:01.3056082Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-99d9f0a4eef6b8a2.xml (deflated 38%) 2024-12-18T02:55:01.3057635Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ef8b3c91cadc9cf0.xml (deflated 38%) 2024-12-18T02:55:01.3059426Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c271c6557ae00e23.xml (deflated 38%) 2024-12-18T02:55:01.3061036Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d482e8995fd12c94.xml (deflated 38%) 2024-12-18T02:55:01.3062627Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-f114e711c90bfc79.xml (deflated 38%) 2024-12-18T02:55:01.3064241Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2fea7ca051637ed3.xml (deflated 38%) 2024-12-18T02:55:01.3065846Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-f8f1580e39a234fd.xml (deflated 39%) 2024-12-18T02:55:01.3067451Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-07b18e980ba21ae0.xml (deflated 38%) 2024-12-18T02:55:01.3069054Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-791ff4035b376c7d.xml (deflated 38%) 2024-12-18T02:55:01.3070653Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9de01a0bd4fd7c09.xml (deflated 39%) 2024-12-18T02:55:01.3072332Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1881aa321a9d3793.xml (deflated 38%) 2024-12-18T02:55:01.3073878Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-00054a711d5d3b73.xml (deflated 38%) 2024-12-18T02:55:01.3075424Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b5847ad86a18bc01.xml (deflated 39%) 2024-12-18T02:55:01.3077054Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-8753d98f2ea46f5f.xml (deflated 38%) 2024-12-18T02:55:01.3078614Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c0b7f5bd281c056e.xml (deflated 39%) 2024-12-18T02:55:01.3080220Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2c05e1b9266faafd.xml (deflated 38%) 2024-12-18T02:55:01.3081776Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4b620fe8ddae6a17.xml (deflated 38%) 2024-12-18T02:55:01.3083330Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-91857de5afcf460c.xml (deflated 38%) 2024-12-18T02:55:01.3084887Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-fc8e008ce7f32dd2.xml (deflated 38%) 2024-12-18T02:55:01.3086447Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b4c6713014553fca.xml (deflated 37%) 2024-12-18T02:55:01.3087993Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-617c3fca3d0d2271.xml (deflated 38%) 2024-12-18T02:55:01.3089546Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e5f346e332c5d106.xml (deflated 38%) 2024-12-18T02:55:01.3091101Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c4cde8453606ca50.xml (deflated 38%) 2024-12-18T02:55:01.3092641Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-7ea1f9a100002a39.xml (deflated 39%) 2024-12-18T02:55:01.3094192Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4d07506910e24d2b.xml (deflated 38%) 2024-12-18T02:55:01.3095833Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_tensor_ops/distributed._shard.sharded_tensor.ops.test_tensor_ops-9f24576d7e2ae976.xml (deflated 68%) 2024-12-18T02:55:01.3097604Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_embedding_bag/distributed._shard.sharded_tensor.ops.test_embedding_bag-a04caea339299a53.xml (deflated 54%) 2024-12-18T02:55:01.3099384Z adding: test/test-reports/python-pytest/distributed.test_c10d_ops_nccl/distributed.test_c10d_ops_nccl-6afe1590234e03b5.xml (deflated 85%) 2024-12-18T02:55:01.3100730Z adding: test/test-reports/python-pytest/distributed.test_c10d_ops_nccl/distributed.test_c10d_ops_nccl-7aa32fc61287dc2a.xml (deflated 85%) 2024-12-18T02:55:01.3102078Z adding: test/test-reports/python-pytest/distributed.test_c10d_ops_nccl/distributed.test_c10d_ops_nccl-290d5ee9ad57e9c6.xml (deflated 85%) 2024-12-18T02:55:01.3103421Z adding: test/test-reports/python-pytest/distributed.test_c10d_ops_nccl/distributed.test_c10d_ops_nccl-3a7384e47334a35f.xml (deflated 85%) 2024-12-18T02:55:01.3104893Z adding: test/test-reports/python-pytest/distributed._tensor.debug.test_op_coverage/distributed._tensor.debug.test_op_coverage-bb3eb2fa7bf04eeb.xml (deflated 35%) 2024-12-18T02:55:01.3106462Z adding: test/test-reports/python-pytest/distributed.test_compute_comm_reordering/distributed.test_compute_comm_reordering-6c2a788705a662b0.xml (deflated 73%) 2024-12-18T02:55:01.3107942Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor/distributed._tensor.test_dtensor-641b24868a0210be.xml (deflated 86%) 2024-12-18T02:55:01.3109425Z adding: test/test-reports/python-pytest/distributed.pipelining.test_unflatten/distributed.pipelining.test_unflatten-ffb7c0e88e974307.xml (deflated 41%) 2024-12-18T02:55:01.3111183Z adding: test/test-reports/python-pytest/distributed.checkpoint.fsdp.test_fsdp_dsd/distributed.checkpoint.fsdp.test_fsdp_dsd-dd511090bea527bf.xml (deflated 73%) 2024-12-18T02:55:01.3112796Z adding: test/test-reports/python-pytest/distributed._tensor.experimental.test_local_map/distributed._tensor.experimental.test_local_map-9e60ec31ef384be4.xml (deflated 65%) 2024-12-18T02:55:01.3114420Z adding: test/test-reports/python-pytest/distributed._tensor.test_redistribute/distributed._tensor.test_redistribute-c4e0a0d716d1f249.xml (deflated 80%) 2024-12-18T02:55:01.3115909Z adding: test/test-reports/python-pytest/distributed._tools.test_fsdp2_mem_tracker/distributed._tools.test_fsdp2_mem_tracker-9f00922115cf93db.xml (deflated 57%) 2024-12-18T02:55:01.3117433Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_format_utils/distributed.checkpoint.test_format_utils-7c27d2bdddc64ed2.xml (deflated 57%) 2024-12-18T02:55:01.3119039Z adding: test/test-reports/python-pytest/distributed._tensor.debug.test_comm_mode_features/distributed._tensor.debug.test_comm_mode_features-4f9c1a5cfe23d4e3.xml (deflated 64%) 2024-12-18T02:55:01.3120657Z adding: test/test-reports/python-pytest/distributed.checkpoint.e2e.test_fine_tuning/distributed.checkpoint.e2e.test_fine_tuning-1d3a7e6633971fb9.xml (deflated 39%) 2024-12-18T02:55:01.3122329Z adding: test/test-reports/python-pytest/distributed._tensor.experimental.test_tp_transform/distributed._tensor.experimental.test_tp_transform-7259e0c4f73a6604.xml (deflated 60%) 2024-12-18T02:55:01.3123976Z adding: test/test-reports/python-pytest/distributed.checkpoint.e2e.test_pipeline/distributed.checkpoint.e2e.test_pipeline-5e34076472b90c59.xml (deflated 39%) 2024-12-18T02:55:01.3125501Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_traverse/distributed.checkpoint.test_traverse-19a8ed90d3e8a6c1.xml (deflated 75%) 2024-12-18T02:55:01.3127102Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_random_state/distributed.tensor.parallel.test_tp_random_state-2e87b262859cc52e.xml (deflated 38%) 2024-12-18T02:55:01.3128787Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dtensor_checkpoint/distributed.checkpoint.test_dtensor_checkpoint-36f7564126ba57ab.xml (deflated 40%) 2024-12-18T02:55:01.3130522Z adding: test/test-reports/python-pytest/distributed._composable.test_replicate_with_compiler/distributed._composable.test_replicate_with_compiler-cacaa4ae37f6b08e.xml (deflated 76%) 2024-12-18T02:55:01.3132189Z adding: test/test-reports/python-pytest/distributed.test_c10d_logger/distributed.test_c10d_logger-5701b070301c871c.xml (deflated 48%) 2024-12-18T02:55:01.3133616Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_state_dict/distributed.fsdp.test_fsdp_state_dict-34c3049531379588.xml (deflated 95%) 2024-12-18T02:55:01.3134960Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2021288a942fba77.xml (deflated 38%) 2024-12-18T02:55:01.3136209Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7faf56e040cd194b.xml (deflated 34%) 2024-12-18T02:55:01.3137448Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-742abb904f2d2c1a.xml (deflated 35%) 2024-12-18T02:55:01.3138770Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f7d8887e294a8597.xml (deflated 34%) 2024-12-18T02:55:01.3140206Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-76b1fb94e95e9f4e.xml (deflated 35%) 2024-12-18T02:55:01.3141488Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-13e343cf17a47f78.xml (deflated 49%) 2024-12-18T02:55:01.3142766Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-80ace4e5d52f0ac1.xml (deflated 49%) 2024-12-18T02:55:01.3144150Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-823abeeb55d00893.xml (deflated 49%) 2024-12-18T02:55:01.3145434Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ca86ba62bc7cc80f.xml (deflated 49%) 2024-12-18T02:55:01.3146716Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5e47bdbf7c0ca645.xml (deflated 35%) 2024-12-18T02:55:01.3148079Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-10a3d7e0cb0dd243.xml (deflated 34%) 2024-12-18T02:55:01.3149354Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2fe539b352e23b76.xml (deflated 34%) 2024-12-18T02:55:01.3150707Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-70e7c53b2dd46929.xml (deflated 35%) 2024-12-18T02:55:01.3151939Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8f83765ebd2c381a.xml (deflated 35%) 2024-12-18T02:55:01.3153181Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-bd3028e0930eb3dd.xml (deflated 35%) 2024-12-18T02:55:01.3154415Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2d6e95241d1d7152.xml (deflated 36%) 2024-12-18T02:55:01.3155647Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-730d253b4c199a4e.xml (deflated 36%) 2024-12-18T02:55:01.3156881Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d688cfde8781a993.xml (deflated 37%) 2024-12-18T02:55:01.3158123Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-844a8de182801d2c.xml (deflated 36%) 2024-12-18T02:55:01.3159355Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a4c2e1012524867b.xml (deflated 36%) 2024-12-18T02:55:01.3160579Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b785f80f74f2f25b.xml (deflated 36%) 2024-12-18T02:55:01.3161813Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-de27f16cf0541e1d.xml (deflated 36%) 2024-12-18T02:55:01.3163054Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b3df2874e41a2afd.xml (deflated 37%) 2024-12-18T02:55:01.3164295Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0b48e95e3337bd40.xml (deflated 37%) 2024-12-18T02:55:01.3165524Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e1329884891ab73c.xml (deflated 36%) 2024-12-18T02:55:01.3166757Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-215d2b576b494cfb.xml (deflated 36%) 2024-12-18T02:55:01.3167991Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a94e7fc608550235.xml (deflated 36%) 2024-12-18T02:55:01.3169202Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b94647458052e874.xml (deflated 37%) 2024-12-18T02:55:01.3170429Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5340279366a5d1fa.xml (deflated 36%) 2024-12-18T02:55:01.3171660Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ee9c1371c0ade670.xml (deflated 36%) 2024-12-18T02:55:01.3172891Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8684085b3d4c5def.xml (deflated 35%) 2024-12-18T02:55:01.3174122Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-97b9dee6f4fd31ba.xml (deflated 37%) 2024-12-18T02:55:01.3175349Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-78a0a2ba02229500.xml (deflated 36%) 2024-12-18T02:55:01.3176637Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8906bbb9582becff.xml (deflated 37%) 2024-12-18T02:55:01.3177881Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-cbb5eacc1bfb693d.xml (deflated 37%) 2024-12-18T02:55:01.3179407Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-32ef27cc97b84b70.xml (deflated 37%) 2024-12-18T02:55:01.3180684Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-18f84e2b2970b370.xml (deflated 35%) 2024-12-18T02:55:01.3181960Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3ff3a581b319b3d3.xml (deflated 35%) 2024-12-18T02:55:01.3183239Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a0eec2e9a271066a.xml (deflated 35%) 2024-12-18T02:55:01.3184521Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-def8ee11fba5a0a8.xml (deflated 36%) 2024-12-18T02:55:01.3185797Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-338b361a6f2c0653.xml (deflated 45%) 2024-12-18T02:55:01.3187073Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-05af9bdbe2e713dc.xml (deflated 36%) 2024-12-18T02:55:01.3188340Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0efb252d06f3c247.xml (deflated 37%) 2024-12-18T02:55:01.3189609Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-30d7c5af296ed70c.xml (deflated 37%) 2024-12-18T02:55:01.3190878Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5e643cf9c77bd6d5.xml (deflated 37%) 2024-12-18T02:55:01.3192231Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-704058ec5b5e32b8.xml (deflated 36%) 2024-12-18T02:55:01.3193463Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c068847c55713692.xml (deflated 36%) 2024-12-18T02:55:01.3194699Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-39f300f8f6c8df58.xml (deflated 35%) 2024-12-18T02:55:01.3195944Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-367aa3bc62e04d4e.xml (deflated 36%) 2024-12-18T02:55:01.3197181Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-26aa9aa0b5d8fc97.xml (deflated 36%) 2024-12-18T02:55:01.3198408Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5e53d1c9a3c4afff.xml (deflated 37%) 2024-12-18T02:55:01.3199647Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-bd7d591c43ba6fe3.xml (deflated 36%) 2024-12-18T02:55:01.3200889Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-99ec08cfe22a5113.xml (deflated 35%) 2024-12-18T02:55:01.3202116Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1a51976152095a22.xml (deflated 34%) 2024-12-18T02:55:01.3203335Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-567085303583965c.xml (deflated 36%) 2024-12-18T02:55:01.3204554Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-68365430290b6144.xml (deflated 35%) 2024-12-18T02:55:01.3205778Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a458f532acfa5924.xml (deflated 36%) 2024-12-18T02:55:01.3207013Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d83817ca60f75adc.xml (deflated 35%) 2024-12-18T02:55:01.3208309Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4685a731d82779a5.xml (deflated 35%) 2024-12-18T02:55:01.3209538Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a08fbde851cf5b96.xml (deflated 34%) 2024-12-18T02:55:01.3210766Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0b707986577115e1.xml (deflated 36%) 2024-12-18T02:55:01.3212057Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9ec594d1e56dcd22.xml (deflated 36%) 2024-12-18T02:55:01.3213286Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ac522a10f6548f70.xml (deflated 36%) 2024-12-18T02:55:01.3214519Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7e783d83c5448fdd.xml (deflated 36%) 2024-12-18T02:55:01.3215764Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ac1c3e9b50cd38f9.xml (deflated 34%) 2024-12-18T02:55:01.3216985Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-995669f9c478aab1.xml (deflated 34%) 2024-12-18T02:55:01.3218214Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0b98a3b9586a085f.xml (deflated 34%) 2024-12-18T02:55:01.3219701Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5364d49ea5d6acf2.xml (deflated 34%) 2024-12-18T02:55:01.3220972Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-970ef486a522381d.xml (deflated 34%) 2024-12-18T02:55:01.3222250Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fdda724badfaab3e.xml (deflated 34%) 2024-12-18T02:55:01.3223532Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-313c37c7bbb2a0d9.xml (deflated 33%) 2024-12-18T02:55:01.3224810Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3502bd7c53c328bd.xml (deflated 34%) 2024-12-18T02:55:01.3226091Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d2ba3a3c9a913ace.xml (deflated 35%) 2024-12-18T02:55:01.3227355Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-97cae8c14afc2b87.xml (deflated 42%) 2024-12-18T02:55:01.3228768Z adding: test/test-reports/python-pytest/distributed.pipelining.test_schedule/distributed.pipelining.test_schedule-193ca99d1810b860.xml (deflated 91%) 2024-12-18T02:55:01.3230261Z adding: test/test-reports/python-pytest/distributed._tensor.test_op_strategy/distributed._tensor.test_op_strategy-1700d4f7a4954b5f.xml (deflated 82%) 2024-12-18T02:55:01.3232296Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_model_state/distributed.checkpoint.test_fsdp_model_state-33f734061eff4018.xml (deflated 52%) 2024-12-18T02:55:01.3233804Z adding: test/test-reports/python-pytest/distributed.rpc.test_share_memory/distributed.rpc.test_share_memory-1caea17497094632.xml (deflated 48%) 2024-12-18T02:55:01.3235294Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_dtensor_state_dict/distributed.fsdp.test_fsdp_dtensor_state_dict-ec0e9e9437951a27.xml (deflated 89%) 2024-12-18T02:55:01.3236864Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_checkpoint/distributed.checkpoint.test_checkpoint-b427a69fd802347a.xml (deflated 74%) 2024-12-18T02:55:01.3238324Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_memory/distributed.fsdp.test_fsdp_memory-d3a1998281afcb95.xml (deflated 51%) 2024-12-18T02:55:01.3239885Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_frozen/distributed._composable.fsdp.test_fully_shard_frozen-3ba44fdb46f38566.xml (deflated 58%) 2024-12-18T02:55:01.3241721Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_parallelize_api/distributed.tensor.parallel.test_parallelize_api-689fd499e5ff91b2.xml (deflated 81%) 2024-12-18T02:55:01.3243309Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-f286b213351555b2.xml (deflated 36%) 2024-12-18T02:55:01.3244869Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_state_dict/distributed.checkpoint.test_state_dict-64f6af59dd99ada0.xml (deflated 83%) 2024-12-18T02:55:01.3246326Z adding: test/test-reports/python-pytest/distributed.test_multi_threaded_pg/distributed.test_multi_threaded_pg-d8d6ddda23fa805f.xml (deflated 81%) 2024-12-18T02:55:01.3247716Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-9002b06629463077.xml (deflated 34%) 2024-12-18T02:55:01.3249226Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-20450ac99703a45d.xml (deflated 51%) 2024-12-18T02:55:01.3267968Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-7d50897a1c5d51c7.xml (deflated 38%) 2024-12-18T02:55:01.3269239Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-259da3c7758dcdb1.xml (deflated 39%) 2024-12-18T02:55:01.3270487Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-72808494dab6c0b8.xml (deflated 37%) 2024-12-18T02:55:01.3271835Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-573a60bc1f305642.xml (deflated 37%) 2024-12-18T02:55:01.3273036Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-0dfc57db96c496a8.xml (deflated 44%) 2024-12-18T02:55:01.3274229Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-c50ab19f723dcdf6.xml (deflated 38%) 2024-12-18T02:55:01.3275431Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-c9b0556d99d92475.xml (deflated 37%) 2024-12-18T02:55:01.3276628Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-37b6cb89b9302af5.xml (deflated 37%) 2024-12-18T02:55:01.3277826Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-2cc9c936bde983d0.xml (deflated 37%) 2024-12-18T02:55:01.3279079Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-657932047a3d2050.xml (deflated 37%) 2024-12-18T02:55:01.3280305Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-f6a12e59034747bf.xml (deflated 39%) 2024-12-18T02:55:01.3281497Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-9cfa93465d3dc307.xml (deflated 39%) 2024-12-18T02:55:01.3282688Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-65f117209066ee3d.xml (deflated 38%) 2024-12-18T02:55:01.3283872Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-63634a99042cbf79.xml (deflated 38%) 2024-12-18T02:55:01.3285068Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-51dcae7fc53f37d3.xml (deflated 38%) 2024-12-18T02:55:01.3286277Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-5d5ac55e0b7b6237.xml (deflated 38%) 2024-12-18T02:55:01.3287494Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-dfdbb7a5fccf22c6.xml (deflated 37%) 2024-12-18T02:55:01.3288696Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-518ebbbf41a8ec67.xml (deflated 38%) 2024-12-18T02:55:01.3289906Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-09e5491c8ac60c2a.xml (deflated 38%) 2024-12-18T02:55:01.3291234Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-c47abd6a7b485293.xml (deflated 38%) 2024-12-18T02:55:01.3292779Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-15038a15a15657f9.xml (deflated 37%) 2024-12-18T02:55:01.3293927Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-7296da85b51e27e5.xml (deflated 38%) 2024-12-18T02:55:01.3295159Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-f19c5ca0e07310da.xml (deflated 38%) 2024-12-18T02:55:01.3296316Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-76b185c2478696f1.xml (deflated 38%) 2024-12-18T02:55:01.3297478Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-e4634af68fe2c438.xml (deflated 38%) 2024-12-18T02:55:01.3298640Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-0171f30c35ba4742.xml (deflated 38%) 2024-12-18T02:55:01.3300168Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-172420ccbc00d21f.xml (deflated 38%) 2024-12-18T02:55:01.3301404Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-0ad1eb0c65e01c85.xml (deflated 38%) 2024-12-18T02:55:01.3302643Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-081f6ae3bdd981cd.xml (deflated 45%) 2024-12-18T02:55:01.3303880Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-6c53b97050dfcfef.xml (deflated 38%) 2024-12-18T02:55:01.3305115Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-e32854b7ba5833c8.xml (deflated 38%) 2024-12-18T02:55:01.3306357Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-11526e1168d8c05d.xml (deflated 44%) 2024-12-18T02:55:01.3307604Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-9c5a163ce669890c.xml (deflated 42%) 2024-12-18T02:55:01.3308841Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-a540c7ad40371116.xml (deflated 42%) 2024-12-18T02:55:01.3310068Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-fbdeaa08261a80a6.xml (deflated 43%) 2024-12-18T02:55:01.3311426Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-1e5a8fdb8674f2e3.xml (deflated 38%) 2024-12-18T02:55:01.3312629Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-83337e3b8cb4b053.xml (deflated 38%) 2024-12-18T02:55:01.3313724Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-d125034b37cec1b1.xml (deflated 38%) 2024-12-18T02:55:01.3314829Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-78e53587f31cf1c4.xml (deflated 59%) 2024-12-18T02:55:01.3315930Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-c0a17fa15e1df1a7.xml (deflated 45%) 2024-12-18T02:55:01.3317028Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-319d1b2bdbf81fc5.xml (deflated 46%) 2024-12-18T02:55:01.3318131Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-763e94c80f02be1d.xml (deflated 38%) 2024-12-18T02:55:01.3319227Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-94888404de6b70cf.xml (deflated 38%) 2024-12-18T02:55:01.3320324Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-a03b60a129dce3f5.xml (deflated 38%) 2024-12-18T02:55:01.3321413Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-de82eefdc002b68b.xml (deflated 38%) 2024-12-18T02:55:01.3322571Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-bb8bfe62a103e4f3.xml (deflated 47%) 2024-12-18T02:55:01.3323674Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-ba4d4bedba254107.xml (deflated 47%) 2024-12-18T02:55:01.3324772Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-227ea1295e2dbc56.xml (deflated 39%) 2024-12-18T02:55:01.3325918Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-8d3df8e3c034be1c.xml (deflated 38%) 2024-12-18T02:55:01.3327022Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-ccb83cdef7d0ae5c.xml (deflated 38%) 2024-12-18T02:55:01.3328127Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-ef3af1e7fdff9e7c.xml (deflated 38%) 2024-12-18T02:55:01.3329219Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-524df6099ad973ed.xml (deflated 38%) 2024-12-18T02:55:01.3330305Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-8ba72d878bd6a85d.xml (deflated 38%) 2024-12-18T02:55:01.3331390Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-c9dd36fcf8344a35.xml (deflated 59%) 2024-12-18T02:55:01.3332904Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-d09990ba170e81ab.xml (deflated 38%) 2024-12-18T02:55:01.3334102Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-39ad0aed648c27a9.xml (deflated 42%) 2024-12-18T02:55:01.3335304Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-450bd4c793e7b2ed.xml (deflated 38%) 2024-12-18T02:55:01.3336497Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-d6afba6341616334.xml (deflated 42%) 2024-12-18T02:55:01.3337696Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-e9b1efefcfda8415.xml (deflated 40%) 2024-12-18T02:55:01.3338960Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-dcefea535bef7359.xml (deflated 41%) 2024-12-18T02:55:01.3340443Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-5da479f4e23304aa.xml (deflated 37%) 2024-12-18T02:55:01.3341677Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-78a4be477c707eb3.xml (deflated 37%) 2024-12-18T02:55:01.3343223Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_state_dict/distributed._composable.fsdp.test_fully_shard_state_dict-692fdefe912ecdd7.xml (deflated 75%) 2024-12-18T02:55:01.3344809Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-6c53a384def2fd7b.xml (deflated 36%) 2024-12-18T02:55:01.3346108Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-3b4d1943798163cc.xml (deflated 36%) 2024-12-18T02:55:01.3347397Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-5cd25a2e9affa542.xml (deflated 37%) 2024-12-18T02:55:01.3348691Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-8613c665c9506ee1.xml (deflated 36%) 2024-12-18T02:55:01.3349963Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-4c01845c73234659.xml (deflated 37%) 2024-12-18T02:55:01.3351402Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-44ea6d40b8effb8a.xml (deflated 36%) 2024-12-18T02:55:01.3352553Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-ced009b987433cf5.xml (deflated 36%) 2024-12-18T02:55:01.3353802Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-0947318d3fedb702.xml (deflated 37%) 2024-12-18T02:55:01.3354945Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-808d742db397d4a5.xml (deflated 37%) 2024-12-18T02:55:01.3356094Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-f2588fefec5f21d8.xml (deflated 37%) 2024-12-18T02:55:01.3357305Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-582f0a402821ca81.xml (deflated 37%) 2024-12-18T02:55:01.3358442Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-b7d75a80581011d0.xml (deflated 37%) 2024-12-18T02:55:01.3359582Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-9656576b3644805d.xml (deflated 37%) 2024-12-18T02:55:01.3360723Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-49a9a5e26559cb5f.xml (deflated 38%) 2024-12-18T02:55:01.3361865Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-3f045716ba6209e5.xml (deflated 37%) 2024-12-18T02:55:01.3363010Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-3e4ac301cf3c40cf.xml (deflated 37%) 2024-12-18T02:55:01.3364159Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-ad648e4b696d4bfa.xml (deflated 37%) 2024-12-18T02:55:01.3365547Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-0b290248fd0994e5.xml (deflated 81%) 2024-12-18T02:55:01.3367131Z adding: test/test-reports/python-pytest/distributed._shard.sharding_plan.test_sharding_plan/distributed._shard.sharding_plan.test_sharding_plan-56498423039b6266.xml (deflated 60%) 2024-12-18T02:55:01.3368620Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_multiple_forward/distributed.fsdp.test_fsdp_multiple_forward-b5e0a011eba92d8b.xml (deflated 28%) 2024-12-18T02:55:01.3370064Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_sharded_grad_scaler/distributed.fsdp.test_fsdp_sharded_grad_scaler-9c06ecfaf6d1a91a.xml (deflated 89%) 2024-12-18T02:55:01.3371419Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_meta/distributed.fsdp.test_fsdp_meta-ae9b2fa2d02d679a.xml (deflated 88%) 2024-12-18T02:55:01.3372811Z adding: test/test-reports/python-pytest/distributed._shard.sharding_spec.test_sharding_spec/distributed._shard.sharding_spec.test_sharding_spec-b2f2117c3a55ce26.xml (deflated 82%) 2024-12-18T02:55:01.3374455Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/distributed._shard.sharded_tensor.test_sharded_tensor_reshard-d224f4d2dcde5264.xml (deflated 56%) 2024-12-18T02:55:01.3375888Z adding: test/test-reports/python-pytest/distributed.test_launcher/distributed.test_launcher-9c52ac7be2e25519.xml (deflated 40%) 2024-12-18T02:55:01.3377239Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_sharded_tensor/distributed._shard.sharded_tensor.test_sharded_tensor-6dfd3b15f30f9349.xml (deflated 89%) 2024-12-18T02:55:01.3378753Z adding: test/test-reports/python-pytest/distributed.elastic.events.lib_test/distributed.elastic.events.lib_test-804e2e29799b53f5.xml (deflated 76%) 2024-12-18T02:55:01.3380357Z adding: test/test-reports/python-pytest/distributed.elastic.metrics.api_test/distributed.elastic.metrics.api_test-f8e3ec4ae69e2b10.xml (deflated 59%) 2024-12-18T02:55:01.3381981Z adding: test/test-reports/python-pytest/distributed.elastic.multiprocessing.api_test/distributed.elastic.multiprocessing.api_test-8f1010877ee6b21f.xml (deflated 84%) 2024-12-18T02:55:01.3383635Z adding: test/test-reports/python-pytest/distributed.elastic.timer.local_timer_test/distributed.elastic.timer.local_timer_test-387c97092cb83e23.xml (deflated 80%) 2024-12-18T02:55:01.3385278Z adding: test/test-reports/python-pytest/distributed.elastic.utils.logging_test/distributed.elastic.utils.logging_test-900db9f8de762fd5.xml (deflated 51%) 2024-12-18T02:55:01.3386803Z adding: test/test-reports/python-pytest/distributed.elastic.utils.util_test/distributed.elastic.utils.util_test-631108aa52849a74.xml (deflated 81%) 2024-12-18T02:55:01.3388095Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-94a644091be1b736.xml (deflated 60%) 2024-12-18T02:55:01.3389069Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-411d9464b2ee0b30.xml (deflated 60%) 2024-12-18T02:55:01.3390046Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-d7d7cf3339845ea1.xml (deflated 69%) 2024-12-18T02:55:01.3391042Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-9cbaaf0430380c7f.xml (deflated 77%) 2024-12-18T02:55:01.3392193Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-d59ce55faeb414d5.xml (deflated 74%) 2024-12-18T02:55:01.3393072Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-f129af15f9ce5703.xml (deflated 76%) 2024-12-18T02:55:01.3393942Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-7bdc5814439d9464.xml (deflated 64%) 2024-12-18T02:55:01.3394811Z adding: test/test-reports/python-pytest/test_linalg/test_linalg-4a17ecb09e942afa.xml (deflated 43%) 2024-12-18T02:55:01.3395869Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-03c29f415ffdf05f.xml (deflated 38%) 2024-12-18T02:55:01.3397098Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d00206c5087f04f.xml (deflated 44%) 2024-12-18T02:55:01.3398340Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b708bf75a755feee.xml (deflated 37%) 2024-12-18T02:55:01.3399800Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-00d7f3aeee779bfa.xml (deflated 37%) 2024-12-18T02:55:01.3401122Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b6857ecfacae26be.xml (deflated 38%) 2024-12-18T02:55:01.3402438Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-04e0d075a2001927.xml (deflated 37%) 2024-12-18T02:55:01.3403850Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4206370929ed492.xml (deflated 44%) 2024-12-18T02:55:01.3405087Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af990af700b1bfc9.xml (deflated 44%) 2024-12-18T02:55:01.3406330Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1d9c916be3e2ef48.xml (deflated 43%) 2024-12-18T02:55:01.3407781Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-df18fcec7bdef21a.xml (deflated 43%) 2024-12-18T02:55:01.3409106Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c55dc38df6b6b92.xml (deflated 43%) 2024-12-18T02:55:01.3410427Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e19934b8461abca.xml (deflated 43%) 2024-12-18T02:55:01.3411744Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4acfb46cf3d8883e.xml (deflated 43%) 2024-12-18T02:55:01.3413056Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c9a65ae690155ef7.xml (deflated 38%) 2024-12-18T02:55:01.3414365Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-08100910093d261c.xml (deflated 44%) 2024-12-18T02:55:01.3415723Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5bc2c7298c28a1b2.xml (deflated 38%) 2024-12-18T02:55:01.3417039Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6c6a10dd861e741c.xml (deflated 44%) 2024-12-18T02:55:01.3418392Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6ef382ecf75489d9.xml (deflated 44%) 2024-12-18T02:55:01.3420084Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-12b24a9122440847.xml (deflated 41%) 2024-12-18T02:55:01.3421488Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c0b8a7e497add44.xml (deflated 37%) 2024-12-18T02:55:01.3422881Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-00c27ad5d572b574.xml (deflated 37%) 2024-12-18T02:55:01.3424271Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0e696c96b82e7b44.xml (deflated 43%) 2024-12-18T02:55:01.3425668Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2fdfe694a424bc4a.xml (deflated 37%) 2024-12-18T02:55:01.3427073Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b62f00979cfd3878.xml (deflated 58%) 2024-12-18T02:55:01.3428465Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ea60a4ce60c3780.xml (deflated 38%) 2024-12-18T02:55:01.3429843Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-60224b87c922e257.xml (deflated 43%) 2024-12-18T02:55:01.3431350Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f3edf96b312823d.xml (deflated 43%) 2024-12-18T02:55:01.3432900Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a33cbe73939e2ffc.xml (deflated 44%) 2024-12-18T02:55:01.3434261Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-40b37aef829bd5d8.xml (deflated 38%) 2024-12-18T02:55:01.3435624Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4bdcfea7ea479fda.xml (deflated 38%) 2024-12-18T02:55:01.3436995Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-afaf0c22af5310d6.xml (deflated 38%) 2024-12-18T02:55:01.3438362Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8c59b9b04c86bb81.xml (deflated 43%) 2024-12-18T02:55:01.3439720Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c5f1b347d8bb4f8f.xml (deflated 38%) 2024-12-18T02:55:01.3441066Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d2c1bc15fa9228ae.xml (deflated 43%) 2024-12-18T02:55:01.3442518Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-616e44989f2792ac.xml (deflated 37%) 2024-12-18T02:55:01.3443942Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-35bd661e943b69a1.xml (deflated 37%) 2024-12-18T02:55:01.3445181Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d48b64475e903941.xml (deflated 44%) 2024-12-18T02:55:01.3446427Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32a2d5bd8a908eb0.xml (deflated 37%) 2024-12-18T02:55:01.3447674Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b3e232727ed32db.xml (deflated 38%) 2024-12-18T02:55:01.3449020Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c89944af2b209c57.xml (deflated 37%) 2024-12-18T02:55:01.3450263Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b16ab3c0bbb676ec.xml (deflated 37%) 2024-12-18T02:55:01.3451574Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e3f2bd98cbba4579.xml (deflated 44%) 2024-12-18T02:55:01.3452806Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5855ab837c82f8be.xml (deflated 44%) 2024-12-18T02:55:01.3454042Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4f36721d636b3b9.xml (deflated 43%) 2024-12-18T02:55:01.3455292Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6bcaf3c39feff579.xml (deflated 43%) 2024-12-18T02:55:01.3456539Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6cca47d702dbfb7e.xml (deflated 43%) 2024-12-18T02:55:01.3457781Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4914a6839593cb8e.xml (deflated 43%) 2024-12-18T02:55:01.3459248Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f68b485589c88fcd.xml (deflated 43%) 2024-12-18T02:55:01.3460654Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ff85b2d9287d3c47.xml (deflated 38%) 2024-12-18T02:55:01.3462053Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d36ab5190270374b.xml (deflated 45%) 2024-12-18T02:55:01.3463442Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-50bbbf1733cb5b58.xml (deflated 38%) 2024-12-18T02:55:01.3464842Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6adbc9e92e09dd63.xml (deflated 44%) 2024-12-18T02:55:01.3466239Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-efc32545d008f3f5.xml (deflated 44%) 2024-12-18T02:55:01.3467655Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f7d7dddc2be9dbe.xml (deflated 41%) 2024-12-18T02:55:01.3469063Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-266fd04355a92224.xml (deflated 37%) 2024-12-18T02:55:01.3470454Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-97df20204b64d2bd.xml (deflated 38%) 2024-12-18T02:55:01.3471998Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-766cdf0de1138029.xml (deflated 43%) 2024-12-18T02:55:01.3473243Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3c08684fa3e38f6c.xml (deflated 37%) 2024-12-18T02:55:01.3474480Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0532e4f9c73daa2d.xml (deflated 58%) 2024-12-18T02:55:01.3475726Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e3cb9e93e4004655.xml (deflated 38%) 2024-12-18T02:55:01.3476974Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fe43e542b9497edd.xml (deflated 43%) 2024-12-18T02:55:01.3478212Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-69a0ad503bc3a2a9.xml (deflated 43%) 2024-12-18T02:55:01.3479508Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3a718b810ac4fbfa.xml (deflated 44%) 2024-12-18T02:55:01.3480751Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-580a739f8c8eac2f.xml (deflated 38%) 2024-12-18T02:55:01.3481994Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f27efc36d4ced99.xml (deflated 38%) 2024-12-18T02:55:01.3483295Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ba139ec67ea5d35d.xml (deflated 38%) 2024-12-18T02:55:01.3484523Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-55be2a3c50d0c386.xml (deflated 43%) 2024-12-18T02:55:01.3485756Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2fba5e3420287662.xml (deflated 38%) 2024-12-18T02:55:01.3487001Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ec5c28a0ffca22fd.xml (deflated 43%) 2024-12-18T02:55:01.3488240Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f267ece1292c5675.xml (deflated 38%) 2024-12-18T02:55:01.3489478Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-518dfaf73fcef247.xml (deflated 39%) 2024-12-18T02:55:01.3490720Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e5b51bb51bf579c.xml (deflated 38%) 2024-12-18T02:55:01.3491960Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3bd90723ffc0823f.xml (deflated 37%) 2024-12-18T02:55:01.3493197Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e476aa081a13d49.xml (deflated 38%) 2024-12-18T02:55:01.3494424Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e3d508523c668450.xml (deflated 37%) 2024-12-18T02:55:01.3495663Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bbe11077390a3bc3.xml (deflated 44%) 2024-12-18T02:55:01.3496905Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34befadb388bac1d.xml (deflated 44%) 2024-12-18T02:55:01.3498150Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cb5a54b8191d7951.xml (deflated 44%) 2024-12-18T02:55:01.3499670Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-79b74c0c8d3cfffa.xml (deflated 44%) 2024-12-18T02:55:01.3501073Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f54346472166eaa9.xml (deflated 43%) 2024-12-18T02:55:01.3502479Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ed579400e5f75006.xml (deflated 43%) 2024-12-18T02:55:01.3503883Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-abaa02f76e6f6216.xml (deflated 43%) 2024-12-18T02:55:01.3505274Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fa9b06bab82c2c27.xml (deflated 43%) 2024-12-18T02:55:01.3506671Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-47047a369832eb76.xml (deflated 48%) 2024-12-18T02:55:01.3508065Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-48efc3a66f990dea.xml (deflated 49%) 2024-12-18T02:55:01.3509465Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-662e57bf812d7f52.xml (deflated 38%) 2024-12-18T02:55:01.3510926Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a102ad1145c237db.xml (deflated 43%) 2024-12-18T02:55:01.3512430Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-850c93797cf2688d.xml (deflated 37%) 2024-12-18T02:55:01.3513714Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4a8679f0d254fce0.xml (deflated 37%) 2024-12-18T02:55:01.3514948Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-009626bc89db6106.xml (deflated 42%) 2024-12-18T02:55:01.3516179Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f2eaa5184ba356c0.xml (deflated 38%) 2024-12-18T02:55:01.3517407Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f0b0b83236782366.xml (deflated 43%) 2024-12-18T02:55:01.3518650Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aea35635f18da312.xml (deflated 38%) 2024-12-18T02:55:01.3519895Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-27d9f7ded9f4cbed.xml (deflated 38%) 2024-12-18T02:55:01.3521138Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9c79c4287776d029.xml (deflated 38%) 2024-12-18T02:55:01.3522378Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-92331fcdc530e32b.xml (deflated 37%) 2024-12-18T02:55:01.3523617Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f38fb4c85b52fd75.xml (deflated 38%) 2024-12-18T02:55:01.3524856Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1cb393b57bd30bc0.xml (deflated 44%) 2024-12-18T02:55:01.3526094Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7388e70a977356de.xml (deflated 43%) 2024-12-18T02:55:01.3527318Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70294f4d28659f04.xml (deflated 43%) 2024-12-18T02:55:01.3528557Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-642ed544b0d1ed9e.xml (deflated 38%) 2024-12-18T02:55:01.3529802Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-05f0a03d63f420ad.xml (deflated 39%) 2024-12-18T02:55:01.3531035Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d240463536a50ba4.xml (deflated 38%) 2024-12-18T02:55:01.3532625Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-42162530f4a657b5.xml (deflated 37%) 2024-12-18T02:55:01.3534043Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-239c5e3fd3e9d667.xml (deflated 38%) 2024-12-18T02:55:01.3535396Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1027ceb07169b8ab.xml (deflated 37%) 2024-12-18T02:55:01.3536769Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d808fc2b29da06ed.xml (deflated 44%) 2024-12-18T02:55:01.3538106Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6996d76610629d31.xml (deflated 44%) 2024-12-18T02:55:01.3539709Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-10b85518fd19c603.xml (deflated 44%) 2024-12-18T02:55:01.3541099Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aa1f1d496192d259.xml (deflated 44%) 2024-12-18T02:55:01.3542601Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-293a4bb78bc65588.xml (deflated 42%) 2024-12-18T02:55:01.3544002Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3345d0ee88c26bf1.xml (deflated 43%) 2024-12-18T02:55:01.3545488Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f504391a874f7095.xml (deflated 43%) 2024-12-18T02:55:01.3546886Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2827ff464fa8f284.xml (deflated 43%) 2024-12-18T02:55:01.3548278Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-945972509e73ba7f.xml (deflated 48%) 2024-12-18T02:55:01.3549668Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e1f9d0b14f16ac2a.xml (deflated 49%) 2024-12-18T02:55:01.3551250Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f3cc210d49bca512.xml (deflated 38%) 2024-12-18T02:55:01.3552493Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-909081b7244c1a20.xml (deflated 43%) 2024-12-18T02:55:01.3553736Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af598aaee276acb6.xml (deflated 37%) 2024-12-18T02:55:01.3554978Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-09dea2313b38efd7.xml (deflated 37%) 2024-12-18T02:55:01.3556211Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34d20f23321de3e8.xml (deflated 42%) 2024-12-18T02:55:01.3557462Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ceb4b53048a5a6e5.xml (deflated 38%) 2024-12-18T02:55:01.3558706Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b6f80f70814b531a.xml (deflated 43%) 2024-12-18T02:55:01.3559929Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f56e15a266210050.xml (deflated 38%) 2024-12-18T02:55:01.3561178Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eac9fa13fe3cda3a.xml (deflated 38%) 2024-12-18T02:55:01.3562415Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5269e6d93acfc274.xml (deflated 38%) 2024-12-18T02:55:01.3563656Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-134f56e9d0589c04.xml (deflated 37%) 2024-12-18T02:55:01.3564896Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13c4f3b81eb729b7.xml (deflated 38%) 2024-12-18T02:55:01.3566135Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4d5d747373f38bf6.xml (deflated 44%) 2024-12-18T02:55:01.3567373Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-491aaf497b1f6597.xml (deflated 43%) 2024-12-18T02:55:01.3568614Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c6af33b5ad733a8f.xml (deflated 43%) 2024-12-18T02:55:01.3569837Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-92b63e708977ce9f.xml (deflated 38%) 2024-12-18T02:55:01.3571066Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0b6f78a57d788faf.xml (deflated 39%) 2024-12-18T02:55:01.3572339Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-da43923d8837961f.xml (deflated 38%) 2024-12-18T02:55:01.3573562Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89b0f0ae57e8bcf6.xml (deflated 38%) 2024-12-18T02:55:01.3574785Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-07a1eb01f5a082e6.xml (deflated 37%) 2024-12-18T02:55:01.3576051Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b4567746f9f27876.xml (deflated 38%) 2024-12-18T02:55:01.3577291Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-531dc80d46f877a0.xml (deflated 44%) 2024-12-18T02:55:01.3578519Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70d17beac298ddd5.xml (deflated 44%) 2024-12-18T02:55:01.3580123Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aa44d3c59361f1b1.xml (deflated 43%) 2024-12-18T02:55:01.3581513Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6cf2cb72f8d7c81e.xml (deflated 38%) 2024-12-18T02:55:01.3582886Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-313c42b5af2135c8.xml (deflated 38%) 2024-12-18T02:55:01.3584270Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-df8adc82874291e3.xml (deflated 44%) 2024-12-18T02:55:01.3585656Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-91630b9e4041225b.xml (deflated 38%) 2024-12-18T02:55:01.3587042Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dd7c36a661776ec9.xml (deflated 44%) 2024-12-18T02:55:01.3588427Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2228973c27c45443.xml (deflated 38%) 2024-12-18T02:55:01.3589803Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7e7b4aaa394ab4cb.xml (deflated 47%) 2024-12-18T02:55:01.3591297Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c3e38a5e26799be7.xml (deflated 38%) 2024-12-18T02:55:01.3592654Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-251672fe7e2141dd.xml (deflated 37%) 2024-12-18T02:55:01.3593872Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6f363be1c46073f.xml (deflated 38%) 2024-12-18T02:55:01.3595092Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-33df313bf6918a62.xml (deflated 37%) 2024-12-18T02:55:01.3596318Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9242802dd2a25e8c.xml (deflated 43%) 2024-12-18T02:55:01.3597545Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-01977a1c98863305.xml (deflated 38%) 2024-12-18T02:55:01.3598765Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f2d373f10b8a04c2.xml (deflated 37%) 2024-12-18T02:55:01.3599996Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c782580024e32338.xml (deflated 37%) 2024-12-18T02:55:01.3601223Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b22d9ce57fc0663.xml (deflated 43%) 2024-12-18T02:55:01.3602445Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bcf8d3561dc35bca.xml (deflated 45%) 2024-12-18T02:55:01.3603717Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f7d7cb31e6adca2b.xml (deflated 38%) 2024-12-18T02:55:01.3604941Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-642dc975f85c262b.xml (deflated 43%) 2024-12-18T02:55:01.3606206Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c872a5104f70dd9.xml (deflated 38%) 2024-12-18T02:55:01.3607430Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-db31b8ad41109ec8.xml (deflated 37%) 2024-12-18T02:55:01.3608653Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-228f692b61547506.xml (deflated 38%) 2024-12-18T02:55:01.3609883Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ba5be2a7234ddc65.xml (deflated 46%) 2024-12-18T02:55:01.3611114Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2d4b660660e2e03e.xml (deflated 45%) 2024-12-18T02:55:01.3612340Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4fe607e643afada.xml (deflated 39%) 2024-12-18T02:55:01.3613573Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-08dde5f0d9979a2b.xml (deflated 38%) 2024-12-18T02:55:01.3614807Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e1e787e53560d7b1.xml (deflated 38%) 2024-12-18T02:55:01.3616031Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b73062061c39e390.xml (deflated 37%) 2024-12-18T02:55:01.3617254Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6952e6237c1adacb.xml (deflated 37%) 2024-12-18T02:55:01.3618484Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-23dcc99a7ed44170.xml (deflated 44%) 2024-12-18T02:55:01.3620082Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-09ced1d5f117e706.xml (deflated 44%) 2024-12-18T02:55:01.3621470Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e57cc1b27a0d2d45.xml (deflated 43%) 2024-12-18T02:55:01.3622860Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5be667493012e25f.xml (deflated 38%) 2024-12-18T02:55:01.3624232Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c5806ac15270762.xml (deflated 38%) 2024-12-18T02:55:01.3625616Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b572e2cfe7e57070.xml (deflated 44%) 2024-12-18T02:55:01.3627008Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7431ba7eb6a0f859.xml (deflated 38%) 2024-12-18T02:55:01.3628388Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ea681133049615a5.xml (deflated 44%) 2024-12-18T02:55:01.3629774Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-729d90c5f356aa8c.xml (deflated 38%) 2024-12-18T02:55:01.3631278Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d7210c1142cecfb0.xml (deflated 47%) 2024-12-18T02:55:01.3632804Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af69fa39b605823c.xml (deflated 38%) 2024-12-18T02:55:01.3634145Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c8babe38d6da6922.xml (deflated 37%) 2024-12-18T02:55:01.3635593Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5010b83930b67d1a.xml (deflated 38%) 2024-12-18T02:55:01.3636942Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-65dc96d15f81079b.xml (deflated 37%) 2024-12-18T02:55:01.3638351Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5290c2818203d841.xml (deflated 43%) 2024-12-18T02:55:01.3639687Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0f80170e7691a827.xml (deflated 38%) 2024-12-18T02:55:01.3641031Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d0dcac697c33caf3.xml (deflated 37%) 2024-12-18T02:55:01.3642380Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15993b6a98cc51f6.xml (deflated 37%) 2024-12-18T02:55:01.3643815Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9235af67db3b46a0.xml (deflated 43%) 2024-12-18T02:55:01.3645040Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-deba156d55745888.xml (deflated 45%) 2024-12-18T02:55:01.3646274Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f5c635fc512d4a80.xml (deflated 38%) 2024-12-18T02:55:01.3647513Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-357edc1361607dce.xml (deflated 43%) 2024-12-18T02:55:01.3648745Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-38350e4b317d9904.xml (deflated 38%) 2024-12-18T02:55:01.3649974Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6f8884f8e24264b0.xml (deflated 38%) 2024-12-18T02:55:01.3651202Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5f9d2c89677cdae2.xml (deflated 38%) 2024-12-18T02:55:01.3652430Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-20460a95a8518927.xml (deflated 46%) 2024-12-18T02:55:01.3653656Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aba68603b0465b3d.xml (deflated 45%) 2024-12-18T02:55:01.3654877Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-20410f69016eb892.xml (deflated 38%) 2024-12-18T02:55:01.3656100Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aba839c7d98cc35e.xml (deflated 38%) 2024-12-18T02:55:01.3657321Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-deae363d8c592d91.xml (deflated 38%) 2024-12-18T02:55:01.3658542Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-817ae675283987d2.xml (deflated 38%) 2024-12-18T02:55:01.3660088Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3753642d11a6161.xml (deflated 45%) 2024-12-18T02:55:01.3661480Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1587edead797b4b.xml (deflated 37%) 2024-12-18T02:55:01.3662860Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-139944d142f6142c.xml (deflated 48%) 2024-12-18T02:55:01.3664287Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5936c9f52f86ed5a.xml (deflated 49%) 2024-12-18T02:55:01.3665721Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-215aee229bba0adc.xml (deflated 38%) 2024-12-18T02:55:01.3667110Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30a52fe58a3bfddb.xml (deflated 38%) 2024-12-18T02:55:01.3668495Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30eb85c24c7290d7.xml (deflated 38%) 2024-12-18T02:55:01.3669930Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3c6646f196b936e1.xml (deflated 38%) 2024-12-18T02:55:01.3671422Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b1a916e4fcbf0628.xml (deflated 37%) 2024-12-18T02:55:01.3672650Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9c7d5217184fb9f7.xml (deflated 38%) 2024-12-18T02:55:01.3673892Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0b05182ec73db73e.xml (deflated 44%) 2024-12-18T02:55:01.3675140Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-de6f8d07bf59ff14.xml (deflated 45%) 2024-12-18T02:55:01.3676389Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e8c4c02527480ef4.xml (deflated 44%) 2024-12-18T02:55:01.3677638Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-76dd8d5f0990fdf5.xml (deflated 44%) 2024-12-18T02:55:01.3678877Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d2d8e5cc128abef.xml (deflated 38%) 2024-12-18T02:55:01.3680125Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8ba7a00f016f1b90.xml (deflated 38%) 2024-12-18T02:55:01.3681372Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-53e89f1e6eb39d5c.xml (deflated 37%) 2024-12-18T02:55:01.3682604Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-35b3a187d5ae3daa.xml (deflated 38%) 2024-12-18T02:55:01.3683845Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9880f04b95c316d4.xml (deflated 38%) 2024-12-18T02:55:01.3685092Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c1ab098643d4bea.xml (deflated 43%) 2024-12-18T02:55:01.3686330Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24b41e5db0749b47.xml (deflated 38%) 2024-12-18T02:55:01.3687562Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0ab6ac9e0077533b.xml (deflated 38%) 2024-12-18T02:55:01.3688805Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1d309735ac7c63f6.xml (deflated 38%) 2024-12-18T02:55:01.3690050Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d43168a3ffe9b239.xml (deflated 37%) 2024-12-18T02:55:01.3691285Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fb5862feab9f1a1d.xml (deflated 38%) 2024-12-18T02:55:01.3692513Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3f363f9e2337fcf5.xml (deflated 38%) 2024-12-18T02:55:01.3693748Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7097a3b904d1838e.xml (deflated 38%) 2024-12-18T02:55:01.3694984Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a7a7edc00b82cb9f.xml (deflated 38%) 2024-12-18T02:55:01.3696278Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-58c19d59dd3b9a8d.xml (deflated 42%) 2024-12-18T02:55:01.3697514Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5d7228e552c1dde.xml (deflated 37%) 2024-12-18T02:55:01.3698892Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0a45f1f33b22059a.xml (deflated 38%) 2024-12-18T02:55:01.3700435Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a47c1ecd46f4f609.xml (deflated 38%) 2024-12-18T02:55:01.3701829Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-373f37d212ffe1ae.xml (deflated 38%) 2024-12-18T02:55:01.3703219Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f9fff5a7ab3cb934.xml (deflated 37%) 2024-12-18T02:55:01.3704625Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3a2c3aef99527111.xml (deflated 37%) 2024-12-18T02:55:01.3706016Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a81b23bcce081b65.xml (deflated 45%) 2024-12-18T02:55:01.3707413Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-115744cbf73e92f1.xml (deflated 37%) 2024-12-18T02:55:01.3708799Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ef80eabca21e0612.xml (deflated 49%) 2024-12-18T02:55:01.3710191Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-85c6775618361bcb.xml (deflated 48%) 2024-12-18T02:55:01.3711737Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4fe6d8a20e249320.xml (deflated 38%) 2024-12-18T02:55:01.3712972Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a703da921a60614.xml (deflated 38%) 2024-12-18T02:55:01.3714192Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-80e732706d2baa98.xml (deflated 38%) 2024-12-18T02:55:01.3715429Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-38364f68af5ba443.xml (deflated 38%) 2024-12-18T02:55:01.3716655Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d0d943c042203703.xml (deflated 38%) 2024-12-18T02:55:01.3717886Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c28565297f150c3a.xml (deflated 38%) 2024-12-18T02:55:01.3719122Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e875cbd8fa2a9e03.xml (deflated 45%) 2024-12-18T02:55:01.3720367Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b4dbea74d4e6e43d.xml (deflated 44%) 2024-12-18T02:55:01.3721601Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-211bff1b1840279c.xml (deflated 44%) 2024-12-18T02:55:01.3722840Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2e231c0d593c7ad6.xml (deflated 44%) 2024-12-18T02:55:01.3724070Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-086160685e637f83.xml (deflated 37%) 2024-12-18T02:55:01.3725290Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6b0f1d1a7a068d0c.xml (deflated 37%) 2024-12-18T02:55:01.3726530Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-45a322909619e5b5.xml (deflated 37%) 2024-12-18T02:55:01.3727813Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c7183c648dcd8c9c.xml (deflated 37%) 2024-12-18T02:55:01.3729041Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-44cd69ac574a3850.xml (deflated 38%) 2024-12-18T02:55:01.3730322Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-439a36f24e0f117c.xml (deflated 43%) 2024-12-18T02:55:01.3731691Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d191fc1ac06c755f.xml (deflated 38%) 2024-12-18T02:55:01.3733197Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-94503c7d84ce05d2.xml (deflated 38%) 2024-12-18T02:55:01.3734536Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-203aca403152f82d.xml (deflated 38%) 2024-12-18T02:55:01.3735880Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fe7883f38b7d1abd.xml (deflated 37%) 2024-12-18T02:55:01.3737221Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ec17a44810a7b32.xml (deflated 38%) 2024-12-18T02:55:01.3738562Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-79ac27f5007d1805.xml (deflated 38%) 2024-12-18T02:55:01.3740219Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-02e0446559034dee.xml (deflated 37%) 2024-12-18T02:55:01.3741606Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af91ce07f264b3ab.xml (deflated 38%) 2024-12-18T02:55:01.3743002Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96c6b91e0a4cc674.xml (deflated 42%) 2024-12-18T02:55:01.3744392Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c0b5ddfe0190634.xml (deflated 38%) 2024-12-18T02:55:01.3745784Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6b640709a8257094.xml (deflated 37%) 2024-12-18T02:55:01.3747162Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-56dc2e2392bcbfcf.xml (deflated 39%) 2024-12-18T02:55:01.3748556Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f220644387abf5a8.xml (deflated 38%) 2024-12-18T02:55:01.3749955Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ea1d82bcf0bcdc51.xml (deflated 37%) 2024-12-18T02:55:01.3751610Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5b48545591304aca.xml (deflated 38%) 2024-12-18T02:55:01.3752835Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f69eddb16e3d2f6.xml (deflated 37%) 2024-12-18T02:55:01.3754281Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-616f7df5ecaf2db3.xml (deflated 37%) 2024-12-18T02:55:01.3755585Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1f6036e1d29510f9.xml (deflated 38%) 2024-12-18T02:55:01.3756886Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f72ca836ecd5f129.xml (deflated 38%) 2024-12-18T02:55:01.3758182Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51a202a7fdc704d2.xml (deflated 38%) 2024-12-18T02:55:01.3759582Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8901fae4c43f9b76.xml (deflated 49%) 2024-12-18T02:55:01.3760890Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-392ea2a76161db48.xml (deflated 38%) 2024-12-18T02:55:01.3762195Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c420fa7f69f1714f.xml (deflated 38%) 2024-12-18T02:55:01.3763562Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a2ac609cc6ab338.xml (deflated 43%) 2024-12-18T02:55:01.3764866Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a734d80b4e9c3b03.xml (deflated 44%) 2024-12-18T02:55:01.3766165Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-44624966941160f1.xml (deflated 49%) 2024-12-18T02:55:01.3767511Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4fd86a1cb2f72616.xml (deflated 42%) 2024-12-18T02:55:01.3768733Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e1f92c292564c4c2.xml (deflated 38%) 2024-12-18T02:55:01.3769961Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c44de02fbc809387.xml (deflated 48%) 2024-12-18T02:55:01.3771205Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fd5c6605f6beb0ed.xml (deflated 38%) 2024-12-18T02:55:01.3772446Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ff9c18490d731bd.xml (deflated 38%) 2024-12-18T02:55:01.3773674Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c906ce7ec707186c.xml (deflated 38%) 2024-12-18T02:55:01.3774903Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9b493dfe02a8471.xml (deflated 43%) 2024-12-18T02:55:01.3776135Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1ed2704c6ef67e22.xml (deflated 38%) 2024-12-18T02:55:01.3777359Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-02058a42215ccbdc.xml (deflated 38%) 2024-12-18T02:55:01.3778582Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1589f08d6a7d4252.xml (deflated 38%) 2024-12-18T02:55:01.3780136Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-07349aa821fd13f2.xml (deflated 37%) 2024-12-18T02:55:01.3781523Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2458becfe7b40952.xml (deflated 38%) 2024-12-18T02:55:01.3782919Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e2e368f31effebab.xml (deflated 38%) 2024-12-18T02:55:01.3784308Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b7a7627c35e56d35.xml (deflated 38%) 2024-12-18T02:55:01.3785700Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-97f3ab2ba536cfbb.xml (deflated 49%) 2024-12-18T02:55:01.3787095Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0ac1dabeddd8d7d6.xml (deflated 43%) 2024-12-18T02:55:01.3788484Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-81a76682d996468d.xml (deflated 39%) 2024-12-18T02:55:01.3789862Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1edc45dd9655f879.xml (deflated 38%) 2024-12-18T02:55:01.3791906Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a5e6ad74cb74986.xml (deflated 37%) 2024-12-18T02:55:01.3793224Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7bf73bcab2ff7487.xml (deflated 38%) 2024-12-18T02:55:01.3794593Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-409c440e130ca4c9.xml (deflated 37%) 2024-12-18T02:55:01.3795911Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-790a1d41faaab9b4.xml (deflated 37%) 2024-12-18T02:55:01.3797219Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-33152df546ce46c8.xml (deflated 37%) 2024-12-18T02:55:01.3798524Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7730b109e52c707f.xml (deflated 37%) 2024-12-18T02:55:01.3799835Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4b5da90cc8d75e02.xml (deflated 37%) 2024-12-18T02:55:01.3801133Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-65350db37c7680e7.xml (deflated 49%) 2024-12-18T02:55:01.3802449Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-37097c4fead77d40.xml (deflated 38%) 2024-12-18T02:55:01.3803844Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9e5b89e3563c892.xml (deflated 38%) 2024-12-18T02:55:01.3805086Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e0401807ffc04d88.xml (deflated 43%) 2024-12-18T02:55:01.3806320Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bce9a878b5991a6c.xml (deflated 44%) 2024-12-18T02:55:01.3807549Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e96b537193ace7b2.xml (deflated 49%) 2024-12-18T02:55:01.3808780Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ba4928c344c19067.xml (deflated 42%) 2024-12-18T02:55:01.3810011Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0b0a4dca2d43a7a7.xml (deflated 38%) 2024-12-18T02:55:01.3811247Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3bf1a191cfab8452.xml (deflated 48%) 2024-12-18T02:55:01.3812465Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-26a09c9e6279a4e8.xml (deflated 37%) 2024-12-18T02:55:01.3813691Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6ff8be5ca7e40c44.xml (deflated 38%) 2024-12-18T02:55:01.3814933Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a6c58cf2d0720de.xml (deflated 38%) 2024-12-18T02:55:01.3816164Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c70d21b09089fc8.xml (deflated 42%) 2024-12-18T02:55:01.3817398Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-841776071b49a647.xml (deflated 38%) 2024-12-18T02:55:01.3818637Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bee29ca9bfd7b7d4.xml (deflated 38%) 2024-12-18T02:55:01.3820216Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6e2fa381f5a69bf5.xml (deflated 38%) 2024-12-18T02:55:01.3821603Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a00ab3c31c3100b.xml (deflated 37%) 2024-12-18T02:55:01.3823045Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e25cf53b9c2de1ce.xml (deflated 38%) 2024-12-18T02:55:01.3824440Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-57fc43e4388930a6.xml (deflated 38%) 2024-12-18T02:55:01.3825882Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fbb45b8eb33b2d77.xml (deflated 38%) 2024-12-18T02:55:01.3827283Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dad72bf316a3efb1.xml (deflated 49%) 2024-12-18T02:55:01.3828711Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ee5446b6c7c416f0.xml (deflated 43%) 2024-12-18T02:55:01.3830133Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c9f321b1dfb9ca43.xml (deflated 39%) 2024-12-18T02:55:01.3831764Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-354ba6cf103843a2.xml (deflated 38%) 2024-12-18T02:55:01.3833109Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-483265d7983f4247.xml (deflated 39%) 2024-12-18T02:55:01.3834457Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d2b2688b91d23cc3.xml (deflated 37%) 2024-12-18T02:55:01.3835796Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f646efbbe5de9a3a.xml (deflated 43%) 2024-12-18T02:55:01.3837147Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-29598236e00dc01b.xml (deflated 37%) 2024-12-18T02:55:01.3838489Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9db89e1723253e66.xml (deflated 37%) 2024-12-18T02:55:01.3839827Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-16654285f51100c3.xml (deflated 43%) 2024-12-18T02:55:01.3841165Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1f5f571bb3e90f65.xml (deflated 43%) 2024-12-18T02:55:01.3842515Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d87afd47824c705c.xml (deflated 44%) 2024-12-18T02:55:01.3843985Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4d96cf4b90b868d7.xml (deflated 38%) 2024-12-18T02:55:01.3845212Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c51a31d89d5b9f4.xml (deflated 38%) 2024-12-18T02:55:01.3846447Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fa2d60c8ccc7cf1c.xml (deflated 49%) 2024-12-18T02:55:01.3847674Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9ecb8926a5a38113.xml (deflated 42%) 2024-12-18T02:55:01.3848909Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-53d7c2ef44b798e9.xml (deflated 47%) 2024-12-18T02:55:01.3850149Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e308750b103d0da.xml (deflated 47%) 2024-12-18T02:55:01.3851388Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1279756f8dad1126.xml (deflated 37%) 2024-12-18T02:55:01.3852615Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ac901315bcc3ce23.xml (deflated 38%) 2024-12-18T02:55:01.3853944Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ddcaf94f9dbb3c32.xml (deflated 37%) 2024-12-18T02:55:01.3855183Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0d80231ed212c348.xml (deflated 49%) 2024-12-18T02:55:01.3856415Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-67b02d87ba0b35b0.xml (deflated 38%) 2024-12-18T02:55:01.3857705Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d65105c44d358095.xml (deflated 38%) 2024-12-18T02:55:01.3859003Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1ad87bc58e3daddf.xml (deflated 37%) 2024-12-18T02:55:01.3860559Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c3467717cd785b2a.xml (deflated 37%) 2024-12-18T02:55:01.3861956Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-df6a2d907efe51c8.xml (deflated 49%) 2024-12-18T02:55:01.3863348Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4449885997ba030f.xml (deflated 45%) 2024-12-18T02:55:01.3864736Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ba42e53baa3bdf31.xml (deflated 38%) 2024-12-18T02:55:01.3866134Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ff173571fad3e89.xml (deflated 43%) 2024-12-18T02:55:01.3867518Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-197cd2a5fb841f94.xml (deflated 41%) 2024-12-18T02:55:01.3868897Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4a85a62546d4cebe.xml (deflated 42%) 2024-12-18T02:55:01.3870294Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b8ffb69c3ffa47fd.xml (deflated 38%) 2024-12-18T02:55:01.3871744Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7e128f2a70f7f758.xml (deflated 37%) 2024-12-18T02:55:01.3872961Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f63502368f7d63ad.xml (deflated 39%) 2024-12-18T02:55:01.3874189Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-308822abe545a7f3.xml (deflated 38%) 2024-12-18T02:55:01.3875423Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2e2c6ee88d8bcca.xml (deflated 38%) 2024-12-18T02:55:01.3876661Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e54ae7c3795299d.xml (deflated 37%) 2024-12-18T02:55:01.3877887Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-94b707f5ff2bdd43.xml (deflated 43%) 2024-12-18T02:55:01.3879101Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a94b0fc658da6a6d.xml (deflated 37%) 2024-12-18T02:55:01.3880339Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7e942bfa8d199552.xml (deflated 38%) 2024-12-18T02:55:01.3881572Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-447524757c662e7f.xml (deflated 43%) 2024-12-18T02:55:01.3882800Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-63d232bcf27c2877.xml (deflated 43%) 2024-12-18T02:55:01.3884037Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-899dc90b7e3c3c4c.xml (deflated 44%) 2024-12-18T02:55:01.3885323Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-838c44c8e7fc0a04.xml (deflated 38%) 2024-12-18T02:55:01.3886580Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cd0402fbc825faa0.xml (deflated 38%) 2024-12-18T02:55:01.3887816Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d70e3b3200267f34.xml (deflated 49%) 2024-12-18T02:55:01.3889090Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3debbc6e57f7f1f6.xml (deflated 42%) 2024-12-18T02:55:01.3890332Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2e31afecefd3410.xml (deflated 47%) 2024-12-18T02:55:01.3891575Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f493d3b880ca1e29.xml (deflated 47%) 2024-12-18T02:55:01.3893024Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b772c37666872b66.xml (deflated 37%) 2024-12-18T02:55:01.3894328Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a92627311d0b536.xml (deflated 38%) 2024-12-18T02:55:01.3895634Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8354d6832a14b31c.xml (deflated 37%) 2024-12-18T02:55:01.3896930Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-16149dc774e40101.xml (deflated 49%) 2024-12-18T02:55:01.3898227Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ff8db8985ae204cf.xml (deflated 38%) 2024-12-18T02:55:01.3899778Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-610e6e87234be60b.xml (deflated 38%) 2024-12-18T02:55:01.3901175Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eac466b66b7680cf.xml (deflated 38%) 2024-12-18T02:55:01.3902577Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf7dfd6a4855eb63.xml (deflated 38%) 2024-12-18T02:55:01.3903979Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-61efa32067ec599e.xml (deflated 49%) 2024-12-18T02:55:01.3905375Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c5780978a1381781.xml (deflated 45%) 2024-12-18T02:55:01.3906763Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9ac9079855bfef86.xml (deflated 38%) 2024-12-18T02:55:01.3908158Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d17074ee3328256.xml (deflated 42%) 2024-12-18T02:55:01.3909557Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d385011cfc8c4cc5.xml (deflated 41%) 2024-12-18T02:55:01.3911050Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5b6df469774a555d.xml (deflated 42%) 2024-12-18T02:55:01.3912294Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-757490622914196f.xml (deflated 38%) 2024-12-18T02:55:01.3913537Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6a4b68ccecac7ac9.xml (deflated 37%) 2024-12-18T02:55:01.3941510Z ##[group]Run # Remove any previous usage logs if they exist 2024-12-18T02:55:01.3942168Z # Remove any previous usage logs if they exist 2024-12-18T02:55:01.3942623Z rm -f logs-*.zip 2024-12-18T02:55:01.3943298Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-12-18T02:55:01.3943976Z # so check to see if the file exists first 2024-12-18T02:55:01.3944430Z if [ -f 'usage_log.txt' ]; then 2024-12-18T02:55:01.3944914Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-12-18T02:55:01.3945333Z fi 2024-12-18T02:55:01.3945782Z if find "test/test-reports" -name "*.log" 2>/dev/null | grep -q .; then 2024-12-18T02:55:01.3946531Z  zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' 2024-12-18T02:55:01.3947011Z fi 2024-12-18T02:55:01.3952754Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:55:01.3953195Z env: 2024-12-18T02:55:01.3953442Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:01.3953831Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:01.3954470Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:01.3955242Z FILE_SUFFIX: test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_34566462667 2024-12-18T02:55:01.3955786Z ##[endgroup] 2024-12-18T02:55:01.4457398Z adding: usage_log.txt (deflated 97%) 2024-12-18T02:55:01.4537826Z adding: test/test-reports/distributed.test_dynamo_distributed_1.1_7f2977431673f942_.log (deflated 89%) 2024-12-18T02:55:01.4538894Z adding: test/test-reports/distributed._tensor.test_dtensor_ops_1.1_33d90eef123d3c83_.log (deflated 52%) 2024-12-18T02:55:01.4540068Z adding: test/test-reports/distributed._tensor.test_random_ops_1.1_b72a2e9902170a0f_.log (deflated 81%) 2024-12-18T02:55:01.4542429Z adding: test/test-reports/distributed.launcher.test_run_1.1_5dda03571d7a3f14_.log (deflated 89%) 2024-12-18T02:55:01.4543984Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_init_1.1_aab645d3723ef400_.log (deflated 89%) 2024-12-18T02:55:01.4545156Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_clip_grad_norm__1.1_b4c392db59610145_.log (deflated 63%) 2024-12-18T02:55:01.4546269Z adding: test/test-reports/distributed._tensor.test_view_ops_1.1_50492ad2f4dedc26_.log (deflated 85%) 2024-12-18T02:55:01.4555720Z adding: test/test-reports/distributed.rpc.cuda.test_tensorpipe_agent_1.2_629243349af919f6_.log (deflated 96%) 2024-12-18T02:55:01.4557936Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_tensor_ops_1.1_6c6a467ca9118a06_.log (deflated 93%) 2024-12-18T02:55:01.4559437Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_embedding_bag_1.1_a416816ab23249c0_.log (deflated 92%) 2024-12-18T02:55:01.4561764Z adding: test/test-reports/distributed.test_c10d_ops_nccl_1.1_6652867542679a86_.log (deflated 93%) 2024-12-18T02:55:01.4562877Z adding: test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_f5cf6091ee2e9fc6_.log (deflated 55%) 2024-12-18T02:55:01.4563894Z adding: test/test-reports/distributed.test_compute_comm_reordering_1.1_752fbf28fccb92eb_.log (deflated 83%) 2024-12-18T02:55:01.4567545Z adding: test/test-reports/distributed._tensor.test_dtensor_1.1_6fc3edc5e3eed925_.log (deflated 92%) 2024-12-18T02:55:01.4568522Z adding: test/test-reports/distributed.pipelining.test_unflatten_1.1_a8d0b05490f8261f_.log (deflated 54%) 2024-12-18T02:55:01.4570263Z adding: test/test-reports/distributed.checkpoint.fsdp.test_fsdp_dsd_1.1_a2961e367e3f034f_.log (deflated 91%) 2024-12-18T02:55:01.4571334Z adding: test/test-reports/distributed._tensor.experimental.test_local_map_1.1_20abb985a7e2c6e1_.log (deflated 66%) 2024-12-18T02:55:01.4572352Z adding: test/test-reports/distributed.test_distributed_spawn_1.9_7e394755e43cf579_.log (deflated 12%) 2024-12-18T02:55:01.4573316Z adding: test/test-reports/distributed.test_distributed_spawn_1.9_a4a4edc02eaeae1b_.log (deflated 12%) 2024-12-18T02:55:01.4582344Z adding: test/test-reports/distributed.test_distributed_spawn_1.9_9e1a74ad1a397900_.log (deflated 94%) 2024-12-18T02:55:01.4593264Z adding: test/test-reports/distributed.test_distributed_spawn_1.9_1d78ff1119b015a5_.log (deflated 94%) 2024-12-18T02:55:01.4603092Z adding: test/test-reports/distributed.test_distributed_spawn_1.9_6ef26a58ab32fcc3_.log (deflated 94%) 2024-12-18T02:55:01.4612949Z adding: test/test-reports/distributed.test_distributed_spawn_1.9_e1ef8b9db8abbc5f_.log (deflated 94%) 2024-12-18T02:55:01.4614124Z adding: test/test-reports/distributed.test_distributed_spawn_4.9_8f89234d0f67fd40_.log (deflated 12%) 2024-12-18T02:55:01.4615171Z adding: test/test-reports/distributed.test_distributed_spawn_4.9_8dd5100d1e74bd3e_.log (deflated 12%) 2024-12-18T02:55:01.4619719Z adding: test/test-reports/distributed.test_distributed_spawn_4.9_a6d493d9d7e6b2af_.log (deflated 93%) 2024-12-18T02:55:01.4625784Z adding: test/test-reports/distributed.test_distributed_spawn_4.9_d634cfba6ecdc720_.log (deflated 93%) 2024-12-18T02:55:01.4633337Z adding: test/test-reports/distributed.test_distributed_spawn_4.9_2c4f089c130ee9e8_.log (deflated 94%) 2024-12-18T02:55:01.4640349Z adding: test/test-reports/distributed.test_distributed_spawn_4.9_b5c578dacbc86ace_.log (deflated 94%) 2024-12-18T02:55:01.4641299Z adding: test/test-reports/distributed.test_distributed_spawn_7.9_73d1bdd0fc2cbc90_.log (deflated 12%) 2024-12-18T02:55:01.4642255Z adding: test/test-reports/distributed.test_distributed_spawn_7.9_e0ae957bf88ce43d_.log (deflated 12%) 2024-12-18T02:55:01.4645613Z adding: test/test-reports/distributed.test_distributed_spawn_7.9_ca4ba1909aa32dd1_.log (deflated 94%) 2024-12-18T02:55:01.4650379Z adding: test/test-reports/distributed.test_distributed_spawn_7.9_f1acca7c7cb7b082_.log (deflated 94%) 2024-12-18T02:55:01.4655473Z adding: test/test-reports/distributed.test_distributed_spawn_7.9_5dcb700ebef37f01_.log (deflated 94%) 2024-12-18T02:55:01.4660768Z adding: test/test-reports/distributed.test_distributed_spawn_7.9_5bdb35fd530124a2_.log (deflated 94%) 2024-12-18T02:55:01.4675281Z adding: test/test-reports/distributed._tensor.test_redistribute_1.1_31a688f553ef4281_.log (deflated 96%) 2024-12-18T02:55:01.4676475Z adding: test/test-reports/distributed._tools.test_fsdp2_mem_tracker_1.1_bee6ec382cb18ebe_.log (deflated 66%) 2024-12-18T02:55:01.4677491Z adding: test/test-reports/distributed.checkpoint.test_format_utils_1.1_be21b584631f774b_.log (deflated 76%) 2024-12-18T02:55:01.4678535Z adding: test/test-reports/distributed._tensor.debug.test_comm_mode_features_1.1_2d5ca764175a67b1_.log (deflated 83%) 2024-12-18T02:55:01.4680481Z adding: test/test-reports/distributed.checkpoint.e2e.test_fine_tuning_1.1_0c15d68a9a2f340c_.log (deflated 94%) 2024-12-18T02:55:01.4681568Z adding: test/test-reports/distributed._tensor.experimental.test_tp_transform_1.1_ec717b929b1f810e_.log (deflated 64%) 2024-12-18T02:55:01.4682657Z adding: test/test-reports/distributed.checkpoint.e2e.test_pipeline_1.1_b00ce770f3777b1d_.log (deflated 91%) 2024-12-18T02:55:01.4683666Z adding: test/test-reports/distributed.checkpoint.test_traverse_1.1_f48fa04480dc491f_.log (deflated 70%) 2024-12-18T02:55:01.4684725Z adding: test/test-reports/distributed.tensor.parallel.test_tp_random_state_1.1_1043dfe5b2b4bc5e_.log (deflated 55%) 2024-12-18T02:55:01.4685815Z adding: test/test-reports/distributed.checkpoint.test_dtensor_checkpoint_1.1_dbd250dbb5f09c33_.log (deflated 55%) 2024-12-18T02:55:01.4686925Z adding: test/test-reports/distributed._composable.test_replicate_with_compiler_1.1_11a7bf26a24fcdfe_.log (deflated 85%) 2024-12-18T02:55:01.4687939Z adding: test/test-reports/distributed.test_collective_utils_1.1_d7c50749f4c551f6_.log (stored 0%) 2024-12-18T02:55:01.4688833Z adding: test/test-reports/distributed.test_c10d_logger_1.1_e661757fab846e4d_.log (deflated 68%) 2024-12-18T02:55:01.4701218Z adding: test/test-reports/distributed.fsdp.test_fsdp_state_dict_1.2_3c83763df6c2269c_.log (deflated 97%) 2024-12-18T02:55:01.4712629Z adding: test/test-reports/distributed.test_c10d_nccl_2.3_898ada9fd3d2861e_.log (deflated 94%) 2024-12-18T02:55:01.4713903Z adding: test/test-reports/distributed.pipelining.test_schedule_1.1_45c09618d6ffe387_.log (deflated 87%) 2024-12-18T02:55:01.4715030Z adding: test/test-reports/distributed._tensor.test_op_strategy_1.1_2b108bfcc1b26b28_.log (deflated 78%) 2024-12-18T02:55:01.4717074Z adding: test/test-reports/distributed.checkpoint.test_fsdp_model_state_1.1_14c464eb9723dcad_.log (deflated 94%) 2024-12-18T02:55:01.4718068Z adding: test/test-reports/distributed.rpc.test_share_memory_1.1_a5856aa9a2422d00_.log (deflated 57%) 2024-12-18T02:55:01.4722344Z adding: test/test-reports/distributed.fsdp.test_fsdp_dtensor_state_dict_1.1_64fea7bb21c59bcd_.log (deflated 97%) 2024-12-18T02:55:01.4724145Z adding: test/test-reports/distributed.checkpoint.test_checkpoint_1.1_b75de875062f8f79_.log (deflated 91%) 2024-12-18T02:55:01.4725115Z adding: test/test-reports/distributed.fsdp.test_fsdp_memory_1.1_c04dba40b7dc4db9_.log (deflated 74%) 2024-12-18T02:55:01.4726129Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_f516fde209b526bf_.log (deflated 83%) 2024-12-18T02:55:01.4727297Z adding: test/test-reports/distributed.tensor.parallel.test_parallelize_api_1.1_df650fe5ac7b691d_.log (deflated 83%) 2024-12-18T02:55:01.4728342Z adding: test/test-reports/distributed.fsdp.test_fsdp_traversal_1.1_d2aa15fbf48b5d69_.log (deflated 54%) 2024-12-18T02:55:01.4755295Z adding: test/test-reports/distributed.checkpoint.test_state_dict_1.1_325f5010e1535b47_.log (deflated 98%) 2024-12-18T02:55:01.4756475Z adding: test/test-reports/distributed.test_multi_threaded_pg_1.1_b98190b55ff89008_.log (deflated 86%) 2024-12-18T02:55:01.4757417Z adding: test/test-reports/distributed.fsdp.test_fsdp_uneven_1.1_9e3c6300d8470ab0_.log (deflated 54%) 2024-12-18T02:55:01.4758342Z adding: test/test-reports/distributed.fsdp.test_fsdp_input_1.1_1614b5d15bcc8270_.log (deflated 61%) 2024-12-18T02:55:01.4767445Z adding: test/test-reports/distributed.test_c10d_ucc_1.1_9b584052081fcd60_.log (deflated 94%) 2024-12-18T02:55:01.4768452Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_state_dict_1.1_e60ebf6b17165b9c_.log (deflated 77%) 2024-12-18T02:55:01.4771162Z adding: test/test-reports/distributed.test_pg_wrapper_1.1_b79301e4e7822c25_.log (deflated 93%) 2024-12-18T02:55:01.5324616Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_61c430d99580d69c_.log (deflated 95%) 2024-12-18T02:55:01.5325842Z adding: test/test-reports/distributed.rpc.test_tensorpipe_agent_1.1_43db6207905a34e1_.log (stored 0%) 2024-12-18T02:55:01.5326879Z adding: test/test-reports/distributed._shard.sharding_plan.test_sharding_plan_1.1_3c5bc94401dbff11_.log (deflated 87%) 2024-12-18T02:55:01.5327941Z adding: test/test-reports/distributed.fsdp.test_fsdp_multiple_forward_1.1_d9bb5da17296518f_.log (deflated 52%) 2024-12-18T02:55:01.5328978Z adding: test/test-reports/distributed.fsdp.test_fsdp_sharded_grad_scaler_1.1_c9a8c18052d0670e_.log (deflated 96%) 2024-12-18T02:55:01.5329961Z adding: test/test-reports/distributed.fsdp.test_fsdp_meta_1.1_3de85f30b1f9093a_.log (deflated 83%) 2024-12-18T02:55:01.5330960Z adding: test/test-reports/distributed._shard.sharding_spec.test_sharding_spec_1.1_8f7a44c1b4ab7eaa_.log (deflated 84%) 2024-12-18T02:55:01.5332283Z adding: test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_reshard_1.1_476e9ada9fb215ff_.log (deflated 86%) 2024-12-18T02:55:01.5333309Z adding: test/test-reports/distributed.test_launcher_1.1_683a3b9ed0b0675c_.log (deflated 59%) 2024-12-18T02:55:01.5350076Z adding: test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_1.1_e1f7a6721f856577_.log (deflated 96%) 2024-12-18T02:55:01.5351252Z adding: test/test-reports/distributed.elastic.events.lib_test_1.1_7f3ec8b7d8dff807_.log (deflated 72%) 2024-12-18T02:55:01.5352231Z adding: test/test-reports/distributed.elastic.metrics.api_test_1.1_4fc1641e259f9755_.log (deflated 62%) 2024-12-18T02:55:01.5353528Z adding: test/test-reports/distributed.elastic.multiprocessing.api_test_1.1_4063b1564aa229b1_.log (deflated 84%) 2024-12-18T02:55:01.5354759Z adding: test/test-reports/distributed.elastic.timer.local_timer_test_1.1_0759355597f56ff3_.log (deflated 77%) 2024-12-18T02:55:01.5355768Z adding: test/test-reports/distributed.elastic.utils.logging_test_1.1_07147dee7fcfe789_.log (deflated 58%) 2024-12-18T02:55:01.5356735Z adding: test/test-reports/distributed.elastic.utils.util_test_1.1_9f848f2c1fc1943c_.log (deflated 75%) 2024-12-18T02:55:01.5357609Z adding: test/test-reports/cpp.FileStoreTest_1.1_5bb18050eeb37473_.log (deflated 71%) 2024-12-18T02:55:01.5358485Z adding: test/test-reports/cpp.HashStoreTest_1.1_755c1b147766363d_.log (deflated 71%) 2024-12-18T02:55:01.5359258Z adding: test/test-reports/cpp.TCPStoreTest_1.1_03bfdc91bd30b668_.log (deflated 78%) 2024-12-18T02:55:01.5360010Z adding: test/test-reports/test_linalg_1.1_dfddd869d1cd41ca_.log (deflated 49%) 2024-12-18T02:55:01.5360811Z adding: test/test-reports/cpp.ProcessGroupGlooTest_1.1_028ec5f62f95aca9_.log (deflated 84%) 2024-12-18T02:55:01.5361683Z adding: test/test-reports/cpp.ProcessGroupNCCLTest_1.1_c642d57b62fef2a5_.log (deflated 82%) 2024-12-18T02:55:01.5362577Z adding: test/test-reports/cpp.ProcessGroupNCCLErrorsTest_1.1_8a0a9aea239df6ef_.log (deflated 78%) 2024-12-18T02:55:01.5363427Z adding: test/test-reports/cpp.test_cpp_rpc_1.1_0a260d833ffb8314_.log (deflated 78%) 2024-12-18T02:55:01.5397022Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-12-18T02:55:01.5397625Z # Remove any previous debugging artifacts if they exist 2024-12-18T02:55:01.5398101Z rm -f debug-*.zip 2024-12-18T02:55:01.5398430Z if [ -d 'test/debug' ]; then 2024-12-18T02:55:01.5398855Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-12-18T02:55:01.5399257Z fi 2024-12-18T02:55:01.5404666Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:55:01.5405092Z env: 2024-12-18T02:55:01.5405339Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:01.5405708Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:01.5406319Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:01.5407101Z FILE_SUFFIX: test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_34566462667 2024-12-18T02:55:01.5407739Z ##[endgroup] 2024-12-18T02:55:01.5497709Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T02:55:01.5498093Z with: 2024-12-18T02:55:01.5498347Z s3-bucket: gha-artifacts 2024-12-18T02:55:01.5498812Z s3-prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:55:01.5499397Z retention-days: 14 2024-12-18T02:55:01.5499694Z if-no-files-found: warn 2024-12-18T02:55:01.5500070Z path: test-jsons-*.zip 2024-12-18T02:55:01.5500381Z name: artifact 2024-12-18T02:55:01.5500647Z region: us-east-1 2024-12-18T02:55:01.5500918Z env: 2024-12-18T02:55:01.5501162Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:01.5501559Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:01.5502207Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:01.5502771Z ##[endgroup] 2024-12-18T02:55:01.9267430Z NOTE: s3-prefix specified, ignoring name parameter 2024-12-18T02:55:01.9267978Z With the provided path, there will be 1 file uploaded 2024-12-18T02:55:01.9268525Z Uploading to s3 prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:55:01.9311652Z Starting upload of test-jsons-test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_34566462667.zip 2024-12-18T02:55:02.1165812Z Finished upload of test-jsons-test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_34566462667.zip 2024-12-18T02:55:02.1344703Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T02:55:02.1345122Z with: 2024-12-18T02:55:02.1345388Z s3-bucket: gha-artifacts 2024-12-18T02:55:02.1345786Z s3-prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:55:02.1346213Z retention-days: 14 2024-12-18T02:55:02.1346531Z if-no-files-found: error 2024-12-18T02:55:02.1347039Z path: test-reports-*.zip 2024-12-18T02:55:02.1347355Z name: artifact 2024-12-18T02:55:02.1347645Z region: us-east-1 2024-12-18T02:55:02.1347936Z env: 2024-12-18T02:55:02.1348205Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:02.1348617Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:02.1349266Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:02.1349855Z ##[endgroup] 2024-12-18T02:55:02.4780612Z NOTE: s3-prefix specified, ignoring name parameter 2024-12-18T02:55:02.4781450Z With the provided path, there will be 1 file uploaded 2024-12-18T02:55:02.4782006Z Uploading to s3 prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:55:02.4823329Z Starting upload of test-reports-test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_34566462667.zip 2024-12-18T02:55:02.7130795Z Finished upload of test-reports-test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_34566462667.zip 2024-12-18T02:55:02.7307447Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T02:55:02.7307874Z with: 2024-12-18T02:55:02.7308137Z s3-bucket: gha-artifacts 2024-12-18T02:55:02.7308527Z s3-prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:55:02.7308949Z retention-days: 14 2024-12-18T02:55:02.7309266Z if-no-files-found: ignore 2024-12-18T02:55:02.7309604Z path: logs-*.zip 2024-12-18T02:55:02.7309882Z name: artifact 2024-12-18T02:55:02.7310164Z region: us-east-1 2024-12-18T02:55:02.7310448Z env: 2024-12-18T02:55:02.7310712Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:02.7311230Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:02.7311921Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:02.7312443Z ##[endgroup] 2024-12-18T02:55:03.0763967Z NOTE: s3-prefix specified, ignoring name parameter 2024-12-18T02:55:03.0764714Z With the provided path, there will be 1 file uploaded 2024-12-18T02:55:03.0765257Z Uploading to s3 prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:55:03.0807316Z Starting upload of logs-test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_34566462667.zip 2024-12-18T02:55:03.2998466Z Finished upload of logs-test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_34566462667.zip 2024-12-18T02:55:03.3182523Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T02:55:03.3182937Z with: 2024-12-18T02:55:03.3183200Z s3-bucket: gha-artifacts 2024-12-18T02:55:03.3183596Z s3-prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:55:03.3184022Z retention-days: 14 2024-12-18T02:55:03.3184351Z if-no-files-found: ignore 2024-12-18T02:55:03.3184688Z path: debug-*.zip 2024-12-18T02:55:03.3184970Z name: artifact 2024-12-18T02:55:03.3185253Z region: us-east-1 2024-12-18T02:55:03.3185536Z env: 2024-12-18T02:55:03.3185799Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:03.3186207Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:03.3186850Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:03.3187453Z ##[endgroup] 2024-12-18T02:55:03.6553398Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-12-18T02:55:03.6749388Z ##[group]Run # shellcheck disable=SC2156 2024-12-18T02:55:03.6749844Z # shellcheck disable=SC2156 2024-12-18T02:55:03.6750664Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-12-18T02:55:03.6756802Z shell: /usr/bin/bash -e {0} 2024-12-18T02:55:03.6757109Z env: 2024-12-18T02:55:03.6757345Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:03.6757712Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:03.6758307Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:03.6758834Z ##[endgroup] 2024-12-18T02:55:03.9186172Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@release/2.6 2024-12-18T02:55:03.9186713Z with: 2024-12-18T02:55:03.9187106Z env: 2024-12-18T02:55:03.9187355Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:03.9187760Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:03.9188419Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:03.9189006Z ##[endgroup] 2024-12-18T02:55:03.9214615Z ##[group]Run set -eou pipefail 2024-12-18T02:55:03.9214980Z set -eou pipefail 2024-12-18T02:55:03.9215294Z  2024-12-18T02:55:03.9215701Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-12-18T02:55:03.9216225Z for _ in $(seq 1440); do 2024-12-18T02:55:03.9216609Z  # Break if no ssh session exists anymore 2024-12-18T02:55:03.9217008Z  if [ "$(who)" = "" ]; then 2024-12-18T02:55:03.9217349Z  break 2024-12-18T02:55:03.9217657Z  fi 2024-12-18T02:55:03.9217920Z  echo "." 2024-12-18T02:55:03.9218191Z  sleep 5 2024-12-18T02:55:03.9218474Z done 2024-12-18T02:55:03.9224826Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:55:03.9225281Z env: 2024-12-18T02:55:03.9225557Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:03.9225962Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:03.9226627Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:03.9227205Z ##[endgroup] 2024-12-18T02:55:03.9250673Z Holding runner for 2 hours until all ssh sessions have logged out 2024-12-18T02:55:03.9346917Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T02:55:03.9347585Z # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T02:55:03.9348106Z # shellcheck disable=SC2046 2024-12-18T02:55:03.9348518Z docker stop $(docker ps -q) || true 2024-12-18T02:55:03.9348941Z # Prune all of the docker images 2024-12-18T02:55:03.9349343Z docker system prune -af 2024-12-18T02:55:03.9355011Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:55:03.9355422Z env: 2024-12-18T02:55:03.9355668Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:55:03.9356032Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:55:03.9356629Z DOCKER_CONTAINER_ID: 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:03.9357156Z ##[endgroup] 2024-12-18T02:55:04.6333956Z 04018ea456d8 2024-12-18T02:55:05.1328658Z Deleted Containers: 2024-12-18T02:55:05.1329253Z 04018ea456d8f4a3a53d20d37f82e22cd0e763f4b65b218ecf4826f08d3cad0b 2024-12-18T02:55:05.1329790Z 2024-12-18T02:55:10.4673461Z Deleted Images: 2024-12-18T02:55:10.4675065Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T02:55:10.4676655Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9@sha256:52918bcdb0e370d1b9413099049464295e6c36e0a09cff6cff645b519e6da502 2024-12-18T02:55:10.4677916Z deleted: sha256:166dd14a00627fbd0d4e062a733b69b733b63b1ac12ec3bd5285db8a8befaaf5 2024-12-18T02:55:10.4678708Z deleted: sha256:2e828a4f851a71a08b9f06842643025da951b3d1a26bc40be945b5dc00f5015f 2024-12-18T02:55:10.4679454Z deleted: sha256:d27246a7406fc9214ce9db75df344bdd8e0a31300352beaf8ee0e0668a1ace5b 2024-12-18T02:55:10.4680205Z deleted: sha256:68c9fa6063c3f2de81e86b8de02d9c520f91cebed9343aecd84d4621681e108b 2024-12-18T02:55:10.4680959Z deleted: sha256:7fba561e060d45fd930ced565d615e1e1f5b5e1266212caaddcbdce3b6ce2157 2024-12-18T02:55:10.4681703Z deleted: sha256:5b5728232cece21786a91a2b7109215ddacbea370a452a5725759be3f9782f36 2024-12-18T02:55:10.4682418Z deleted: sha256:dc257cb01440638b647e8f88d00e791443a113161d0d4f469015690bab702c18 2024-12-18T02:55:10.4683463Z deleted: sha256:3ee1ba287c278eaf901c09c7e521f0a405bbb9fb8062aa51674fb9ea3ab18789 2024-12-18T02:55:10.4684216Z deleted: sha256:2d66661697c95587c70e4f77dc01731d10271a2a2d5ae52b240971a0554e6d0e 2024-12-18T02:55:10.4685103Z deleted: sha256:eaa304b04f52287b7bdefe4e276ed9444a9f2924aab757a4112d857162d00aae 2024-12-18T02:55:10.4685849Z deleted: sha256:4b999f756c8d5d7c8444b89e5466899b4bbd4f3adec20d788c443754a4cb2cf6 2024-12-18T02:55:10.4686678Z deleted: sha256:031bf045cac35b9a41c06ca4b85ca3419cf104dcaef931590111a4cbc99225d5 2024-12-18T02:55:10.4687584Z deleted: sha256:4a6da4e6593178794969f47a35ec4c667bd2a620051de682f4f6476200528ad7 2024-12-18T02:55:10.4688322Z deleted: sha256:d49018d0fd9477af622a59aae4a080650d82601c3ff10f5e413d54b61605cbc7 2024-12-18T02:55:10.4689191Z deleted: sha256:eaeb72d8e211a811359bbae6cdb944e506296e3593e02a7845d1de2b020db64d 2024-12-18T02:55:10.4690379Z deleted: sha256:4443a93278f4af70ebbc86c00611e1229086bcd73ba6f46cfbac959f5a3ea13e 2024-12-18T02:55:10.4691127Z deleted: sha256:7da4d3af5a970120b30198ce7c0d5758fd4180887250d2e1d20b1bb19a7ac558 2024-12-18T02:55:10.4691851Z deleted: sha256:51d1e0cf0c3b813663526e6d529bfb7f0374c2474f5a9581ccfd66f6dd722384 2024-12-18T02:55:10.4692597Z deleted: sha256:6d152e7f1f9932d4367ae21fbdcaf7e300d258e862c3c3104defca22e1f67f82 2024-12-18T02:55:10.4693337Z deleted: sha256:5f3d462f5b23f997262b6a535998f846bc071e1cbfb2039b507881fb8d873078 2024-12-18T02:55:10.4694075Z deleted: sha256:b5aa4897eb3166e0b560548c1ccbf716422a3b134bc63aa4da378db45a578506 2024-12-18T02:55:10.4694814Z deleted: sha256:19e8d1b6e7033c3cf97c2220cc96c9d4860bc5a09ad25b706a7260176331e59b 2024-12-18T02:55:10.4695552Z deleted: sha256:f82be94fab4a78f872875e44a2a54d01d6b7c7c9a73ad193593549037b0c9c6c 2024-12-18T02:55:10.4696286Z deleted: sha256:45e1395581d647e78b6fae987a785c08acea4b6176c97188b5a5010e9004d947 2024-12-18T02:55:10.4697012Z deleted: sha256:6a116d37e354cd470a6dee7600017fd2c240860da82a19939bd1451416af0972 2024-12-18T02:55:10.4697753Z deleted: sha256:96d8d1f880e7957a35c4060d1ae56eaa05c1e73ff5e7128cc5b42531a996ffeb 2024-12-18T02:55:10.4698625Z deleted: sha256:a186beea437cb709b244be99550467206b5bdcb190e032c091f15a6a065ddc36 2024-12-18T02:55:10.4699643Z deleted: sha256:e5f8bcff21a8fbb2d507150a82a1b26ebcec9caf9ee8fa7dce844574ef8da2ad 2024-12-18T02:55:10.4700417Z deleted: sha256:13091955acb9200fc139971ba2434c90657de11ff38ac954e35f3461fa4a6b8a 2024-12-18T02:55:10.4701175Z deleted: sha256:be10df8394bb253584f3f2f4c84a28bd53744e8c6892f1efcb4b12b67abe23da 2024-12-18T02:55:10.4701938Z deleted: sha256:ca58fcd63e3576eddbb8aaef07131e18631c53e4e9b6e8d9dbec5b8ddfa7726a 2024-12-18T02:55:10.4702710Z deleted: sha256:57c7d561a44e019b14a0f4aae3354ec1734b088b539cab6fb0e8ecca850ee714 2024-12-18T02:55:10.4703476Z deleted: sha256:a960411fa30719c90510a13403dcde638fbee3cbbb4ab71f14e52f40e9dd4533 2024-12-18T02:55:10.4704287Z deleted: sha256:8a2ef497a8ab2819802ad459d2dafe5195a85d11b38ef6d6d4d4fe872d9c6b6b 2024-12-18T02:55:10.4705065Z deleted: sha256:cbf589ae40cc4496ffa854c9108d58905a2ea5ac3162669b3b4606287b751d28 2024-12-18T02:55:10.4705826Z deleted: sha256:88c378f8d5104ea7a03bf49a1a7db4df53518519236345cf29bc2df76377d0ac 2024-12-18T02:55:10.4706583Z deleted: sha256:db2a844a68747daa51bb2b28c083df270c19bfbcb7e038723a1db1bacf20462e 2024-12-18T02:55:10.4707350Z deleted: sha256:f548fd4a27a545794a88afc9f9f3460001efb98677ae40f9737387bffedd901d 2024-12-18T02:55:10.4708117Z deleted: sha256:42f4cf4d61dad97d4c5f3bd5cb9374fe260e5166e36f4ae2d00a9c59127abb3a 2024-12-18T02:55:10.4708894Z deleted: sha256:1e465635bd263f73940e977a6d9ab8b3beceabc5eaa9d08c65a28aafdf72abf0 2024-12-18T02:55:10.4709656Z deleted: sha256:b890305c75102c7bd0a645b98b0e1893d45433319a1394c457afb7ffc3ac1070 2024-12-18T02:55:10.4710518Z deleted: sha256:c9d4902893199761be10cecc8c1de9ce1242b5df3c161bc452e5ffcf2d3bff38 2024-12-18T02:55:10.4711253Z deleted: sha256:254c14be940e16bbbe4d6da795ac3a987a11fde9842d614f31c1caae910c7263 2024-12-18T02:55:10.4711994Z deleted: sha256:137abe9e7f1c1726800d1ae709a40b92e48267f52dd7fa7b966be15345a94850 2024-12-18T02:55:10.4712866Z deleted: sha256:771d7f9e8bf6f532b16c3daa3a9167d4030f919eec29b18d99387ad1d1a277cb 2024-12-18T02:55:10.4713618Z deleted: sha256:22ee16d76a83cee720de0aafa52532ff1f3335a9048ff43877cee2aa6a370f5f 2024-12-18T02:55:10.4714446Z deleted: sha256:dbbad5d1e6d1785ee41424b56457a16a1d882ff1c09f41486b443b8c1e7c7eb4 2024-12-18T02:55:10.4715193Z deleted: sha256:df9a3657ce5f83f7ab775078ff74a472438fc5c21e1e0de53cc66509788e12bc 2024-12-18T02:55:10.4715923Z deleted: sha256:245eebe495c94ddc69ab83759089101d2edfdbf90474a3cb772e7bf1cce893bd 2024-12-18T02:55:10.4716667Z deleted: sha256:235e5b71d4fbb1eaf808e6fd789604752f653c1176b8acb1ba97604f35418396 2024-12-18T02:55:10.4717514Z deleted: sha256:c96d511cee1be7af8a8fedcab42cac5233cf66be05d188a2ba32ba4db1d2f715 2024-12-18T02:55:10.4718253Z deleted: sha256:704760c42e98a22e0f5ce40f10bdc3baec3e4ca679c2b046e90e6e12228a7164 2024-12-18T02:55:10.4718977Z deleted: sha256:78ec6b9f299f8114edc65d21c0c2e730f8423fcc496b819fe25df22af7477bd6 2024-12-18T02:55:10.4719709Z deleted: sha256:3d2201aa2c2a2db1e240c872ac8b84d4c92eed331c5c0053eaea4a8d292dd29a 2024-12-18T02:55:10.4720421Z deleted: sha256:2f8151f506d01c5be81b7547d7c20a36a2f30f6713adecaa91adb633fab9d809 2024-12-18T02:55:10.4721230Z deleted: sha256:d1f62ced0636553b623466ced7568917bb2b453997168ceb2635926f5c069ae6 2024-12-18T02:55:10.4721912Z deleted: sha256:bcb018d693dd3d8346bc43a253a42de37aad6ae11f42ee748dd9fdf1d3b259fc 2024-12-18T02:55:10.4722595Z deleted: sha256:40f224dcd4345d71537636d5c3bdb9c0659430771d9ac90c9e7cf7da30fabdd3 2024-12-18T02:55:10.4723265Z deleted: sha256:113e5e34c2859806f96efe4d47aa5e0d438e3f72211b7d539de8750bbfed7a98 2024-12-18T02:55:10.4723942Z deleted: sha256:a4786d6f71e2eec542ec8c4ff41191481de085b7e1b4e0139995017b6b7d6477 2024-12-18T02:55:10.4724611Z deleted: sha256:84faffca53886fb81b780b885baa8b6b4a4c16795bf93c9eed6f165c8d333545 2024-12-18T02:55:10.4725295Z deleted: sha256:22f20f90960f1663da6dbde7ea9b18026e0e1b03006890d56f64fc084a7550e3 2024-12-18T02:55:10.4725964Z deleted: sha256:d53c4a66cd75e088673985522b5ba099599960ac06350e59cfd330572327bff2 2024-12-18T02:55:10.4726644Z deleted: sha256:e95f4ec4da5fef7e54c22d270b51d9077a8c577a82bf0304e200fce309551084 2024-12-18T02:55:10.4727329Z deleted: sha256:ddbe476a0432522a73e2b052342ebc6f8610f7437c076ff6bf7e4501dc79efbc 2024-12-18T02:55:10.4728013Z deleted: sha256:ad7883a4c2954aebf555ea86e018fee1cc12d159b9e4d149263eb92c7cd21708 2024-12-18T02:55:10.4728693Z deleted: sha256:656ce67e45b8dab0cf77d1d21cd380f5ddfa64bfd486ebaa47b3af7c9b9df7b5 2024-12-18T02:55:10.4729389Z deleted: sha256:ae5754644efe96c6b01bbe838a7db9aac49333bf618c0a05160cdd0dd4b38fe8 2024-12-18T02:55:10.4730140Z deleted: sha256:23058ae084a55e5fd11db1aabe073f1886a1047be70eeb78dada3e6d4949c4d7 2024-12-18T02:55:10.4730942Z deleted: sha256:55c98b096fa43eb44df581de08d34a280f6eb359a04b1562fedccca5d56748e3 2024-12-18T02:55:10.4731999Z deleted: sha256:4a1d549deb2a6a9e8c52225d513aefeffbe0097e1eeb8217ca34963724db0b53 2024-12-18T02:55:10.4733207Z deleted: sha256:89e3b1c20a2d617b0ed103dc78296d410c1611d2044d8426fcc757b64c3113fe 2024-12-18T02:55:10.4734085Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-12-18T02:55:10.4735084Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-12-18T02:55:10.4735965Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-12-18T02:55:10.4736799Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-12-18T02:55:10.4737610Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-12-18T02:55:10.4738493Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-12-18T02:55:10.4739624Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-12-18T02:55:10.4740515Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-12-18T02:55:10.4741401Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-12-18T02:55:10.4742394Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-12-18T02:55:10.4743277Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-12-18T02:55:10.4743907Z 2024-12-18T02:55:10.4744089Z Total reclaimed space: 36.94GB 2024-12-18T02:55:10.4814260Z Post job cleanup. 2024-12-18T02:55:10.4867117Z Post job cleanup. 2024-12-18T02:55:10.5642046Z [command]/usr/bin/git version 2024-12-18T02:55:10.5677502Z git version 2.40.1 2024-12-18T02:55:10.5720501Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/e10c0e47-1730-442c-8843-2c3dcdf6c02d' before making global git config changes 2024-12-18T02:55:10.5721617Z Adding repository directory to the temporary git global config as a safe directory 2024-12-18T02:55:10.5723342Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T02:55:10.5753066Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-12-18T02:55:10.5780092Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-12-18T02:55:10.6091460Z Entering 'android/libs/fbjni' 2024-12-18T02:55:10.6148296Z Entering 'third_party/FP16' 2024-12-18T02:55:10.6205159Z Entering 'third_party/FXdiv' 2024-12-18T02:55:10.6260369Z Entering 'third_party/NNPACK' 2024-12-18T02:55:10.6317565Z Entering 'third_party/NVTX' 2024-12-18T02:55:10.6376182Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T02:55:10.6432493Z Entering 'third_party/XNNPACK' 2024-12-18T02:55:10.6504264Z Entering 'third_party/benchmark' 2024-12-18T02:55:10.6561697Z Entering 'third_party/composable_kernel' 2024-12-18T02:55:10.6624095Z Entering 'third_party/cpp-httplib' 2024-12-18T02:55:10.6681529Z Entering 'third_party/cpuinfo' 2024-12-18T02:55:10.6739533Z Entering 'third_party/cudnn_frontend' 2024-12-18T02:55:10.6797675Z Entering 'third_party/cutlass' 2024-12-18T02:55:10.6860898Z Entering 'third_party/eigen' 2024-12-18T02:55:10.6921937Z Entering 'third_party/fbgemm' 2024-12-18T02:55:10.6979748Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T02:55:10.7037450Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T02:55:10.7098370Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T02:55:10.7159306Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T02:55:10.7214689Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T02:55:10.7278237Z Entering 'third_party/flatbuffers' 2024-12-18T02:55:10.7336171Z Entering 'third_party/fmt' 2024-12-18T02:55:10.7394858Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T02:55:10.7451479Z Entering 'third_party/gloo' 2024-12-18T02:55:10.7506169Z Entering 'third_party/googletest' 2024-12-18T02:55:10.7564155Z Entering 'third_party/ideep' 2024-12-18T02:55:10.7620192Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T02:55:10.7685465Z Entering 'third_party/ittapi' 2024-12-18T02:55:10.7741325Z Entering 'third_party/kineto' 2024-12-18T02:55:10.7799009Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T02:55:10.7853697Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T02:55:10.7918164Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T02:55:10.7974710Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T02:55:10.8031241Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T02:55:10.8087642Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T02:55:10.8154470Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T02:55:10.8220962Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T02:55:10.8278736Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T02:55:10.8335942Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T02:55:10.8404645Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T02:55:10.8459522Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T02:55:10.8518687Z Entering 'third_party/mimalloc' 2024-12-18T02:55:10.8577297Z Entering 'third_party/nccl/nccl' 2024-12-18T02:55:10.8633744Z Entering 'third_party/nlohmann' 2024-12-18T02:55:10.8693278Z Entering 'third_party/onnx' 2024-12-18T02:55:10.8766533Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T02:55:10.8826150Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T02:55:10.8883994Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T02:55:10.8940220Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T02:55:10.8996361Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T02:55:10.9050937Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T02:55:10.9108014Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T02:55:10.9164670Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T02:55:10.9220785Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T02:55:10.9278555Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T02:55:10.9337467Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T02:55:10.9398082Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T02:55:10.9475805Z Entering 'third_party/pocketfft' 2024-12-18T02:55:10.9535098Z Entering 'third_party/protobuf' 2024-12-18T02:55:10.9595646Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T02:55:10.9650925Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T02:55:10.9707335Z Entering 'third_party/psimd' 2024-12-18T02:55:10.9765329Z Entering 'third_party/pthreadpool' 2024-12-18T02:55:10.9822391Z Entering 'third_party/pybind11' 2024-12-18T02:55:10.9880391Z Entering 'third_party/python-peachpy' 2024-12-18T02:55:10.9936428Z Entering 'third_party/sleef' 2024-12-18T02:55:10.9995266Z Entering 'third_party/tensorpipe' 2024-12-18T02:55:11.0051541Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T02:55:11.0112475Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T02:55:11.0165536Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T02:55:11.0230249Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T02:55:11.0287492Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T02:55:11.0362411Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-12-18T02:55:11.0380828Z http.https://github.com/.extraheader 2024-12-18T02:55:11.0389704Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-12-18T02:55:11.0415871Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-12-18T02:55:11.0723290Z Entering 'android/libs/fbjni' 2024-12-18T02:55:11.0761825Z http.https://github.com/.extraheader 2024-12-18T02:55:11.0798182Z Entering 'third_party/FP16' 2024-12-18T02:55:11.0835539Z http.https://github.com/.extraheader 2024-12-18T02:55:11.0876484Z Entering 'third_party/FXdiv' 2024-12-18T02:55:11.0914675Z http.https://github.com/.extraheader 2024-12-18T02:55:11.0948764Z Entering 'third_party/NNPACK' 2024-12-18T02:55:11.0988440Z http.https://github.com/.extraheader 2024-12-18T02:55:11.1021772Z Entering 'third_party/NVTX' 2024-12-18T02:55:11.1060761Z http.https://github.com/.extraheader 2024-12-18T02:55:11.1101856Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T02:55:11.1139697Z http.https://github.com/.extraheader 2024-12-18T02:55:11.1181453Z Entering 'third_party/XNNPACK' 2024-12-18T02:55:11.1220511Z http.https://github.com/.extraheader 2024-12-18T02:55:11.1270424Z Entering 'third_party/benchmark' 2024-12-18T02:55:11.1308392Z http.https://github.com/.extraheader 2024-12-18T02:55:11.1343813Z Entering 'third_party/composable_kernel' 2024-12-18T02:55:11.1380979Z http.https://github.com/.extraheader 2024-12-18T02:55:11.1427547Z Entering 'third_party/cpp-httplib' 2024-12-18T02:55:11.1464706Z http.https://github.com/.extraheader 2024-12-18T02:55:11.1503206Z Entering 'third_party/cpuinfo' 2024-12-18T02:55:11.1540644Z http.https://github.com/.extraheader 2024-12-18T02:55:11.1579000Z Entering 'third_party/cudnn_frontend' 2024-12-18T02:55:11.1616737Z http.https://github.com/.extraheader 2024-12-18T02:55:11.1652489Z Entering 'third_party/cutlass' 2024-12-18T02:55:11.1691118Z http.https://github.com/.extraheader 2024-12-18T02:55:11.1734236Z Entering 'third_party/eigen' 2024-12-18T02:55:11.1773212Z http.https://github.com/.extraheader 2024-12-18T02:55:11.1812180Z Entering 'third_party/fbgemm' 2024-12-18T02:55:11.1851784Z http.https://github.com/.extraheader 2024-12-18T02:55:11.1887149Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T02:55:11.1924872Z http.https://github.com/.extraheader 2024-12-18T02:55:11.1960183Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T02:55:11.1996870Z http.https://github.com/.extraheader 2024-12-18T02:55:11.2031431Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T02:55:11.2068895Z http.https://github.com/.extraheader 2024-12-18T02:55:11.2110660Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T02:55:11.2147232Z http.https://github.com/.extraheader 2024-12-18T02:55:11.2182246Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T02:55:11.2220742Z http.https://github.com/.extraheader 2024-12-18T02:55:11.2256127Z Entering 'third_party/flatbuffers' 2024-12-18T02:55:11.2295414Z http.https://github.com/.extraheader 2024-12-18T02:55:11.2334057Z Entering 'third_party/fmt' 2024-12-18T02:55:11.2373107Z http.https://github.com/.extraheader 2024-12-18T02:55:11.2409731Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T02:55:11.2450213Z http.https://github.com/.extraheader 2024-12-18T02:55:11.2484756Z Entering 'third_party/gloo' 2024-12-18T02:55:11.2523468Z http.https://github.com/.extraheader 2024-12-18T02:55:11.2558681Z Entering 'third_party/googletest' 2024-12-18T02:55:11.2596973Z http.https://github.com/.extraheader 2024-12-18T02:55:11.2631775Z Entering 'third_party/ideep' 2024-12-18T02:55:11.2671128Z http.https://github.com/.extraheader 2024-12-18T02:55:11.2714563Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T02:55:11.2750107Z http.https://github.com/.extraheader 2024-12-18T02:55:11.2803440Z Entering 'third_party/ittapi' 2024-12-18T02:55:11.2842196Z http.https://github.com/.extraheader 2024-12-18T02:55:11.2882690Z Entering 'third_party/kineto' 2024-12-18T02:55:11.2920636Z http.https://github.com/.extraheader 2024-12-18T02:55:11.2955895Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T02:55:11.2994850Z http.https://github.com/.extraheader 2024-12-18T02:55:11.3032650Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T02:55:11.3071061Z http.https://github.com/.extraheader 2024-12-18T02:55:11.3115847Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T02:55:11.3154252Z http.https://github.com/.extraheader 2024-12-18T02:55:11.3189109Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T02:55:11.3227546Z http.https://github.com/.extraheader 2024-12-18T02:55:11.3263400Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T02:55:11.3300369Z http.https://github.com/.extraheader 2024-12-18T02:55:11.3338724Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T02:55:11.3376962Z http.https://github.com/.extraheader 2024-12-18T02:55:11.3419169Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T02:55:11.3457162Z http.https://github.com/.extraheader 2024-12-18T02:55:11.3495323Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T02:55:11.3533977Z http.https://github.com/.extraheader 2024-12-18T02:55:11.3569130Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T02:55:11.3607474Z http.https://github.com/.extraheader 2024-12-18T02:55:11.3643199Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T02:55:11.3683351Z http.https://github.com/.extraheader 2024-12-18T02:55:11.3721576Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T02:55:11.3761086Z http.https://github.com/.extraheader 2024-12-18T02:55:11.3798592Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T02:55:11.3834103Z http.https://github.com/.extraheader 2024-12-18T02:55:11.3878562Z Entering 'third_party/mimalloc' 2024-12-18T02:55:11.3917188Z http.https://github.com/.extraheader 2024-12-18T02:55:11.3953231Z Entering 'third_party/nccl/nccl' 2024-12-18T02:55:11.3989255Z http.https://github.com/.extraheader 2024-12-18T02:55:11.4027610Z Entering 'third_party/nlohmann' 2024-12-18T02:55:11.4064652Z http.https://github.com/.extraheader 2024-12-18T02:55:11.4104071Z Entering 'third_party/onnx' 2024-12-18T02:55:11.4141717Z http.https://github.com/.extraheader 2024-12-18T02:55:11.4195030Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T02:55:11.4230746Z http.https://github.com/.extraheader 2024-12-18T02:55:11.4276430Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T02:55:11.4315389Z http.https://github.com/.extraheader 2024-12-18T02:55:11.4354402Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T02:55:11.4389617Z http.https://github.com/.extraheader 2024-12-18T02:55:11.4431906Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T02:55:11.4468501Z http.https://github.com/.extraheader 2024-12-18T02:55:11.4510539Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T02:55:11.4547147Z http.https://github.com/.extraheader 2024-12-18T02:55:11.4581140Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T02:55:11.4619638Z http.https://github.com/.extraheader 2024-12-18T02:55:11.4655777Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T02:55:11.4693291Z http.https://github.com/.extraheader 2024-12-18T02:55:11.4729042Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T02:55:11.4766804Z http.https://github.com/.extraheader 2024-12-18T02:55:11.4801215Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T02:55:11.4836729Z http.https://github.com/.extraheader 2024-12-18T02:55:11.4876586Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T02:55:11.4914691Z http.https://github.com/.extraheader 2024-12-18T02:55:11.4953960Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T02:55:11.4989417Z http.https://github.com/.extraheader 2024-12-18T02:55:11.5032075Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T02:55:11.5069644Z http.https://github.com/.extraheader 2024-12-18T02:55:11.5135092Z Entering 'third_party/pocketfft' 2024-12-18T02:55:11.5173683Z http.https://github.com/.extraheader 2024-12-18T02:55:11.5209982Z Entering 'third_party/protobuf' 2024-12-18T02:55:11.5249550Z http.https://github.com/.extraheader 2024-12-18T02:55:11.5289574Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T02:55:11.5327401Z http.https://github.com/.extraheader 2024-12-18T02:55:11.5362824Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T02:55:11.5401400Z http.https://github.com/.extraheader 2024-12-18T02:55:11.5437547Z Entering 'third_party/psimd' 2024-12-18T02:55:11.5477160Z http.https://github.com/.extraheader 2024-12-18T02:55:11.5514414Z Entering 'third_party/pthreadpool' 2024-12-18T02:55:11.5554234Z http.https://github.com/.extraheader 2024-12-18T02:55:11.5587992Z Entering 'third_party/pybind11' 2024-12-18T02:55:11.5624664Z http.https://github.com/.extraheader 2024-12-18T02:55:11.5660513Z Entering 'third_party/python-peachpy' 2024-12-18T02:55:11.5698136Z http.https://github.com/.extraheader 2024-12-18T02:55:11.5735341Z Entering 'third_party/sleef' 2024-12-18T02:55:11.5773988Z http.https://github.com/.extraheader 2024-12-18T02:55:11.5810086Z Entering 'third_party/tensorpipe' 2024-12-18T02:55:11.5849738Z http.https://github.com/.extraheader 2024-12-18T02:55:11.5885031Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T02:55:11.5923618Z http.https://github.com/.extraheader 2024-12-18T02:55:11.5959276Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T02:55:11.5996583Z http.https://github.com/.extraheader 2024-12-18T02:55:11.6034174Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T02:55:11.6073240Z http.https://github.com/.extraheader 2024-12-18T02:55:11.6110428Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T02:55:11.6147038Z http.https://github.com/.extraheader 2024-12-18T02:55:11.6180926Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T02:55:11.6219356Z http.https://github.com/.extraheader 2024-12-18T02:55:11.6353702Z A job completed hook has been configured by the self-hosted runner administrator 2024-12-18T02:55:11.6379972Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-12-18T02:55:11.6385892Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:55:11.6386313Z ##[endgroup] 2024-12-18T02:55:11.6467514Z [!ALERT!] Swap in detected! [!ALERT!] 2024-12-18T02:55:23.4223084Z [!ALERT!] Swap out detected [!ALERT!] 2024-12-18T02:55:41.7755743Z Cleaning up orphan processes