2024-12-17T23:58:03.3323630Z Current runner version: '2.321.0' 2024-12-17T23:58:03.3331417Z Runner name: 'i-0c6125b7fa38a3d09' 2024-12-17T23:58:03.3332402Z Runner group name: 'Default' 2024-12-17T23:58:03.3333486Z Machine name: 'ip-10-0-46-48' 2024-12-17T23:58:03.3338948Z ##[group]GITHUB_TOKEN Permissions 2024-12-17T23:58:03.3341627Z Actions: read 2024-12-17T23:58:03.3342309Z Attestations: read 2024-12-17T23:58:03.3343001Z Checks: read 2024-12-17T23:58:03.3343657Z Contents: read 2024-12-17T23:58:03.3344248Z Deployments: read 2024-12-17T23:58:03.3344949Z Discussions: read 2024-12-17T23:58:03.3345622Z Issues: read 2024-12-17T23:58:03.3346256Z Metadata: read 2024-12-17T23:58:03.3346923Z Packages: read 2024-12-17T23:58:03.3347518Z Pages: read 2024-12-17T23:58:03.3348146Z PullRequests: read 2024-12-17T23:58:03.3348856Z RepositoryProjects: read 2024-12-17T23:58:03.3349572Z SecurityEvents: read 2024-12-17T23:58:03.3350230Z Statuses: read 2024-12-17T23:58:03.3350859Z ##[endgroup] 2024-12-17T23:58:03.3354253Z Secret source: Actions 2024-12-17T23:58:03.3355363Z Prepare workflow directory 2024-12-17T23:58:03.3893784Z Prepare all required actions 2024-12-17T23:58:03.3935826Z Getting action download info 2024-12-17T23:58:03.6349610Z Download action repository 'pytorch/test-infra@release/2.6' (SHA:eb0adf5a84668865394af69e26428b32c8105c1c) 2024-12-17T23:58:05.0424492Z Download action repository 'pytorch/pytorch@release/2.6' (SHA:0cdf8b1d09254cfda66191d1bd01e3041c3c76f7) 2024-12-17T23:58:18.2723668Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-12-17T23:58:18.5356314Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-12-17T23:58:18.8198615Z Getting action download info 2024-12-17T23:58:18.9695671Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-12-17T23:58:19.1833808Z Getting action download info 2024-12-17T23:58:19.4365107Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2024-12-17T23:58:19.5971942Z Getting action download info 2024-12-17T23:58:19.7348528Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-12-17T23:58:19.8983424Z Getting action download info 2024-12-17T23:58:20.0614298Z Download action repository 'pytorch/test-infra@main' (SHA:a07505a74641a4ff5123d635defac481ef28ef1e) 2024-12-17T23:58:21.6641235Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/release/2.6 (0cdf8b1d09254cfda66191d1bd01e3041c3c76f7) 2024-12-17T23:58:21.6643872Z ##[group] Inputs 2024-12-17T23:58:21.6644388Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-12-17T23:58:21.6646043Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]} 2024-12-17T23:58:21.6648116Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-17T23:58:21.6649061Z sync-tag: 2024-12-17T23:58:21.6649956Z timeout-minutes: 360 2024-12-17T23:58:21.6650271Z use-gha: 2024-12-17T23:58:21.6650550Z dashboard-tag: 2024-12-17T23:58:21.6650852Z s3-bucket: gha-artifacts 2024-12-17T23:58:21.6651190Z aws-role-to-assume: 2024-12-17T23:58:21.6651995Z disable-monitor: false 2024-12-17T23:58:21.6652324Z ##[endgroup] 2024-12-17T23:58:21.6652939Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-17T23:58:21.7336723Z A job started hook has been configured by the self-hosted runner administrator 2024-12-17T23:58:21.7444569Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-12-17T23:58:21.7455077Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-17T23:58:21.7455937Z ##[endgroup] 2024-12-17T23:58:22.8684759Z Runner Type: linux.g4dn.12xlarge.nvidia.gpu 2024-12-17T23:58:22.8685403Z Instance Type: g4dn.12xlarge 2024-12-17T23:58:22.8685750Z AMI Name: unknown 2024-12-17T23:58:22.8716009Z AMI ID: ami-0fff1b9a61dec8a5f 2024-12-17T23:58:28.7399618Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@release/2.6 2024-12-17T23:58:28.7400131Z with: 2024-12-17T23:58:28.7400822Z github-secret: *** 2024-12-17T23:58:28.7401576Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-12-17T23:58:28.7402402Z activate-with-label: false 2024-12-17T23:58:28.7402706Z label: with-ssh 2024-12-17T23:58:28.7402974Z remove-existing-keys: true 2024-12-17T23:58:28.7403394Z fail-silently: true 2024-12-17T23:58:28.7403886Z env: 2024-12-17T23:58:28.7404147Z GIT_DEFAULT_BRANCH: main 2024-12-17T23:58:28.7404502Z ##[endgroup] 2024-12-17T23:58:28.8645555Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-12-17T23:58:28.8646613Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2024-12-17T23:58:28.8781803Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@release/2.6 2024-12-17T23:58:28.8782305Z with: 2024-12-17T23:58:28.8782528Z no-sudo: true 2024-12-17T23:58:28.8782794Z submodules: recursive 2024-12-17T23:58:28.8783073Z fetch-depth: 0 2024-12-17T23:58:28.8783326Z env: 2024-12-17T23:58:28.8783546Z GIT_DEFAULT_BRANCH: main 2024-12-17T23:58:28.8783832Z ##[endgroup] 2024-12-17T23:58:28.8874975Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-17T23:58:28.8876102Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-17T23:58:28.8885166Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-17T23:58:28.8885635Z env: 2024-12-17T23:58:28.8885884Z GIT_DEFAULT_BRANCH: main 2024-12-17T23:58:28.8886206Z ##[endgroup] 2024-12-17T23:58:28.8992922Z ##[group]Run retry () { 2024-12-17T23:58:28.8993246Z retry () { 2024-12-17T23:58:28.8993630Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-12-17T23:58:28.8994082Z } 2024-12-17T23:58:28.8994340Z echo "${GITHUB_WORKSPACE}" 2024-12-17T23:58:28.8994689Z if [ -z "${NO_SUDO}" ]; then 2024-12-17T23:58:28.8995066Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-12-17T23:58:28.8995419Z else 2024-12-17T23:58:28.8995703Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-12-17T23:58:28.8996053Z fi 2024-12-17T23:58:28.8996308Z mkdir "${GITHUB_WORKSPACE}" 2024-12-17T23:58:28.9001763Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-17T23:58:28.9002187Z env: 2024-12-17T23:58:28.9002412Z GIT_DEFAULT_BRANCH: main 2024-12-17T23:58:28.9002703Z NO_SUDO: true 2024-12-17T23:58:28.9002949Z ##[endgroup] 2024-12-17T23:58:28.9028429Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-17T23:58:28.9124009Z ##[group]Run malfet/checkout@silent-checkout 2024-12-17T23:58:28.9124451Z with: 2024-12-17T23:58:28.9124785Z ref: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-17T23:58:28.9125183Z fetch-depth: 0 2024-12-17T23:58:28.9125473Z submodules: recursive 2024-12-17T23:58:28.9125774Z quiet-checkout: true 2024-12-17T23:58:28.9126097Z repository: pytorch/pytorch 2024-12-17T23:58:28.9126576Z token: *** 2024-12-17T23:58:28.9126847Z ssh-strict: true 2024-12-17T23:58:28.9127147Z persist-credentials: true 2024-12-17T23:58:28.9127480Z clean: true 2024-12-17T23:58:28.9127762Z sparse-checkout-cone-mode: true 2024-12-17T23:58:28.9128120Z lfs: false 2024-12-17T23:58:28.9128397Z set-safe-directory: true 2024-12-17T23:58:28.9130222Z env: 2024-12-17T23:58:28.9130472Z GIT_DEFAULT_BRANCH: main 2024-12-17T23:58:28.9130790Z ##[endgroup] 2024-12-17T23:58:29.0043668Z Syncing repository: pytorch/pytorch 2024-12-17T23:58:29.0045215Z ##[group]Getting Git version info 2024-12-17T23:58:29.0045822Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-12-17T23:58:29.0046617Z [command]/usr/bin/git version 2024-12-17T23:58:29.0046969Z git version 2.40.1 2024-12-17T23:58:29.0055396Z ##[endgroup] 2024-12-17T23:58:29.0069682Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/6e471baa-5e93-4be5-94d0-05ff9176a9ba' before making global git config changes 2024-12-17T23:58:29.0070817Z Adding repository directory to the temporary git global config as a safe directory 2024-12-17T23:58:29.0073569Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-17T23:58:29.0102137Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-12-17T23:58:29.0105653Z ##[group]Initializing the repository 2024-12-17T23:58:29.0107999Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-17T23:58:29.0132847Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-12-17T23:58:29.0133577Z hint: is subject to change. To configure the initial branch name to use in all 2024-12-17T23:58:29.0134264Z hint: of your new repositories, which will suppress this warning, call: 2024-12-17T23:58:29.0134769Z hint: 2024-12-17T23:58:29.0135122Z hint: git config --global init.defaultBranch 2024-12-17T23:58:29.0135658Z hint: 2024-12-17T23:58:29.0136039Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-12-17T23:58:29.0136708Z hint: 'development'. The just-created branch can be renamed via this command: 2024-12-17T23:58:29.0137225Z hint: 2024-12-17T23:58:29.0137486Z hint: git branch -m 2024-12-17T23:58:29.0138086Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-12-17T23:58:29.0142912Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-12-17T23:58:29.0167495Z ##[endgroup] 2024-12-17T23:58:29.0168066Z ##[group]Disabling automatic garbage collection 2024-12-17T23:58:29.0170014Z [command]/usr/bin/git config --local gc.auto 0 2024-12-17T23:58:29.0195086Z ##[endgroup] 2024-12-17T23:58:29.0195593Z ##[group]Setting up auth 2024-12-17T23:58:29.0197168Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-12-17T23:58:29.0220574Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-12-17T23:58:29.0530692Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-12-17T23:58:29.0552371Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-12-17T23:58:29.0837304Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-12-17T23:58:29.0883345Z ##[endgroup] 2024-12-17T23:58:29.0884087Z ##[group]Fetching the repository 2024-12-17T23:58:29.0887971Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-12-17T23:58:32.0724683Z remote: Enumerating objects: 1056559 2024-12-17T23:58:32.0726029Z remote: Enumerating objects: 1057261, done. 2024-12-17T23:58:32.0727160Z remote: Counting objects: 0% (1/702) 2024-12-17T23:58:32.0728241Z remote: Counting objects: 1% (8/702) 2024-12-17T23:58:32.0729317Z remote: Counting objects: 2% (15/702) 2024-12-17T23:58:32.0730407Z remote: Counting objects: 3% (22/702) 2024-12-17T23:58:32.0731265Z remote: Counting objects: 4% (29/702) 2024-12-17T23:58:32.0731700Z remote: Counting objects: 5% (36/702) 2024-12-17T23:58:32.0732121Z remote: Counting objects: 6% (43/702) 2024-12-17T23:58:32.0732577Z remote: Counting objects: 7% (50/702) 2024-12-17T23:58:32.0733011Z remote: Counting objects: 8% (57/702) 2024-12-17T23:58:32.0733441Z remote: Counting objects: 9% (64/702) 2024-12-17T23:58:32.0733874Z remote: Counting objects: 10% (71/702) 2024-12-17T23:58:32.0734294Z remote: Counting objects: 11% (78/702) 2024-12-17T23:58:32.0734775Z remote: Counting objects: 12% (85/702) 2024-12-17T23:58:32.0735206Z remote: Counting objects: 13% (92/702) 2024-12-17T23:58:32.0735634Z remote: Counting objects: 14% (99/702) 2024-12-17T23:58:32.0736176Z remote: Counting objects: 15% (106/702) 2024-12-17T23:58:32.0736603Z remote: Counting objects: 16% (113/702) 2024-12-17T23:58:32.0737035Z remote: Counting objects: 17% (120/702) 2024-12-17T23:58:32.0737445Z remote: Counting objects: 18% (127/702) 2024-12-17T23:58:32.0737869Z remote: Counting objects: 19% (134/702) 2024-12-17T23:58:32.0738417Z remote: Counting objects: 20% (141/702) 2024-12-17T23:58:32.0738845Z remote: Counting objects: 21% (148/702) 2024-12-17T23:58:32.0773722Z remote: Counting objects: 22% (155/702) 2024-12-17T23:58:32.0774203Z remote: Counting objects: 23% (162/702) 2024-12-17T23:58:32.0774654Z remote: Counting objects: 24% (169/702) 2024-12-17T23:58:32.0775116Z remote: Counting objects: 25% (176/702) 2024-12-17T23:58:32.0775656Z remote: Counting objects: 26% (183/702) 2024-12-17T23:58:32.0776139Z remote: Counting objects: 27% (190/702) 2024-12-17T23:58:32.0776573Z remote: Counting objects: 28% (197/702) 2024-12-17T23:58:32.0777103Z remote: Counting objects: 29% (204/702) 2024-12-17T23:58:32.0777538Z remote: Counting objects: 30% (211/702) 2024-12-17T23:58:32.0777956Z remote: Counting objects: 31% (218/702) 2024-12-17T23:58:32.0778358Z remote: Counting objects: 32% (225/702) 2024-12-17T23:58:32.0778779Z remote: Counting objects: 33% (232/702) 2024-12-17T23:58:32.0779194Z remote: Counting objects: 34% (239/702) 2024-12-17T23:58:32.0779610Z remote: Counting objects: 35% (246/702) 2024-12-17T23:58:32.0780025Z remote: Counting objects: 36% (253/702) 2024-12-17T23:58:32.0780430Z remote: Counting objects: 37% (260/702) 2024-12-17T23:58:32.0780847Z remote: Counting objects: 38% (267/702) 2024-12-17T23:58:32.0781260Z remote: Counting objects: 39% (274/702) 2024-12-17T23:58:32.0781673Z remote: Counting objects: 40% (281/702) 2024-12-17T23:58:32.0782087Z remote: Counting objects: 41% (288/702) 2024-12-17T23:58:32.0782487Z remote: Counting objects: 42% (295/702) 2024-12-17T23:58:32.0782908Z remote: Counting objects: 43% (302/702) 2024-12-17T23:58:32.0783325Z remote: Counting objects: 44% (309/702) 2024-12-17T23:58:32.0783765Z remote: Counting objects: 45% (316/702) 2024-12-17T23:58:32.0784248Z remote: Counting objects: 46% (323/702) 2024-12-17T23:58:32.0784660Z remote: Counting objects: 47% (330/702) 2024-12-17T23:58:32.0785061Z remote: Counting objects: 48% (337/702) 2024-12-17T23:58:32.0785473Z remote: Counting objects: 49% (344/702) 2024-12-17T23:58:32.0785886Z remote: Counting objects: 50% (351/702) 2024-12-17T23:58:32.0786299Z remote: Counting objects: 51% (359/702) 2024-12-17T23:58:32.0786713Z remote: Counting objects: 52% (366/702) 2024-12-17T23:58:32.0787111Z remote: Counting objects: 53% (373/702) 2024-12-17T23:58:32.0787525Z remote: Counting objects: 54% (380/702) 2024-12-17T23:58:32.0787940Z remote: Counting objects: 55% (387/702) 2024-12-17T23:58:32.0788536Z remote: Counting objects: 56% (394/702) 2024-12-17T23:58:32.0788951Z remote: Counting objects: 57% (401/702) 2024-12-17T23:58:32.0789352Z remote: Counting objects: 58% (408/702) 2024-12-17T23:58:32.0789763Z remote: Counting objects: 59% (415/702) 2024-12-17T23:58:32.0790186Z remote: Counting objects: 60% (422/702) 2024-12-17T23:58:32.0790602Z remote: Counting objects: 61% (429/702) 2024-12-17T23:58:32.0791193Z remote: Counting objects: 62% (436/702) 2024-12-17T23:58:32.0791752Z remote: Counting objects: 63% (443/702) 2024-12-17T23:58:32.0792167Z remote: Counting objects: 64% (450/702) 2024-12-17T23:58:32.0792567Z remote: Counting objects: 65% (457/702) 2024-12-17T23:58:32.0792981Z remote: Counting objects: 66% (464/702) 2024-12-17T23:58:32.0793395Z remote: Counting objects: 67% (471/702) 2024-12-17T23:58:32.0793810Z remote: Counting objects: 68% (478/702) 2024-12-17T23:58:32.0794231Z remote: Counting objects: 69% (485/702) 2024-12-17T23:58:32.0794632Z remote: Counting objects: 70% (492/702) 2024-12-17T23:58:32.0795048Z remote: Counting objects: 71% (499/702) 2024-12-17T23:58:32.0795463Z remote: Counting objects: 72% (506/702) 2024-12-17T23:58:32.0795972Z remote: Counting objects: 73% (513/702) 2024-12-17T23:58:32.0796391Z remote: Counting objects: 74% (520/702) 2024-12-17T23:58:32.0796794Z remote: Counting objects: 75% (527/702) 2024-12-17T23:58:32.0797209Z remote: Counting objects: 76% (534/702) 2024-12-17T23:58:32.0797624Z remote: Counting objects: 77% (541/702) 2024-12-17T23:58:32.0798036Z remote: Counting objects: 78% (548/702) 2024-12-17T23:58:32.0798450Z remote: Counting objects: 79% (555/702) 2024-12-17T23:58:32.0798864Z remote: Counting objects: 80% (562/702) 2024-12-17T23:58:32.0799266Z remote: Counting objects: 81% (569/702) 2024-12-17T23:58:32.0799681Z remote: Counting objects: 82% (576/702) 2024-12-17T23:58:32.0800100Z remote: Counting objects: 83% (583/702) 2024-12-17T23:58:32.0800513Z remote: Counting objects: 84% (590/702) 2024-12-17T23:58:32.0800924Z remote: Counting objects: 85% (597/702) 2024-12-17T23:58:32.0801329Z remote: Counting objects: 86% (604/702) 2024-12-17T23:58:32.0801741Z remote: Counting objects: 87% (611/702) 2024-12-17T23:58:32.0802158Z remote: Counting objects: 88% (618/702) 2024-12-17T23:58:32.0802571Z remote: Counting objects: 89% (625/702) 2024-12-17T23:58:32.0802987Z remote: Counting objects: 90% (632/702) 2024-12-17T23:58:32.0803667Z remote: Counting objects: 91% (639/702) 2024-12-17T23:58:32.0804134Z remote: Counting objects: 92% (646/702) 2024-12-17T23:58:32.0804573Z remote: Counting objects: 93% (653/702) 2024-12-17T23:58:32.0805008Z remote: Counting objects: 94% (660/702) 2024-12-17T23:58:32.0805444Z remote: Counting objects: 95% (667/702) 2024-12-17T23:58:32.0805888Z remote: Counting objects: 96% (674/702) 2024-12-17T23:58:32.0806311Z remote: Counting objects: 97% (681/702) 2024-12-17T23:58:32.0806751Z remote: Counting objects: 98% (688/702) 2024-12-17T23:58:32.0807192Z remote: Counting objects: 99% (695/702) 2024-12-17T23:58:32.0807632Z remote: Counting objects: 100% (702/702) 2024-12-17T23:58:32.0808105Z remote: Counting objects: 100% (702/702), done. 2024-12-17T23:58:32.0829931Z remote: Compressing objects: 0% (1/334) 2024-12-17T23:58:32.0896605Z remote: Compressing objects: 1% (4/334) 2024-12-17T23:58:32.1073704Z remote: Compressing objects: 2% (7/334) 2024-12-17T23:58:32.1232097Z remote: Compressing objects: 3% (11/334) 2024-12-17T23:58:32.1263543Z remote: Compressing objects: 4% (14/334) 2024-12-17T23:58:32.1383685Z remote: Compressing objects: 5% (17/334) 2024-12-17T23:58:32.1439645Z remote: Compressing objects: 6% (21/334) 2024-12-17T23:58:32.1580551Z remote: Compressing objects: 7% (24/334) 2024-12-17T23:58:32.2046271Z remote: Compressing objects: 8% (27/334) 2024-12-17T23:58:32.2529117Z remote: Compressing objects: 9% (31/334) 2024-12-17T23:58:32.3211376Z remote: Compressing objects: 10% (34/334) 2024-12-17T23:58:32.3849424Z remote: Compressing objects: 11% (37/334) 2024-12-17T23:58:32.4067669Z remote: Compressing objects: 12% (41/334) 2024-12-17T23:58:32.4169086Z remote: Compressing objects: 13% (44/334) 2024-12-17T23:58:32.4219355Z remote: Compressing objects: 14% (47/334) 2024-12-17T23:58:32.4219856Z remote: Compressing objects: 15% (51/334) 2024-12-17T23:58:32.4220314Z remote: Compressing objects: 16% (54/334) 2024-12-17T23:58:32.4222774Z remote: Compressing objects: 17% (57/334) 2024-12-17T23:58:32.4226824Z remote: Compressing objects: 18% (61/334) 2024-12-17T23:58:32.4230147Z remote: Compressing objects: 19% (64/334) 2024-12-17T23:58:32.4232792Z remote: Compressing objects: 20% (67/334) 2024-12-17T23:58:32.4233224Z remote: Compressing objects: 21% (71/334) 2024-12-17T23:58:32.4235159Z remote: Compressing objects: 22% (74/334) 2024-12-17T23:58:32.4238114Z remote: Compressing objects: 23% (77/334) 2024-12-17T23:58:32.4241622Z remote: Compressing objects: 24% (81/334) 2024-12-17T23:58:32.4246468Z remote: Compressing objects: 25% (84/334) 2024-12-17T23:58:32.4249007Z remote: Compressing objects: 26% (87/334) 2024-12-17T23:58:32.4251687Z remote: Compressing objects: 27% (91/334) 2024-12-17T23:58:32.4252127Z remote: Compressing objects: 28% (94/334) 2024-12-17T23:58:32.4256223Z remote: Compressing objects: 29% (97/334) 2024-12-17T23:58:32.4263799Z remote: Compressing objects: 30% (101/334) 2024-12-17T23:58:32.4271063Z remote: Compressing objects: 31% (104/334) 2024-12-17T23:58:32.4273520Z remote: Compressing objects: 32% (107/334) 2024-12-17T23:58:32.4276337Z remote: Compressing objects: 33% (111/334) 2024-12-17T23:58:32.4278518Z remote: Compressing objects: 34% (114/334) 2024-12-17T23:58:32.4288013Z remote: Compressing objects: 35% (117/334) 2024-12-17T23:58:32.4290551Z remote: Compressing objects: 36% (121/334) 2024-12-17T23:58:32.4291008Z remote: Compressing objects: 37% (124/334) 2024-12-17T23:58:32.4297564Z remote: Compressing objects: 38% (127/334) 2024-12-17T23:58:32.4302527Z remote: Compressing objects: 39% (131/334) 2024-12-17T23:58:32.4308272Z remote: Compressing objects: 40% (134/334) 2024-12-17T23:58:32.4314146Z remote: Compressing objects: 41% (137/334) 2024-12-17T23:58:32.4314588Z remote: Compressing objects: 42% (141/334) 2024-12-17T23:58:32.4317193Z remote: Compressing objects: 43% (144/334) 2024-12-17T23:58:32.4319714Z remote: Compressing objects: 44% (147/334) 2024-12-17T23:58:32.4324153Z remote: Compressing objects: 45% (151/334) 2024-12-17T23:58:32.4326781Z remote: Compressing objects: 46% (154/334) 2024-12-17T23:58:32.4330692Z remote: Compressing objects: 47% (157/334) 2024-12-17T23:58:32.4334898Z remote: Compressing objects: 48% (161/334) 2024-12-17T23:58:32.4338790Z remote: Compressing objects: 49% (164/334) 2024-12-17T23:58:32.4342183Z remote: Compressing objects: 50% (167/334) 2024-12-17T23:58:32.4345117Z remote: Compressing objects: 51% (171/334) 2024-12-17T23:58:32.4347914Z remote: Compressing objects: 52% (174/334) 2024-12-17T23:58:32.4350506Z remote: Compressing objects: 53% (178/334) 2024-12-17T23:58:32.4355365Z remote: Compressing objects: 54% (181/334) 2024-12-17T23:58:32.4355810Z remote: Compressing objects: 55% (184/334) 2024-12-17T23:58:32.4356248Z remote: Compressing objects: 56% (188/334) 2024-12-17T23:58:32.4356982Z remote: Compressing objects: 57% (191/334) 2024-12-17T23:58:32.4359083Z remote: Compressing objects: 58% (194/334) 2024-12-17T23:58:32.4361735Z remote: Compressing objects: 59% (198/334) 2024-12-17T23:58:32.4364350Z remote: Compressing objects: 60% (201/334) 2024-12-17T23:58:32.4366063Z remote: Compressing objects: 61% (204/334) 2024-12-17T23:58:32.4368247Z remote: Compressing objects: 62% (208/334) 2024-12-17T23:58:32.4370737Z remote: Compressing objects: 63% (211/334) 2024-12-17T23:58:32.4374887Z remote: Compressing objects: 64% (214/334) 2024-12-17T23:58:32.4376155Z remote: Compressing objects: 65% (218/334) 2024-12-17T23:58:32.4377599Z remote: Compressing objects: 66% (221/334) 2024-12-17T23:58:32.4380692Z remote: Compressing objects: 67% (224/334) 2024-12-17T23:58:32.4385083Z remote: Compressing objects: 68% (228/334) 2024-12-17T23:58:32.4388920Z remote: Compressing objects: 69% (231/334) 2024-12-17T23:58:32.4391429Z remote: Compressing objects: 70% (234/334) 2024-12-17T23:58:32.4392754Z remote: Compressing objects: 71% (238/334) 2024-12-17T23:58:32.4396426Z remote: Compressing objects: 72% (241/334) 2024-12-17T23:58:32.4399536Z remote: Compressing objects: 73% (244/334) 2024-12-17T23:58:32.4401001Z remote: Compressing objects: 74% (248/334) 2024-12-17T23:58:32.4402777Z remote: Compressing objects: 75% (251/334) 2024-12-17T23:58:32.4404429Z remote: Compressing objects: 76% (254/334) 2024-12-17T23:58:32.4406315Z remote: Compressing objects: 77% (258/334) 2024-12-17T23:58:32.4406774Z remote: Compressing objects: 78% (261/334) 2024-12-17T23:58:32.4407825Z remote: Compressing objects: 79% (264/334) 2024-12-17T23:58:32.4409218Z remote: Compressing objects: 80% (268/334) 2024-12-17T23:58:32.4411223Z remote: Compressing objects: 81% (271/334) 2024-12-17T23:58:32.4414326Z remote: Compressing objects: 82% (274/334) 2024-12-17T23:58:32.4416014Z remote: Compressing objects: 83% (278/334) 2024-12-17T23:58:32.4417359Z remote: Compressing objects: 84% (281/334) 2024-12-17T23:58:32.4419197Z remote: Compressing objects: 85% (284/334) 2024-12-17T23:58:32.4420451Z remote: Compressing objects: 86% (288/334) 2024-12-17T23:58:32.4421631Z remote: Compressing objects: 87% (291/334) 2024-12-17T23:58:32.4423139Z remote: Compressing objects: 88% (294/334) 2024-12-17T23:58:32.4424409Z remote: Compressing objects: 89% (298/334) 2024-12-17T23:58:32.4425743Z remote: Compressing objects: 90% (301/334) 2024-12-17T23:58:32.4426958Z remote: Compressing objects: 91% (304/334) 2024-12-17T23:58:32.4427405Z remote: Compressing objects: 92% (308/334) 2024-12-17T23:58:32.4428050Z remote: Compressing objects: 93% (311/334) 2024-12-17T23:58:32.4429609Z remote: Compressing objects: 94% (314/334) 2024-12-17T23:58:32.4430065Z remote: Compressing objects: 95% (318/334) 2024-12-17T23:58:32.4431589Z remote: Compressing objects: 96% (321/334) 2024-12-17T23:58:32.4432027Z remote: Compressing objects: 97% (324/334) 2024-12-17T23:58:32.4432480Z remote: Compressing objects: 98% (328/334) 2024-12-17T23:58:32.4432930Z remote: Compressing objects: 99% (331/334) 2024-12-17T23:58:32.4433376Z remote: Compressing objects: 100% (334/334) 2024-12-17T23:58:32.4433854Z remote: Compressing objects: 100% (334/334), done. 2024-12-17T23:58:55.3378973Z remote: Total 1057261 (delta 532), reused 392 (delta 368), pack-reused 1056559 (from 4) 2024-12-17T23:59:12.1615085Z [command]/usr/bin/git rev-parse --verify --quiet 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7^{object} 2024-12-17T23:59:12.1637355Z 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-17T23:59:12.1639621Z ##[endgroup] 2024-12-17T23:59:12.1640107Z ##[group]Determining the checkout info 2024-12-17T23:59:12.1640818Z ##[endgroup] 2024-12-17T23:59:12.1641614Z ##[group]Checking out the ref 2024-12-17T23:59:12.1644384Z [command]/usr/bin/git checkout --quiet --force 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-17T23:59:13.6771193Z ##[endgroup] 2024-12-17T23:59:13.6771775Z ##[group]Setting up auth for fetching submodules 2024-12-17T23:59:13.6775191Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-12-17T23:59:13.6819832Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-12-17T23:59:13.6841943Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-12-17T23:59:13.6866737Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-12-17T23:59:13.6890752Z ##[endgroup] 2024-12-17T23:59:13.6891246Z ##[group]Fetching submodules 2024-12-17T23:59:13.6892796Z [command]/usr/bin/git submodule sync --recursive 2024-12-17T23:59:13.7206623Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-12-17T23:59:13.7514226Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-12-17T23:59:13.7515702Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-12-17T23:59:13.7516891Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-12-17T23:59:13.7518879Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-12-17T23:59:13.7521395Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2024-12-17T23:59:13.7524943Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-12-17T23:59:13.7527494Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-12-17T23:59:13.7530572Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-12-17T23:59:13.7533868Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2024-12-17T23:59:13.7537124Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-12-17T23:59:13.7540369Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-12-17T23:59:13.7543875Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-12-17T23:59:13.7547255Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-12-17T23:59:13.7550846Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-12-17T23:59:13.7554523Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-12-17T23:59:13.7560112Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-12-17T23:59:13.7566612Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-12-17T23:59:13.7571293Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-12-17T23:59:13.7575414Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-12-17T23:59:13.7580035Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-12-17T23:59:13.7584309Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-12-17T23:59:13.7588794Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-12-17T23:59:13.7593272Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-12-17T23:59:13.7597975Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-12-17T23:59:13.7602589Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-12-17T23:59:13.7607935Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-12-17T23:59:13.7613073Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-12-17T23:59:13.7618583Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-12-17T23:59:13.7623560Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-12-17T23:59:13.7629044Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-12-17T23:59:13.7634264Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-12-17T23:59:13.7639803Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-12-17T23:59:13.7645563Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-12-17T23:59:13.7653021Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-12-17T23:59:13.7658964Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-12-17T23:59:13.7664945Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-12-17T23:59:13.7695347Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-12-17T23:59:14.0523536Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-12-17T23:59:14.2303503Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-12-17T23:59:14.4875712Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-12-17T23:59:14.8388415Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2024-12-17T23:59:15.1454683Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-12-17T23:59:17.0548130Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-12-17T23:59:26.0591002Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-12-17T23:59:26.4644059Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2024-12-17T23:59:28.4821664Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-12-17T23:59:28.9558395Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-12-17T23:59:29.5816752Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-12-17T23:59:30.7684747Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-12-17T23:59:32.7794975Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-12-17T23:59:38.1903160Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-12-17T23:59:39.6135646Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-12-17T23:59:40.7014895Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-12-17T23:59:41.7633338Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-12-17T23:59:42.2266785Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-12-17T23:59:42.6189552Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-12-17T23:59:43.5875712Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-12-17T23:59:43.9665393Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-12-17T23:59:44.2782959Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-12-17T23:59:46.2275215Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-12-17T23:59:47.0387735Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-12-17T23:59:47.3853687Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-12-17T23:59:53.9324193Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-12-17T23:59:55.7328154Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-12-17T23:59:59.2455933Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-12-17T23:59:59.4945628Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-12-18T00:00:07.7592017Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-12-18T00:00:07.9624087Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-12-18T00:00:08.3272069Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-12-18T00:00:09.3413433Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-12-18T00:00:09.6564704Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-12-18T00:00:10.2978005Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-12-18T00:00:10.7196155Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-12-18T00:00:10.7325955Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-12-18T00:00:10.7422122Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-12-18T00:00:10.7669391Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-12-18T00:00:10.8014293Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1' 2024-12-18T00:00:10.8405738Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-12-18T00:00:11.5707297Z Submodule path 'third_party/XNNPACK': checked out '4ea82e595b36106653175dcb04b2aa532660d0d8' 2024-12-18T00:00:11.5950457Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-12-18T00:00:11.8315525Z Submodule path 'third_party/composable_kernel': checked out '50ee4267e27b875d149e642f4cebd47be1dc3b57' 2024-12-18T00:00:11.8806550Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-12-18T00:00:11.9802102Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99' 2024-12-18T00:00:12.0136497Z Submodule path 'third_party/cudnn_frontend': checked out '936021bfed8c91dc416af1588b2c4eca631a9e45' 2024-12-18T00:00:12.5307741Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-12-18T00:00:12.7920208Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-12-18T00:00:12.8892678Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-12-18T00:00:12.8911986Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:00:12.8915470Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:00:12.8916742Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:00:12.8918024Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:00:12.8920257Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:00:12.8951723Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-12-18T00:00:13.7184430Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-12-18T00:00:14.3776878Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-12-18T00:00:16.3809870Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-12-18T00:00:17.3033748Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-12-18T00:00:17.6556818Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-12-18T00:00:17.7559177Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-12-18T00:00:18.1717871Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-12-18T00:00:18.2359369Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-12-18T00:00:18.2487760Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-12-18T00:00:18.3824075Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-12-18T00:00:18.4229797Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-12-18T00:00:18.4627847Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-12-18T00:00:18.4893118Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-12-18T00:00:18.5347559Z Submodule path 'third_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59' 2024-12-18T00:00:18.5482831Z Submodule path 'third_party/ideep': checked out 'c7ccd5bdbe5434ba156f4e856dcef0601637334b' 2024-12-18T00:00:18.5501231Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-12-18T00:00:18.5526164Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-12-18T00:00:31.6199360Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af' 2024-12-18T00:00:31.6390700Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-12-18T00:00:31.7286946Z Submodule path 'third_party/kineto': checked out '338140f58a28d599da3434ced4fd2d75dd1a213d' 2024-12-18T00:00:31.7306135Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:00:31.7307896Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:00:31.7309322Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:00:31.7339145Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-12-18T00:00:32.5913732Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-12-18T00:00:33.6396034Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-12-18T00:00:34.6327880Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-12-18T00:00:34.6346933Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:00:34.6348631Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:00:34.6350027Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:00:34.6352534Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:00:34.6355314Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:00:34.6358375Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:00:34.6361444Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:00:34.6365011Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:00:34.6395688Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-12-18T00:00:35.7686367Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-12-18T00:00:36.2178124Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-12-18T00:00:37.2630018Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-12-18T00:00:37.5687224Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-12-18T00:00:38.0347977Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-12-18T00:00:38.9335903Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-12-18T00:00:45.7045770Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-12-18T00:00:46.1412709Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-12-18T00:00:46.1603247Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-12-18T00:00:46.1969387Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-12-18T00:00:46.2109217Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-12-18T00:00:46.2125459Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:00:46.2152974Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-12-18T00:00:46.5498594Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-12-18T00:00:46.5687785Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-12-18T00:00:46.6115903Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-12-18T00:00:46.7127623Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-12-18T00:00:46.7301451Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-12-18T00:00:46.7695058Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-12-18T00:00:46.8294516Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-12-18T00:00:46.8679113Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-12-18T00:00:46.8973399Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-12-18T00:00:47.0115230Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-12-18T00:00:47.3784219Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c' 2024-12-18T00:00:47.3821236Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-12-18T00:00:47.3847930Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-12-18T00:00:48.4212428Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-12-18T00:00:48.4941311Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-12-18T00:00:48.4960009Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:00:48.4961481Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:00:48.4963112Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:00:48.4966738Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:00:48.4969790Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:00:48.4972645Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:00:48.4975693Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:00:48.4978722Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:00:48.5007602Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-12-18T00:00:48.9457188Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-12-18T00:00:49.8716957Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-12-18T00:00:50.1964951Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-12-18T00:00:56.8527175Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-12-18T00:00:57.1685469Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-12-18T00:00:57.4354237Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-12-18T00:00:57.7467648Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-12-18T00:01:04.1485796Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-12-18T00:01:04.1919455Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-12-18T00:01:04.2087591Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-12-18T00:01:04.3238247Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-12-18T00:01:04.3383729Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-12-18T00:01:04.3541784Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-12-18T00:01:04.3704421Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-12-18T00:01:04.3721004Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:01:04.3722617Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:01:04.3750223Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-12-18T00:01:06.0408841Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-12-18T00:01:07.1896452Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-12-18T00:01:07.2374668Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-12-18T00:01:07.7012698Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-12-18T00:01:07.7135980Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-12-18T00:01:07.9995069Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-12-18T00:01:08.0019251Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:01:08.0020660Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-12-18T00:01:08.0047904Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-12-18T00:01:08.4658516Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-12-18T00:01:09.3849221Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-12-18T00:01:09.4614168Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-12-18T00:01:09.4718860Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-12-18T00:01:09.4852378Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-12-18T00:01:09.5222656Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2024-12-18T00:01:09.5518730Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-12-18T00:01:09.5954369Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-12-18T00:01:09.6224223Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-12-18T00:01:09.6242854Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:01:09.6244502Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:01:09.6246385Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:01:09.6249391Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:01:09.6278402Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-12-18T00:01:10.5835561Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-12-18T00:01:10.8653135Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-12-18T00:01:13.1375507Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-12-18T00:01:14.2081830Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-12-18T00:01:14.2238394Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-12-18T00:01:14.2864235Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-12-18T00:01:14.3158460Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-12-18T00:01:14.3173980Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:01:14.3202123Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-12-18T00:01:14.5695123Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-12-18T00:01:14.5732205Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-12-18T00:01:14.6041633Z Entering 'android/libs/fbjni' 2024-12-18T00:01:14.6084861Z Entering 'third_party/FP16' 2024-12-18T00:01:14.6126247Z Entering 'third_party/FXdiv' 2024-12-18T00:01:14.6176098Z Entering 'third_party/NNPACK' 2024-12-18T00:01:14.6220243Z Entering 'third_party/NVTX' 2024-12-18T00:01:14.6262532Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:01:14.6306583Z Entering 'third_party/XNNPACK' 2024-12-18T00:01:14.6362813Z Entering 'third_party/benchmark' 2024-12-18T00:01:14.6413932Z Entering 'third_party/composable_kernel' 2024-12-18T00:01:14.6467013Z Entering 'third_party/cpp-httplib' 2024-12-18T00:01:14.6510251Z Entering 'third_party/cpuinfo' 2024-12-18T00:01:14.6554985Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:01:14.6599893Z Entering 'third_party/cutlass' 2024-12-18T00:01:14.6648781Z Entering 'third_party/eigen' 2024-12-18T00:01:14.6694334Z Entering 'third_party/fbgemm' 2024-12-18T00:01:14.6738671Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:01:14.6782352Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:01:14.6827016Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:01:14.6882439Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:01:14.6925217Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:01:14.6974527Z Entering 'third_party/flatbuffers' 2024-12-18T00:01:14.7021949Z Entering 'third_party/fmt' 2024-12-18T00:01:14.7063303Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:01:14.7106890Z Entering 'third_party/gloo' 2024-12-18T00:01:14.7148926Z Entering 'third_party/googletest' 2024-12-18T00:01:14.7193656Z Entering 'third_party/ideep' 2024-12-18T00:01:14.7236621Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:01:14.7287190Z Entering 'third_party/ittapi' 2024-12-18T00:01:14.7328161Z Entering 'third_party/kineto' 2024-12-18T00:01:14.7370526Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:01:14.7412259Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:01:14.7456725Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:01:14.7500771Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:01:14.7543618Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:01:14.7585461Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:01:14.7629225Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:01:14.7670278Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:01:14.7719040Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:01:14.7762955Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:01:14.7806026Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:01:14.7856129Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:01:14.7908372Z Entering 'third_party/mimalloc' 2024-12-18T00:01:14.7950531Z Entering 'third_party/nccl/nccl' 2024-12-18T00:01:14.7996464Z Entering 'third_party/nlohmann' 2024-12-18T00:01:14.8040302Z Entering 'third_party/onnx' 2024-12-18T00:01:14.8098645Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:01:14.8143118Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:01:14.8189355Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:01:14.8230654Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:01:14.8272844Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:01:14.8324625Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:01:14.8366522Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:01:14.8407081Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:01:14.8447580Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:01:14.8488010Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:01:14.8532709Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:01:14.8576362Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:01:14.8640057Z Entering 'third_party/pocketfft' 2024-12-18T00:01:14.8684035Z Entering 'third_party/protobuf' 2024-12-18T00:01:14.8728479Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:01:14.8774186Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:01:14.8827716Z Entering 'third_party/psimd' 2024-12-18T00:01:14.8869947Z Entering 'third_party/pthreadpool' 2024-12-18T00:01:14.8913827Z Entering 'third_party/pybind11' 2024-12-18T00:01:14.8956454Z Entering 'third_party/python-peachpy' 2024-12-18T00:01:14.9000976Z Entering 'third_party/sleef' 2024-12-18T00:01:14.9043972Z Entering 'third_party/tensorpipe' 2024-12-18T00:01:14.9085453Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:01:14.9126115Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:01:14.9175139Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:01:14.9227307Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:01:14.9267273Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:01:14.9324063Z ##[endgroup] 2024-12-18T00:01:14.9325317Z ##[group]Persisting credentials for submodules 2024-12-18T00:01:14.9329825Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-12-18T00:01:14.9631791Z Entering 'android/libs/fbjni' 2024-12-18T00:01:14.9685296Z Entering 'third_party/FP16' 2024-12-18T00:01:14.9740973Z Entering 'third_party/FXdiv' 2024-12-18T00:01:14.9796974Z Entering 'third_party/NNPACK' 2024-12-18T00:01:14.9851926Z Entering 'third_party/NVTX' 2024-12-18T00:01:14.9909407Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:01:14.9964928Z Entering 'third_party/XNNPACK' 2024-12-18T00:01:15.0036472Z Entering 'third_party/benchmark' 2024-12-18T00:01:15.0092760Z Entering 'third_party/composable_kernel' 2024-12-18T00:01:15.0160320Z Entering 'third_party/cpp-httplib' 2024-12-18T00:01:15.0218170Z Entering 'third_party/cpuinfo' 2024-12-18T00:01:15.0273244Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:01:15.0328818Z Entering 'third_party/cutlass' 2024-12-18T00:01:15.0392584Z Entering 'third_party/eigen' 2024-12-18T00:01:15.0450973Z Entering 'third_party/fbgemm' 2024-12-18T00:01:15.0507168Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:01:15.0562235Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:01:15.0624478Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:01:15.0692775Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:01:15.0746873Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:01:15.0804761Z Entering 'third_party/flatbuffers' 2024-12-18T00:01:15.0863658Z Entering 'third_party/fmt' 2024-12-18T00:01:15.0920992Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:01:15.0978097Z Entering 'third_party/gloo' 2024-12-18T00:01:15.1034442Z Entering 'third_party/googletest' 2024-12-18T00:01:15.1089069Z Entering 'third_party/ideep' 2024-12-18T00:01:15.1143707Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:01:15.1206277Z Entering 'third_party/ittapi' 2024-12-18T00:01:15.1262173Z Entering 'third_party/kineto' 2024-12-18T00:01:15.1317841Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:01:15.1375175Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:01:15.1431706Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:01:15.1485987Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:01:15.1543609Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:01:15.1603123Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:01:15.1664595Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:01:15.1725102Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:01:15.1785782Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:01:15.1845458Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:01:15.1907380Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:01:15.1961442Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:01:15.2022585Z Entering 'third_party/mimalloc' 2024-12-18T00:01:15.2080902Z Entering 'third_party/nccl/nccl' 2024-12-18T00:01:15.2136890Z Entering 'third_party/nlohmann' 2024-12-18T00:01:15.2194972Z Entering 'third_party/onnx' 2024-12-18T00:01:15.2271422Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:01:15.2332892Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:01:15.2395551Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:01:15.2448521Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:01:15.2504349Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:01:15.2559818Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:01:15.2622016Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:01:15.2676364Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:01:15.2732487Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:01:15.2787150Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:01:15.2846630Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:01:15.2905391Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:01:15.2982910Z Entering 'third_party/pocketfft' 2024-12-18T00:01:15.3036349Z Entering 'third_party/protobuf' 2024-12-18T00:01:15.3094783Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:01:15.3150161Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:01:15.3210041Z Entering 'third_party/psimd' 2024-12-18T00:01:15.3263932Z Entering 'third_party/pthreadpool' 2024-12-18T00:01:15.3319975Z Entering 'third_party/pybind11' 2024-12-18T00:01:15.3380460Z Entering 'third_party/python-peachpy' 2024-12-18T00:01:15.3435393Z Entering 'third_party/sleef' 2024-12-18T00:01:15.3490433Z Entering 'third_party/tensorpipe' 2024-12-18T00:01:15.3545133Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:01:15.3600646Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:01:15.3659682Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:01:15.3716778Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:01:15.3774144Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:01:15.3846846Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-12-18T00:01:15.4150823Z Entering 'android/libs/fbjni' 2024-12-18T00:01:15.4202137Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-12-18T00:01:15.4220261Z Entering 'third_party/FP16' 2024-12-18T00:01:15.4273790Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-12-18T00:01:15.4289364Z Entering 'third_party/FXdiv' 2024-12-18T00:01:15.4341187Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-12-18T00:01:15.4358266Z Entering 'third_party/NNPACK' 2024-12-18T00:01:15.4409359Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-12-18T00:01:15.4426996Z Entering 'third_party/NVTX' 2024-12-18T00:01:15.4483042Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2024-12-18T00:01:15.4500573Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:01:15.4551518Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-12-18T00:01:15.4567932Z Entering 'third_party/XNNPACK' 2024-12-18T00:01:15.4619135Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-12-18T00:01:15.4651810Z Entering 'third_party/benchmark' 2024-12-18T00:01:15.4705570Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-12-18T00:01:15.4722504Z Entering 'third_party/composable_kernel' 2024-12-18T00:01:15.4772969Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2024-12-18T00:01:15.4797171Z Entering 'third_party/cpp-httplib' 2024-12-18T00:01:15.4846948Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-12-18T00:01:15.4864614Z Entering 'third_party/cpuinfo' 2024-12-18T00:01:15.4917243Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-12-18T00:01:15.4933959Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:01:15.4987126Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-12-18T00:01:15.5003953Z Entering 'third_party/cutlass' 2024-12-18T00:01:15.5054082Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-12-18T00:01:15.5081209Z Entering 'third_party/eigen' 2024-12-18T00:01:15.5133442Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-12-18T00:01:15.5153799Z Entering 'third_party/fbgemm' 2024-12-18T00:01:15.5207517Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-12-18T00:01:15.5225290Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:01:15.5276152Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-12-18T00:01:15.5291875Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:01:15.5342978Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-12-18T00:01:15.5360612Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:01:15.5413524Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-12-18T00:01:15.5438485Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:01:15.5489670Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:01:15.5506710Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:01:15.5557892Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-12-18T00:01:15.5575929Z Entering 'third_party/flatbuffers' 2024-12-18T00:01:15.5629813Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-12-18T00:01:15.5648739Z Entering 'third_party/fmt' 2024-12-18T00:01:15.5702398Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-12-18T00:01:15.5719921Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:01:15.5770032Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-12-18T00:01:15.5787816Z Entering 'third_party/gloo' 2024-12-18T00:01:15.5840141Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-12-18T00:01:15.5858569Z Entering 'third_party/googletest' 2024-12-18T00:01:15.5911409Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:01:15.5927657Z Entering 'third_party/ideep' 2024-12-18T00:01:15.5980713Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-12-18T00:01:15.5997927Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:01:15.6046192Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-12-18T00:01:15.6070621Z Entering 'third_party/ittapi' 2024-12-18T00:01:15.6124806Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-12-18T00:01:15.6142464Z Entering 'third_party/kineto' 2024-12-18T00:01:15.6193611Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-12-18T00:01:15.6209687Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:01:15.6261454Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-12-18T00:01:15.6279057Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:01:15.6329399Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-12-18T00:01:15.6348803Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:01:15.6401064Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-12-18T00:01:15.6419386Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:01:15.6470768Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-12-18T00:01:15.6487334Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:01:15.6539537Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-12-18T00:01:15.6555427Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:01:15.6608432Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-12-18T00:01:15.6629323Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:01:15.6682316Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-12-18T00:01:15.6701334Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:01:15.6753125Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:01:15.6769475Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:01:15.6822963Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-12-18T00:01:15.6842377Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:01:15.6894961Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-12-18T00:01:15.6915499Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:01:15.6966857Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-12-18T00:01:15.6984385Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:01:15.7034116Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-12-18T00:01:15.7051647Z Entering 'third_party/mimalloc' 2024-12-18T00:01:15.7104914Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-12-18T00:01:15.7122105Z Entering 'third_party/nccl/nccl' 2024-12-18T00:01:15.7173289Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-12-18T00:01:15.7190784Z Entering 'third_party/nlohmann' 2024-12-18T00:01:15.7243031Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-12-18T00:01:15.7260960Z Entering 'third_party/onnx' 2024-12-18T00:01:15.7316429Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-12-18T00:01:15.7350000Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:01:15.7400840Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-12-18T00:01:15.7422944Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:01:15.7475039Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-12-18T00:01:15.7492341Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:01:15.7544066Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-12-18T00:01:15.7560966Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:01:15.7614190Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:01:15.7631450Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:01:15.7684051Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-12-18T00:01:15.7702108Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:01:15.7752774Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-12-18T00:01:15.7770182Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:01:15.7821811Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-12-18T00:01:15.7838995Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:01:15.7890672Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-12-18T00:01:15.7907917Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:01:15.7958632Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-12-18T00:01:15.7974184Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:01:15.8025688Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-12-18T00:01:15.8045177Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:01:15.8098026Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-12-18T00:01:15.8118309Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:01:15.8168570Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-12-18T00:01:15.8206606Z Entering 'third_party/pocketfft' 2024-12-18T00:01:15.8258575Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-12-18T00:01:15.8275465Z Entering 'third_party/protobuf' 2024-12-18T00:01:15.8327627Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-12-18T00:01:15.8348715Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:01:15.8400369Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-12-18T00:01:15.8418293Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:01:15.8467861Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:01:15.8486049Z Entering 'third_party/psimd' 2024-12-18T00:01:15.8537296Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-12-18T00:01:15.8554599Z Entering 'third_party/pthreadpool' 2024-12-18T00:01:15.8605900Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-12-18T00:01:15.8623150Z Entering 'third_party/pybind11' 2024-12-18T00:01:15.8675980Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-12-18T00:01:15.8691266Z Entering 'third_party/python-peachpy' 2024-12-18T00:01:15.8742921Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-12-18T00:01:15.8760044Z Entering 'third_party/sleef' 2024-12-18T00:01:15.8811968Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-12-18T00:01:15.8829466Z Entering 'third_party/tensorpipe' 2024-12-18T00:01:15.8882518Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-12-18T00:01:15.8899950Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:01:15.8950784Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:01:15.8966494Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:01:15.9019478Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-12-18T00:01:15.9036128Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:01:15.9086485Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-12-18T00:01:15.9104416Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:01:15.9156697Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-12-18T00:01:15.9171557Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:01:15.9223933Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-12-18T00:01:15.9753021Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-12-18T00:01:16.0063331Z Entering 'android/libs/fbjni' 2024-12-18T00:01:16.0105956Z Entering 'third_party/FP16' 2024-12-18T00:01:16.0149893Z Entering 'third_party/FXdiv' 2024-12-18T00:01:16.0192820Z Entering 'third_party/NNPACK' 2024-12-18T00:01:16.0238739Z Entering 'third_party/NVTX' 2024-12-18T00:01:16.0285376Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:01:16.0333182Z Entering 'third_party/XNNPACK' 2024-12-18T00:01:16.0395136Z Entering 'third_party/benchmark' 2024-12-18T00:01:16.0438328Z Entering 'third_party/composable_kernel' 2024-12-18T00:01:16.0487741Z Entering 'third_party/cpp-httplib' 2024-12-18T00:01:16.0529194Z Entering 'third_party/cpuinfo' 2024-12-18T00:01:16.0572202Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:01:16.0614382Z Entering 'third_party/cutlass' 2024-12-18T00:01:16.0667062Z Entering 'third_party/eigen' 2024-12-18T00:01:16.0713207Z Entering 'third_party/fbgemm' 2024-12-18T00:01:16.0757986Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:01:16.0804795Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:01:16.0845445Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:01:16.0894116Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:01:16.0943773Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:01:16.0986770Z Entering 'third_party/flatbuffers' 2024-12-18T00:01:16.1032456Z Entering 'third_party/fmt' 2024-12-18T00:01:16.1078241Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:01:16.1122160Z Entering 'third_party/gloo' 2024-12-18T00:01:16.1165467Z Entering 'third_party/googletest' 2024-12-18T00:01:16.1208064Z Entering 'third_party/ideep' 2024-12-18T00:01:16.1250220Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:01:16.1302875Z Entering 'third_party/ittapi' 2024-12-18T00:01:16.1344875Z Entering 'third_party/kineto' 2024-12-18T00:01:16.1387449Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:01:16.1429905Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:01:16.1476369Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:01:16.1519082Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:01:16.1566962Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:01:16.1607442Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:01:16.1652016Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:01:16.1699165Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:01:16.1743206Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:01:16.1786182Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:01:16.1830037Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:01:16.1870546Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:01:16.1916142Z Entering 'third_party/mimalloc' 2024-12-18T00:01:16.1959877Z Entering 'third_party/nccl/nccl' 2024-12-18T00:01:16.2004685Z Entering 'third_party/nlohmann' 2024-12-18T00:01:16.2046957Z Entering 'third_party/onnx' 2024-12-18T00:01:16.2107820Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:01:16.2153370Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:01:16.2199189Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:01:16.2240809Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:01:16.2283568Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:01:16.2325481Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:01:16.2368326Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:01:16.2409020Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:01:16.2449853Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:01:16.2491182Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:01:16.2533903Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:01:16.2578628Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:01:16.2641152Z Entering 'third_party/pocketfft' 2024-12-18T00:01:16.2684187Z Entering 'third_party/protobuf' 2024-12-18T00:01:16.2731234Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:01:16.2784850Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:01:16.2826751Z Entering 'third_party/psimd' 2024-12-18T00:01:16.2868726Z Entering 'third_party/pthreadpool' 2024-12-18T00:01:16.2912213Z Entering 'third_party/pybind11' 2024-12-18T00:01:16.2954887Z Entering 'third_party/python-peachpy' 2024-12-18T00:01:16.2999347Z Entering 'third_party/sleef' 2024-12-18T00:01:16.3044885Z Entering 'third_party/tensorpipe' 2024-12-18T00:01:16.3086223Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:01:16.3127115Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:01:16.3168759Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:01:16.3210488Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:01:16.3251995Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:01:16.3314288Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-12-18T00:01:16.3623619Z Entering 'android/libs/fbjni' 2024-12-18T00:01:16.3666073Z Entering 'third_party/FP16' 2024-12-18T00:01:16.3708793Z Entering 'third_party/FXdiv' 2024-12-18T00:01:16.3752470Z Entering 'third_party/NNPACK' 2024-12-18T00:01:16.3799737Z Entering 'third_party/NVTX' 2024-12-18T00:01:16.3842838Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:01:16.3885327Z Entering 'third_party/XNNPACK' 2024-12-18T00:01:16.3944225Z Entering 'third_party/benchmark' 2024-12-18T00:01:16.3986974Z Entering 'third_party/composable_kernel' 2024-12-18T00:01:16.4035819Z Entering 'third_party/cpp-httplib' 2024-12-18T00:01:16.4080504Z Entering 'third_party/cpuinfo' 2024-12-18T00:01:16.4123966Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:01:16.4166632Z Entering 'third_party/cutlass' 2024-12-18T00:01:16.4218845Z Entering 'third_party/eigen' 2024-12-18T00:01:16.4264655Z Entering 'third_party/fbgemm' 2024-12-18T00:01:16.4309198Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:01:16.4350716Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:01:16.4392427Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:01:16.4443122Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:01:16.4484482Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:01:16.4528652Z Entering 'third_party/flatbuffers' 2024-12-18T00:01:16.4574082Z Entering 'third_party/fmt' 2024-12-18T00:01:16.4615987Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:01:16.4660423Z Entering 'third_party/gloo' 2024-12-18T00:01:16.4704025Z Entering 'third_party/googletest' 2024-12-18T00:01:16.4746143Z Entering 'third_party/ideep' 2024-12-18T00:01:16.4787949Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:01:16.4837390Z Entering 'third_party/ittapi' 2024-12-18T00:01:16.4882229Z Entering 'third_party/kineto' 2024-12-18T00:01:16.4924557Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:01:16.4968755Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:01:16.5012115Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:01:16.5054125Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:01:16.5101006Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:01:16.5141815Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:01:16.5187612Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:01:16.5229073Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:01:16.5270249Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:01:16.5316213Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:01:16.5361058Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:01:16.5403510Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:01:16.5445889Z Entering 'third_party/mimalloc' 2024-12-18T00:01:16.5488691Z Entering 'third_party/nccl/nccl' 2024-12-18T00:01:16.5530268Z Entering 'third_party/nlohmann' 2024-12-18T00:01:16.5573749Z Entering 'third_party/onnx' 2024-12-18T00:01:16.5632781Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:01:16.5682019Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:01:16.5725611Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:01:16.5766502Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:01:16.5810890Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:01:16.5851936Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:01:16.5900992Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:01:16.5941944Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:01:16.5982591Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:01:16.6022272Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:01:16.6066149Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:01:16.6110302Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:01:16.6174214Z Entering 'third_party/pocketfft' 2024-12-18T00:01:16.6217511Z Entering 'third_party/protobuf' 2024-12-18T00:01:16.6262408Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:01:16.6304269Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:01:16.6347951Z Entering 'third_party/psimd' 2024-12-18T00:01:16.6390306Z Entering 'third_party/pthreadpool' 2024-12-18T00:01:16.6433475Z Entering 'third_party/pybind11' 2024-12-18T00:01:16.6479580Z Entering 'third_party/python-peachpy' 2024-12-18T00:01:16.6522513Z Entering 'third_party/sleef' 2024-12-18T00:01:16.6564694Z Entering 'third_party/tensorpipe' 2024-12-18T00:01:16.6606524Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:01:16.6647401Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:01:16.6688731Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:01:16.6730438Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:01:16.6773685Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:01:16.6834881Z ##[endgroup] 2024-12-18T00:01:16.6870733Z [command]/usr/bin/git log -1 --format='%H' 2024-12-18T00:01:16.6891785Z '0cdf8b1d09254cfda66191d1bd01e3041c3c76f7' 2024-12-18T00:01:16.7064360Z Prepare all required actions 2024-12-18T00:01:16.7064907Z Getting action download info 2024-12-18T00:01:16.8688388Z ##[group]Run ./.github/actions/setup-linux 2024-12-18T00:01:16.8688773Z env: 2024-12-18T00:01:16.8689018Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:16.8689333Z ##[endgroup] 2024-12-18T00:01:16.8739945Z ##[group]Run set -euo pipefail 2024-12-18T00:01:16.8740314Z set -euo pipefail 2024-12-18T00:01:16.8740619Z function get_ec2_metadata() { 2024-12-18T00:01:16.8741028Z  # Pulled from instance metadata endpoint for EC2 2024-12-18T00:01:16.8741705Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-12-18T00:01:16.8742318Z  category=$1 2024-12-18T00:01:16.8742711Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-12-18T00:01:16.8743197Z  runner_name_str=i-0c6125b7fa38a3d09 2024-12-18T00:01:16.8743598Z  if [[ -f /.inarc ]]; then 2024-12-18T00:01:16.8743971Z  echo "ARC Runner, no info on ec2 metadata" 2024-12-18T00:01:16.8744381Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-12-18T00:01:16.8744882Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-12-18T00:01:16.8745343Z  else 2024-12-18T00:01:16.8746262Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-12-18T00:01:16.8747231Z  fi 2024-12-18T00:01:16.8747468Z } 2024-12-18T00:01:16.8747742Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-12-18T00:01:16.8748203Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-12-18T00:01:16.8748718Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-12-18T00:01:16.8749173Z echo "system info $(uname -a)" 2024-12-18T00:01:16.8756743Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:16.8757156Z env: 2024-12-18T00:01:16.8757378Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:16.8757664Z ##[endgroup] 2024-12-18T00:01:16.8920767Z ami-id: ami-0fff1b9a61dec8a5f 2024-12-18T00:01:16.9029211Z instance-id: i-0c6125b7fa38a3d09 2024-12-18T00:01:16.9128966Z instance-type: g4dn.12xlarge 2024-12-18T00:01:16.9139591Z system info Linux ip-10-0-46-48.ec2.internal 6.1.109-118.189.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 10 08:59:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-12-18T00:01:16.9171244Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:01:16.9172363Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:01:16.9178814Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:16.9179234Z env: 2024-12-18T00:01:16.9179452Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:16.9179735Z ##[endgroup] 2024-12-18T00:01:16.9236289Z ##[group]Run if systemctl is-active --quiet docker; then 2024-12-18T00:01:16.9236798Z if systemctl is-active --quiet docker; then 2024-12-18T00:01:16.9237238Z  echo "Docker daemon is running..."; 2024-12-18T00:01:16.9237597Z else 2024-12-18T00:01:16.9238004Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-12-18T00:01:16.9238483Z fi 2024-12-18T00:01:16.9244287Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:16.9244817Z env: 2024-12-18T00:01:16.9245075Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:16.9245377Z ##[endgroup] 2024-12-18T00:01:16.9322236Z Docker daemon is running... 2024-12-18T00:01:16.9368251Z ##[group]Run nick-fields/retry@v3.0.0 2024-12-18T00:01:16.9368748Z with: 2024-12-18T00:01:16.9368996Z shell: bash 2024-12-18T00:01:16.9369379Z timeout_minutes: 5 2024-12-18T00:01:16.9369677Z max_attempts: 3 2024-12-18T00:01:16.9369963Z retry_wait_seconds: 30 2024-12-18T00:01:16.9372685Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-12-18T00:01:16.9375415Z polling_interval_seconds: 1 2024-12-18T00:01:16.9375864Z warning_on_retry: true 2024-12-18T00:01:16.9376184Z continue_on_error: false 2024-12-18T00:01:16.9376472Z env: 2024-12-18T00:01:16.9376720Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:16.9377253Z AWS_RETRY_MODE: standard 2024-12-18T00:01:16.9377532Z AWS_MAX_ATTEMPTS: 5 2024-12-18T00:01:16.9377808Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T00:01:16.9378088Z ##[endgroup] 2024-12-18T00:01:18.2212282Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:01:18.2213056Z Configure a credential helper to remove this warning. See 2024-12-18T00:01:18.2213768Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:01:18.2214249Z 2024-12-18T00:01:18.2214371Z Login Succeeded 2024-12-18T00:01:19.0812787Z Command completed after 1 attempt(s). 2024-12-18T00:01:19.0904774Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T00:01:19.0905394Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T00:01:19.0905916Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T00:01:19.0914880Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:19.0915314Z env: 2024-12-18T00:01:19.0915553Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:19.0915851Z ##[endgroup] 2024-12-18T00:01:19.1007471Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T00:01:19.1008150Z # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T00:01:19.1008663Z # shellcheck disable=SC2046 2024-12-18T00:01:19.1009053Z docker stop $(docker ps -q) || true 2024-12-18T00:01:19.1009468Z # Prune all of the docker images 2024-12-18T00:01:19.1009860Z docker system prune -af 2024-12-18T00:01:19.1016045Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:19.1016456Z env: 2024-12-18T00:01:19.1016691Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:19.1016960Z ##[endgroup] 2024-12-18T00:01:19.1288642Z "docker stop" requires at least 1 argument. 2024-12-18T00:01:19.1289110Z See 'docker stop --help'. 2024-12-18T00:01:19.1289331Z 2024-12-18T00:01:19.1289544Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-12-18T00:01:19.1289866Z 2024-12-18T00:01:19.1289997Z Stop one or more running containers 2024-12-18T00:01:19.1441734Z Total reclaimed space: 0B 2024-12-18T00:01:19.1481172Z ##[group]Run set +e 2024-12-18T00:01:19.1481472Z set +e 2024-12-18T00:01:19.1481728Z set -x 2024-12-18T00:01:19.1481971Z  2024-12-18T00:01:19.1482222Z PT_DOMAIN=download.pytorch.org 2024-12-18T00:01:19.1482836Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-12-18T00:01:19.1483933Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-12-18T00:01:19.1484585Z # one is returned at random 2024-12-18T00:01:19.1485058Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-12-18T00:01:19.1485671Z  2024-12-18T00:01:19.1486090Z if [ -z "${RESOLVED_IP}" ]; then 2024-12-18T00:01:19.1486610Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-12-18T00:01:19.1487239Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-12-18T00:01:19.1487711Z  2024-12-18T00:01:19.1487992Z  if [ -z "${RESOLVED_IP}" ]; then 2024-12-18T00:01:19.1488457Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-12-18T00:01:19.1488877Z  exit 1 2024-12-18T00:01:19.1489163Z  fi 2024-12-18T00:01:19.1489424Z fi 2024-12-18T00:01:19.1489678Z  2024-12-18T00:01:19.1489993Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-12-18T00:01:19.1490425Z  # Clean up any old records first 2024-12-18T00:01:19.1490863Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-12-18T00:01:19.1491254Z fi 2024-12-18T00:01:19.1491520Z  2024-12-18T00:01:19.1491894Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-12-18T00:01:19.1492377Z cat /etc/hosts 2024-12-18T00:01:19.1498382Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:19.1498788Z env: 2024-12-18T00:01:19.1499011Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:19.1499299Z ##[endgroup] 2024-12-18T00:01:19.1523370Z + PT_DOMAIN=download.pytorch.org 2024-12-18T00:01:19.1528072Z ++ dig -4 +short download.pytorch.org 2024-12-18T00:01:19.1528891Z ++ tail -n1 2024-12-18T00:01:19.2267738Z + RESOLVED_IP=3.167.99.125 2024-12-18T00:01:19.2268173Z + '[' -z 3.167.99.125 ']' 2024-12-18T00:01:19.2268521Z + grep -r download.pytorch.org /etc/hosts 2024-12-18T00:01:19.2284112Z + echo '3.167.99.125 download.pytorch.org' 2024-12-18T00:01:19.2284657Z + sudo tee -a /etc/hosts 2024-12-18T00:01:19.4930740Z 3.167.99.125 download.pytorch.org 2024-12-18T00:01:19.4947635Z + cat /etc/hosts 2024-12-18T00:01:19.4956526Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-12-18T00:01:19.4961360Z ::1 localhost6 localhost6.localdomain6 2024-12-18T00:01:19.4961797Z 3.167.99.125 download.pytorch.org 2024-12-18T00:01:19.5137887Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@release/2.6 2024-12-18T00:01:19.5138435Z with: 2024-12-18T00:01:19.5139246Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:19.5140169Z docker-build-dir: .ci/docker 2024-12-18T00:01:19.5140494Z working-directory: . 2024-12-18T00:01:19.5140882Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:19.5141333Z force-push: false 2024-12-18T00:01:19.5141579Z env: 2024-12-18T00:01:19.5141818Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:19.5142117Z ##[endgroup] 2024-12-18T00:01:19.5167646Z ##[group]Run set -ex 2024-12-18T00:01:19.5168023Z set -ex 2024-12-18T00:01:19.5168298Z  2024-12-18T00:01:19.5168778Z # If the docker build directory or the build script doesn't exist, the action will 2024-12-18T00:01:19.5169625Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-12-18T00:01:19.5170293Z # job could then download the pre-built image as usual 2024-12-18T00:01:19.5170905Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-12-18T00:01:19.5171471Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:19.5172006Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:19.5172496Z  2024-12-18T00:01:19.5172919Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-12-18T00:01:19.5173434Z  exit 0 2024-12-18T00:01:19.5173705Z else 2024-12-18T00:01:19.5174024Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:19.5174546Z fi 2024-12-18T00:01:19.5174801Z  2024-12-18T00:01:19.5175192Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-12-18T00:01:19.5175913Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-12-18T00:01:19.5176555Z  # use it as it is, but first let's extract the tag 2024-12-18T00:01:19.5177359Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-12-18T00:01:19.5177900Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:19.5178420Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:19.5178854Z else 2024-12-18T00:01:19.5179173Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-12-18T00:01:19.5179664Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:19.5180344Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:19.5180944Z fi 2024-12-18T00:01:19.5187715Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:19.5188126Z env: 2024-12-18T00:01:19.5188359Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:19.5188647Z REPO_NAME: pytorch 2024-12-18T00:01:19.5189440Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:19.5190293Z DOCKER_BUILD_DIR: .ci/docker 2024-12-18T00:01:19.5190673Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:19.5191129Z ##[endgroup] 2024-12-18T00:01:19.5216461Z + [[ ! -d .ci/docker ]] 2024-12-18T00:01:19.5216786Z + [[ ! -f .ci/docker/build.sh ]] 2024-12-18T00:01:19.5217116Z + echo skip=false 2024-12-18T00:01:19.5218548Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-12-18T00:01:19.5224607Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:19.5225566Z ++ awk -F '[:,]' '{print $2}' 2024-12-18T00:01:19.5247270Z + DOCKER_TAG=45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:19.5247794Z + echo docker-tag=45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:19.5248848Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:19.5289460Z ##[group]Run set +e 2024-12-18T00:01:19.5289854Z set +e 2024-12-18T00:01:19.5290141Z set -x 2024-12-18T00:01:19.5290415Z  2024-12-18T00:01:19.5290661Z login() { 2024-12-18T00:01:19.5291248Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-12-18T00:01:19.5291875Z } 2024-12-18T00:01:19.5292135Z  2024-12-18T00:01:19.5292401Z retry () { 2024-12-18T00:01:19.5292724Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-12-18T00:01:19.5293107Z } 2024-12-18T00:01:19.5293360Z  2024-12-18T00:01:19.5293644Z retry login "${DOCKER_REGISTRY}" 2024-12-18T00:01:19.5294021Z  2024-12-18T00:01:19.5294279Z START_TIME=$(date +%s) 2024-12-18T00:01:19.5294635Z # Wait up to 90 minutes 2024-12-18T00:01:19.5295072Z while [[ $(( $(date +%s) - 5400 )) -lt $START_TIME ]]; do 2024-12-18T00:01:19.5295771Z  # Check if image already exists, if it does then skip building it 2024-12-18T00:01:19.5296347Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-12-18T00:01:19.5296773Z  exit 0 2024-12-18T00:01:19.5297180Z  fi 2024-12-18T00:01:19.5297444Z  2024-12-18T00:01:19.5297898Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2024-12-18T00:01:19.5298682Z  # use this to differentiate between the Docker build and regular build jobs. For the 2024-12-18T00:01:19.5299459Z  # latter, it will wait for the Docker images to become available before continuing 2024-12-18T00:01:19.5300072Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2024-12-18T00:01:19.5300529Z  # It's a Docker build job, let's build the image 2024-12-18T00:01:19.5300938Z  break 2024-12-18T00:01:19.5301214Z  else 2024-12-18T00:01:19.5301615Z  # It's a regular build job, wait for the image to become available 2024-12-18T00:01:19.5302111Z  sleep 300 2024-12-18T00:01:19.5302404Z  fi 2024-12-18T00:01:19.5302658Z done 2024-12-18T00:01:19.5302942Z  2024-12-18T00:01:19.5303357Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-12-18T00:01:19.5304037Z # be empty. The default action would be to continue rebuild the image 2024-12-18T00:01:19.5304642Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-12-18T00:01:19.5305165Z  # if we're on the base branch then use the parent commit 2024-12-18T00:01:19.5305643Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-12-18T00:01:19.5306024Z else 2024-12-18T00:01:19.5306411Z  # otherwise we're on a PR, so use the most recent base commit 2024-12-18T00:01:19.5306972Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-12-18T00:01:19.5307403Z fi 2024-12-18T00:01:19.5307646Z  2024-12-18T00:01:19.5307929Z if [[ -z "${MERGE_BASE}" ]]; then 2024-12-18T00:01:19.5308350Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:19.5308749Z  2024-12-18T00:01:19.5309294Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-12-18T00:01:19.5309925Z  exit 0 2024-12-18T00:01:19.5310313Z fi 2024-12-18T00:01:19.5310569Z  2024-12-18T00:01:19.5310935Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-12-18T00:01:19.5311728Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-12-18T00:01:19.5312412Z  exit 1 2024-12-18T00:01:19.5312667Z fi 2024-12-18T00:01:19.5312917Z  2024-12-18T00:01:19.5313337Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-12-18T00:01:19.5314108Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-12-18T00:01:19.5314894Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-12-18T00:01:19.5315667Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-12-18T00:01:19.5316523Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-12-18T00:01:19.5317044Z fi 2024-12-18T00:01:19.5317293Z  2024-12-18T00:01:19.5317597Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:19.5323574Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:19.5324196Z env: 2024-12-18T00:01:19.5324486Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:19.5324821Z DOCKER_BUILD_DIR: .ci/docker 2024-12-18T00:01:19.5325237Z BASE_REVISION: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:01:19.5326248Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:19.5327266Z DOCKER_TAG: 45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:19.5327883Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:19.5328359Z DOCKER_PUSH: 2024-12-18T00:01:19.5328647Z ##[endgroup] 2024-12-18T00:01:19.5352632Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:19.5353201Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:19.5355978Z + aws ecr get-login-password --region us-east-1 2024-12-18T00:01:19.5357258Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:20.0967311Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:01:20.0968050Z Configure a credential helper to remove this warning. See 2024-12-18T00:01:20.0968728Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:01:20.0969200Z 2024-12-18T00:01:20.0969318Z Login Succeeded 2024-12-18T00:01:20.0986882Z ++ date +%s 2024-12-18T00:01:20.0996211Z + START_TIME=1734480080 2024-12-18T00:01:20.0999428Z ++ date +%s 2024-12-18T00:01:20.1011444Z + [[ 1734474680 -lt 1734480080 ]] 2024-12-18T00:01:20.1012552Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:20.3471018Z { 2024-12-18T00:01:20.3471339Z "schemaVersion": 2, 2024-12-18T00:01:20.3471833Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-12-18T00:01:20.3472340Z "config": { 2024-12-18T00:01:20.3472731Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-12-18T00:01:20.3473200Z "size": 50450, 2024-12-18T00:01:20.3473667Z "digest": "sha256:166dd14a00627fbd0d4e062a733b69b733b63b1ac12ec3bd5285db8a8befaaf5" 2024-12-18T00:01:20.3474218Z }, 2024-12-18T00:01:20.3474451Z "layers": [ 2024-12-18T00:01:20.3474696Z { 2024-12-18T00:01:20.3475079Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3475573Z "size": 28580681, 2024-12-18T00:01:20.3476060Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-12-18T00:01:20.3476606Z }, 2024-12-18T00:01:20.3476835Z { 2024-12-18T00:01:20.3477452Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3477926Z "size": 7943353, 2024-12-18T00:01:20.3478416Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-12-18T00:01:20.3478961Z }, 2024-12-18T00:01:20.3479183Z { 2024-12-18T00:01:20.3479561Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3480030Z "size": 56226389, 2024-12-18T00:01:20.3480514Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-12-18T00:01:20.3481055Z }, 2024-12-18T00:01:20.3481281Z { 2024-12-18T00:01:20.3481655Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3482222Z "size": 187, 2024-12-18T00:01:20.3482698Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-12-18T00:01:20.3483347Z }, 2024-12-18T00:01:20.3483575Z { 2024-12-18T00:01:20.3484131Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3484643Z "size": 6885, 2024-12-18T00:01:20.3485134Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-12-18T00:01:20.3485747Z }, 2024-12-18T00:01:20.3485975Z { 2024-12-18T00:01:20.3486360Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3486856Z "size": 1383662475, 2024-12-18T00:01:20.3487333Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-12-18T00:01:20.3487877Z }, 2024-12-18T00:01:20.3488105Z { 2024-12-18T00:01:20.3488489Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3488966Z "size": 62282, 2024-12-18T00:01:20.3489598Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-12-18T00:01:20.3490280Z }, 2024-12-18T00:01:20.3490505Z { 2024-12-18T00:01:20.3490886Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3491353Z "size": 1684, 2024-12-18T00:01:20.3491929Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-12-18T00:01:20.3492571Z }, 2024-12-18T00:01:20.3492781Z { 2024-12-18T00:01:20.3493127Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3493553Z "size": 1522, 2024-12-18T00:01:20.3493985Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-12-18T00:01:20.3494479Z }, 2024-12-18T00:01:20.3494686Z { 2024-12-18T00:01:20.3495031Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3495460Z "size": 2445884909, 2024-12-18T00:01:20.3495899Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-12-18T00:01:20.3496389Z }, 2024-12-18T00:01:20.3496592Z { 2024-12-18T00:01:20.3496934Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3497391Z "size": 85882, 2024-12-18T00:01:20.3497826Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-12-18T00:01:20.3498327Z }, 2024-12-18T00:01:20.3498536Z { 2024-12-18T00:01:20.3498871Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3499313Z "size": 1825, 2024-12-18T00:01:20.3499747Z "digest": "sha256:38b3bd834f10691245159178ede4a97949d80aafef88c4da88bc8c59b128c195" 2024-12-18T00:01:20.3500246Z }, 2024-12-18T00:01:20.3500455Z { 2024-12-18T00:01:20.3500789Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3501236Z "size": 655189618, 2024-12-18T00:01:20.3501685Z "digest": "sha256:e45523662aa5eb3b174fdd800641a054a960911c585cea9b0782baf969a4dfa5" 2024-12-18T00:01:20.3502187Z }, 2024-12-18T00:01:20.3502397Z { 2024-12-18T00:01:20.3502731Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3503171Z "size": 704, 2024-12-18T00:01:20.3503678Z "digest": "sha256:5a6d97ee6e47c9a76f0de956b2f0e28bfc3947fdf421470b6202b5ba2702f3a9" 2024-12-18T00:01:20.3504181Z }, 2024-12-18T00:01:20.3504390Z { 2024-12-18T00:01:20.3504720Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3505162Z "size": 1260, 2024-12-18T00:01:20.3505604Z "digest": "sha256:8b9cee92f4081f7ae7b4ac3842ed028a3595044d35c3aabdfdf61fdf414b422c" 2024-12-18T00:01:20.3506106Z }, 2024-12-18T00:01:20.3506311Z { 2024-12-18T00:01:20.3506637Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3507075Z "size": 484, 2024-12-18T00:01:20.3507500Z "digest": "sha256:05a25e0e092fec6d2dd4a72016ea313485058594415703c2be2cc9fb041d483b" 2024-12-18T00:01:20.3507991Z }, 2024-12-18T00:01:20.3508200Z { 2024-12-18T00:01:20.3508528Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3508968Z "size": 91714827, 2024-12-18T00:01:20.3509421Z "digest": "sha256:32f9bd3454b2990eb47fa1b21e333f7e8b560c5c086f531c5ee9389676c218ac" 2024-12-18T00:01:20.3509914Z }, 2024-12-18T00:01:20.3510124Z { 2024-12-18T00:01:20.3510456Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3510896Z "size": 3707, 2024-12-18T00:01:20.3511338Z "digest": "sha256:7cef0be11b192948fbbde9ab3ef1a6d84a3008372f6c8dd6b2bc29b049f76e74" 2024-12-18T00:01:20.3511842Z }, 2024-12-18T00:01:20.3512045Z { 2024-12-18T00:01:20.3512377Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3512810Z "size": 1860, 2024-12-18T00:01:20.3513244Z "digest": "sha256:3a7f27881c57c1035aa6d2109802ac6da11afd93eaa81d48ba8e07f910242248" 2024-12-18T00:01:20.3513739Z }, 2024-12-18T00:01:20.3513945Z { 2024-12-18T00:01:20.3514346Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3514782Z "size": 700, 2024-12-18T00:01:20.3515228Z "digest": "sha256:7feada69620ed9505fc525cf4e6d81f7d25589cb3c26fe8fef7479807372f91a" 2024-12-18T00:01:20.3515735Z }, 2024-12-18T00:01:20.3515945Z { 2024-12-18T00:01:20.3516277Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3516717Z "size": 478, 2024-12-18T00:01:20.3517148Z "digest": "sha256:1f019fb4d944db0c3f6f320a880353d7ebeae6df3456f7e492b7a6c19e7430f2" 2024-12-18T00:01:20.3517643Z }, 2024-12-18T00:01:20.3517845Z { 2024-12-18T00:01:20.3518172Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3518612Z "size": 2953611806, 2024-12-18T00:01:20.3519055Z "digest": "sha256:f900f32845825b74c796aa12c8ca7412b38ae05813a53257f53b49bb2b501bcd" 2024-12-18T00:01:20.3519545Z }, 2024-12-18T00:01:20.3519750Z { 2024-12-18T00:01:20.3520078Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3520521Z "size": 380, 2024-12-18T00:01:20.3520951Z "digest": "sha256:03ef53c6b863b93f161c2bf5eb906c9cae887d4e3cb793fa06212162c09b3584" 2024-12-18T00:01:20.3521454Z }, 2024-12-18T00:01:20.3521659Z { 2024-12-18T00:01:20.3521987Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3522424Z "size": 12909, 2024-12-18T00:01:20.3522861Z "digest": "sha256:8c33d01f9e39d69c625c8a507af96fef2fa8f95ff019b7e060062fbd98da0a3b" 2024-12-18T00:01:20.3523462Z }, 2024-12-18T00:01:20.3523845Z { 2024-12-18T00:01:20.3524249Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3524741Z "size": 863, 2024-12-18T00:01:20.3525224Z "digest": "sha256:49be5867b6ab28064a789b446635d6ec60a1c639deffec9c581499d09a20d31a" 2024-12-18T00:01:20.3525778Z }, 2024-12-18T00:01:20.3526002Z { 2024-12-18T00:01:20.3526369Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3526864Z "size": 106, 2024-12-18T00:01:20.3527342Z "digest": "sha256:2c921c2b926223806ddf443e5a591ae34c19c50f553adea16a2ae986a821b47e" 2024-12-18T00:01:20.3527898Z }, 2024-12-18T00:01:20.3528210Z { 2024-12-18T00:01:20.3528583Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3529077Z "size": 504, 2024-12-18T00:01:20.3529554Z "digest": "sha256:f276569c48f345d048de43d8296869ea278af215142ba5741b765adeac6c238f" 2024-12-18T00:01:20.3530106Z }, 2024-12-18T00:01:20.3530334Z { 2024-12-18T00:01:20.3530706Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3531196Z "size": 121477378, 2024-12-18T00:01:20.3566502Z "digest": "sha256:cac6488770e66c796a9bae0e25b007efd1ff47a360bc5714b7fd2946a39ff59b" 2024-12-18T00:01:20.3567078Z }, 2024-12-18T00:01:20.3567303Z { 2024-12-18T00:01:20.3567685Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3568173Z "size": 109, 2024-12-18T00:01:20.3568664Z "digest": "sha256:9695d05a4951d9c024dd2d0726270c2acd5c81cbb1986ae03f60e167357f43cf" 2024-12-18T00:01:20.3569206Z }, 2024-12-18T00:01:20.3569416Z { 2024-12-18T00:01:20.3569802Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3570281Z "size": 489, 2024-12-18T00:01:20.3570760Z "digest": "sha256:2857ab4d43aab8ad9d8ba6af295f20bfb83a30f5442e8502e1db44c385916241" 2024-12-18T00:01:20.3571307Z }, 2024-12-18T00:01:20.3571516Z { 2024-12-18T00:01:20.3571888Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3572370Z "size": 296, 2024-12-18T00:01:20.3572843Z "digest": "sha256:fcdec1f5e2fde1f1945361f0d3c56b33a33f65d42378ff4ca797060040518bf0" 2024-12-18T00:01:20.3573385Z }, 2024-12-18T00:01:20.3573622Z { 2024-12-18T00:01:20.3573988Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3574464Z "size": 103, 2024-12-18T00:01:20.3574932Z "digest": "sha256:ff1478fbefd979ac80d1b176b2c3f6ddcf4cecf685433398b03bc4f0cbd06110" 2024-12-18T00:01:20.3575700Z }, 2024-12-18T00:01:20.3575918Z { 2024-12-18T00:01:20.3576293Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3576770Z "size": 1476, 2024-12-18T00:01:20.3577343Z "digest": "sha256:f8a6d90f0b3ef982ba39428cb0d40dc4c9e03df7c4011258916d4d5dab1bef27" 2024-12-18T00:01:20.3577968Z }, 2024-12-18T00:01:20.3578168Z { 2024-12-18T00:01:20.3578515Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3579078Z "size": 430258066, 2024-12-18T00:01:20.3579494Z "digest": "sha256:011542937262656639e466e38068de398e4565825a5b1d02acc425bd4da3ed3d" 2024-12-18T00:01:20.3579952Z }, 2024-12-18T00:01:20.3580141Z { 2024-12-18T00:01:20.3580468Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3580890Z "size": 162, 2024-12-18T00:01:20.3581304Z "digest": "sha256:812caf3e433816d6c7bea249be5ff35aa976d1d044d32c02741ac161aade0327" 2024-12-18T00:01:20.3581787Z }, 2024-12-18T00:01:20.3581974Z { 2024-12-18T00:01:20.3582300Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3582729Z "size": 564, 2024-12-18T00:01:20.3583135Z "digest": "sha256:7e2aa240b70cc8e1f87f625b28c79751d71f025760c2402d0c695d6333847c71" 2024-12-18T00:01:20.3583602Z }, 2024-12-18T00:01:20.3583793Z { 2024-12-18T00:01:20.3584116Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3584538Z "size": 35806330, 2024-12-18T00:01:20.3584962Z "digest": "sha256:a0e5fd2630de64c400e6cc5c43094e8ac4550325904f24db8f710842f33cc348" 2024-12-18T00:01:20.3585432Z }, 2024-12-18T00:01:20.3585624Z { 2024-12-18T00:01:20.3585946Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3586366Z "size": 104, 2024-12-18T00:01:20.3586780Z "digest": "sha256:a7253f4f13102b2c05f098066e2c11157d6b24c4cb879b3f3f9a5e29e73aa482" 2024-12-18T00:01:20.3587255Z }, 2024-12-18T00:01:20.3587449Z { 2024-12-18T00:01:20.3587922Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3588346Z "size": 425, 2024-12-18T00:01:20.3588860Z "digest": "sha256:d4bbe33feb5a0018de6d8445ad79aa990043b624c90752a4f264235b00ce9c71" 2024-12-18T00:01:20.3589344Z }, 2024-12-18T00:01:20.3589537Z { 2024-12-18T00:01:20.3589869Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3590297Z "size": 20262180, 2024-12-18T00:01:20.3590744Z "digest": "sha256:c7da3e1fda207aae43390fd25dc953ee6bb0fb29d6f92fe8ea45e25af003040e" 2024-12-18T00:01:20.3591241Z }, 2024-12-18T00:01:20.3591427Z { 2024-12-18T00:01:20.3591758Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3592188Z "size": 643, 2024-12-18T00:01:20.3592610Z "digest": "sha256:a1fa96c8688be0cd5c869e56cb39672774ca1466e3b55b296c07e6ddafc196db" 2024-12-18T00:01:20.3593084Z }, 2024-12-18T00:01:20.3593281Z { 2024-12-18T00:01:20.3593602Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3594023Z "size": 700, 2024-12-18T00:01:20.3594443Z "digest": "sha256:7feada69620ed9505fc525cf4e6d81f7d25589cb3c26fe8fef7479807372f91a" 2024-12-18T00:01:20.3594916Z }, 2024-12-18T00:01:20.3595102Z { 2024-12-18T00:01:20.3595428Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3595848Z "size": 141, 2024-12-18T00:01:20.3596262Z "digest": "sha256:600e7beb6ed65b70903fb3d10a7720f76a0e7ed4225e6a2d974edab4ded9985c" 2024-12-18T00:01:20.3596737Z }, 2024-12-18T00:01:20.3596924Z { 2024-12-18T00:01:20.3597250Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3597666Z "size": 136, 2024-12-18T00:01:20.3598078Z "digest": "sha256:7fee2a426d6e9e46b8fa661021c0cb6cbe7f52186061b61d7f32df2ec7f1dbae" 2024-12-18T00:01:20.3598551Z }, 2024-12-18T00:01:20.3598739Z { 2024-12-18T00:01:20.3599115Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3599537Z "size": 32, 2024-12-18T00:01:20.3599970Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:01:20.3600457Z }, 2024-12-18T00:01:20.3600661Z { 2024-12-18T00:01:20.3601003Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3601440Z "size": 196, 2024-12-18T00:01:20.3601863Z "digest": "sha256:321081f536b035434bb8104a9d93a930eba4415dbeffb4ce926a7c9560a818a9" 2024-12-18T00:01:20.3602340Z }, 2024-12-18T00:01:20.3602711Z { 2024-12-18T00:01:20.3603077Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3603817Z "size": 565, 2024-12-18T00:01:20.3604315Z "digest": "sha256:b2671014f386836740538595eb3d3d98580b1d88d3d63ade2ac59313e53526bd" 2024-12-18T00:01:20.3604848Z }, 2024-12-18T00:01:20.3605077Z { 2024-12-18T00:01:20.3605466Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3605973Z "size": 43163802, 2024-12-18T00:01:20.3606469Z "digest": "sha256:94a3028693ada8037297f5512a3a90c4d307bf06337ef43fcbce578505dbb02d" 2024-12-18T00:01:20.3607011Z }, 2024-12-18T00:01:20.3607247Z { 2024-12-18T00:01:20.3607634Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3608131Z "size": 106, 2024-12-18T00:01:20.3608607Z "digest": "sha256:838f3043a48a3fccad46b59399809bd6a67048404a30c44f8d04272c826634c4" 2024-12-18T00:01:20.3609146Z }, 2024-12-18T00:01:20.3609379Z { 2024-12-18T00:01:20.3609763Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3610253Z "size": 1402, 2024-12-18T00:01:20.3610777Z "digest": "sha256:3bf4f9678a1173c728cf0aa3632bbc5de29b674a2c547de331ecd9d3dd86e8c4" 2024-12-18T00:01:20.3611326Z }, 2024-12-18T00:01:20.3611552Z { 2024-12-18T00:01:20.3611933Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3612427Z "size": 700, 2024-12-18T00:01:20.3612914Z "digest": "sha256:7feada69620ed9505fc525cf4e6d81f7d25589cb3c26fe8fef7479807372f91a" 2024-12-18T00:01:20.3613460Z }, 2024-12-18T00:01:20.3613689Z { 2024-12-18T00:01:20.3614153Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3614647Z "size": 137, 2024-12-18T00:01:20.3615140Z "digest": "sha256:cbd527c6febd14371f782c2752d8947f86adeca1e46e6e1c9dfc0dff6e32df6f" 2024-12-18T00:01:20.3615795Z }, 2024-12-18T00:01:20.3616138Z { 2024-12-18T00:01:20.3616482Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3616923Z "size": 120, 2024-12-18T00:01:20.3617346Z "digest": "sha256:70c75642ca1573a70d7ab208940c68c9ec0518797e55e05b621cef283f348bbd" 2024-12-18T00:01:20.3617829Z }, 2024-12-18T00:01:20.3618036Z { 2024-12-18T00:01:20.3618379Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3618815Z "size": 4886117515, 2024-12-18T00:01:20.3619273Z "digest": "sha256:975ffa249994efd3380d6baae778fb7f9ae31ef5208ceb19da52ab66f447401d" 2024-12-18T00:01:20.3619763Z }, 2024-12-18T00:01:20.3619964Z { 2024-12-18T00:01:20.3620308Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3620745Z "size": 176, 2024-12-18T00:01:20.3621161Z "digest": "sha256:1de45071cffa26dd3b9959d220f37f09e3ebc5282b49c8910ca1b7045f37c763" 2024-12-18T00:01:20.3621659Z }, 2024-12-18T00:01:20.3621864Z { 2024-12-18T00:01:20.3622205Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3622641Z "size": 908, 2024-12-18T00:01:20.3623049Z "digest": "sha256:6f1f1c73799125dbfc36617f78ae22c9507955fb3870b11886da704f63cd79b9" 2024-12-18T00:01:20.3623538Z }, 2024-12-18T00:01:20.3623742Z { 2024-12-18T00:01:20.3624083Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3624521Z "size": 700, 2024-12-18T00:01:20.3624937Z "digest": "sha256:7feada69620ed9505fc525cf4e6d81f7d25589cb3c26fe8fef7479807372f91a" 2024-12-18T00:01:20.3625489Z }, 2024-12-18T00:01:20.3625693Z { 2024-12-18T00:01:20.3626037Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3626479Z "size": 135, 2024-12-18T00:01:20.3626894Z "digest": "sha256:8da387044f457792831b74e2facde5dc612cb058a3fcc22e503a778ad0e9125e" 2024-12-18T00:01:20.3627391Z }, 2024-12-18T00:01:20.3627597Z { 2024-12-18T00:01:20.3627937Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3628380Z "size": 32, 2024-12-18T00:01:20.3628797Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:01:20.3629297Z }, 2024-12-18T00:01:20.3629500Z { 2024-12-18T00:01:20.3629840Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3630273Z "size": 159, 2024-12-18T00:01:20.3630688Z "digest": "sha256:25b1aa58f5caff295c81a716174c70d1207b7efc44a594cb4b4c730d11ca6f58" 2024-12-18T00:01:20.3631190Z }, 2024-12-18T00:01:20.3631390Z { 2024-12-18T00:01:20.3631731Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3632169Z "size": 1931, 2024-12-18T00:01:20.3632595Z "digest": "sha256:f4c8ff23e407e7f06a6b9e83db042522c9cb14a2872e033bf0aec10e0cd68af8" 2024-12-18T00:01:20.3633094Z }, 2024-12-18T00:01:20.3633299Z { 2024-12-18T00:01:20.3633640Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3634082Z "size": 196807254, 2024-12-18T00:01:20.3634527Z "digest": "sha256:d9a1d16ac9fb7fb0a5b04cf9bd8a051fedd0f6ef1467103fc8796790e941e1bb" 2024-12-18T00:01:20.3635032Z }, 2024-12-18T00:01:20.3635242Z { 2024-12-18T00:01:20.3635585Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3636022Z "size": 163, 2024-12-18T00:01:20.3636429Z "digest": "sha256:4c24861203037ee0fc72cda05c5446739232f8fc027622a4982a392434699737" 2024-12-18T00:01:20.3636917Z }, 2024-12-18T00:01:20.3637122Z { 2024-12-18T00:01:20.3637463Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3637897Z "size": 7942, 2024-12-18T00:01:20.3638390Z "digest": "sha256:1c1a1b6b59a8cfdf0a6b0dd0fbf35ea01e8d3d9fbfbf7a1cc3c5129697842953" 2024-12-18T00:01:20.3638913Z }, 2024-12-18T00:01:20.3639126Z { 2024-12-18T00:01:20.3639473Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3639915Z "size": 8071, 2024-12-18T00:01:20.3640343Z "digest": "sha256:65ab75609fc9c1a8878dee49da05ea4c3cae04ec5ecd96bff7a59ba5f6716161" 2024-12-18T00:01:20.3640854Z }, 2024-12-18T00:01:20.3641065Z { 2024-12-18T00:01:20.3641416Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3641857Z "size": 304, 2024-12-18T00:01:20.3642273Z "digest": "sha256:41fe6d82c37a17ff1a0e760821e5ffd58423b2c50138348b2d99214fa8c7f7b9" 2024-12-18T00:01:20.3642773Z }, 2024-12-18T00:01:20.3642986Z { 2024-12-18T00:01:20.3643420Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3644067Z "size": 7632783, 2024-12-18T00:01:20.3644615Z "digest": "sha256:9456ad12e83f1f01e0a64ef29e45b8ce1657f0c45a5f334956fed566782f34d9" 2024-12-18T00:01:20.3645170Z }, 2024-12-18T00:01:20.3645403Z { 2024-12-18T00:01:20.3645787Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3646278Z "size": 108, 2024-12-18T00:01:20.3646757Z "digest": "sha256:7dd812a149578ae7aa4d747b9c0cfdd91a1edc15b63e7ec826ca42a601c7c03f" 2024-12-18T00:01:20.3647329Z }, 2024-12-18T00:01:20.3647558Z { 2024-12-18T00:01:20.3647938Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3648430Z "size": 54145664, 2024-12-18T00:01:20.3648919Z "digest": "sha256:e47d2dd30bb9d7d5f7295bbf4b68683cf7e0a3ec443e3d47696213edb61bb0ad" 2024-12-18T00:01:20.3649473Z }, 2024-12-18T00:01:20.3649699Z { 2024-12-18T00:01:20.3650079Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3650641Z "size": 486, 2024-12-18T00:01:20.3651116Z "digest": "sha256:9a255e925aee3dd6711c86af0e487d1c5ff1b2cde447bd2f64f5cc6aa08a3666" 2024-12-18T00:01:20.3651685Z }, 2024-12-18T00:01:20.3651916Z { 2024-12-18T00:01:20.3652305Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3652797Z "size": 1373229207, 2024-12-18T00:01:20.3653283Z "digest": "sha256:266e2409e545ef9c13ef48d69773c27f5ebe37e17c77808c473870b99ccaceca" 2024-12-18T00:01:20.3653830Z }, 2024-12-18T00:01:20.3654057Z { 2024-12-18T00:01:20.3654440Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3654926Z "size": 106, 2024-12-18T00:01:20.3655386Z "digest": "sha256:100d759b6098f93451cdf0ab691600451c05cbf8de2592d370f32f6be20b6127" 2024-12-18T00:01:20.3656042Z }, 2024-12-18T00:01:20.3656244Z { 2024-12-18T00:01:20.3656583Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3657021Z "size": 568, 2024-12-18T00:01:20.3657438Z "digest": "sha256:a3111847bfa2e9e8629c6c7a6cbdd52dde3359882add44762805cb47464c285f" 2024-12-18T00:01:20.3657935Z }, 2024-12-18T00:01:20.3658143Z { 2024-12-18T00:01:20.3658486Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3658925Z "size": 36532350, 2024-12-18T00:01:20.3659372Z "digest": "sha256:7e6ea32f1b1cfa63f3df243c6bfa8b2659be396b488adcbdf9a9c0d9a1e65193" 2024-12-18T00:01:20.3659866Z }, 2024-12-18T00:01:20.3660070Z { 2024-12-18T00:01:20.3660410Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3660846Z "size": 111, 2024-12-18T00:01:20.3661260Z "digest": "sha256:38bf4ff6a6aca492ad60288413a8a2462e07f715d05252617bb4fa942c0ca565" 2024-12-18T00:01:20.3661755Z }, 2024-12-18T00:01:20.3661958Z { 2024-12-18T00:01:20.3662294Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3662720Z "size": 531, 2024-12-18T00:01:20.3663114Z "digest": "sha256:231d0f8577895b4225081015421c272a6f6c601e9b367fcf49106b9a8e3ff74c" 2024-12-18T00:01:20.3663590Z }, 2024-12-18T00:01:20.3663788Z { 2024-12-18T00:01:20.3664207Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3664648Z "size": 26033, 2024-12-18T00:01:20.3665084Z "digest": "sha256:0fe4c18889ad4b2d23fac8e5c1a1bba4cc65ed4abf30d84ab871f507fd27ee03" 2024-12-18T00:01:20.3665927Z }, 2024-12-18T00:01:20.3666155Z { 2024-12-18T00:01:20.3666614Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3667087Z "size": 106, 2024-12-18T00:01:20.3667574Z "digest": "sha256:c5d910c7fb9c4e239cfeafa9695f9e8b9a2892da466239b362451bb0d9a539df" 2024-12-18T00:01:20.3668130Z }, 2024-12-18T00:01:20.3668351Z { 2024-12-18T00:01:20.3668721Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3669185Z "size": 32, 2024-12-18T00:01:20.3669662Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:01:20.3670221Z }, 2024-12-18T00:01:20.3670455Z { 2024-12-18T00:01:20.3670843Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3671321Z "size": 32, 2024-12-18T00:01:20.3671807Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:01:20.3672371Z }, 2024-12-18T00:01:20.3672599Z { 2024-12-18T00:01:20.3672980Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3673457Z "size": 32, 2024-12-18T00:01:20.3673938Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:01:20.3674494Z }, 2024-12-18T00:01:20.3674719Z { 2024-12-18T00:01:20.3675102Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:20.3675581Z "size": 32, 2024-12-18T00:01:20.3676058Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:01:20.3676725Z } 2024-12-18T00:01:20.3676955Z ] 2024-12-18T00:01:20.3677187Z } 2024-12-18T00:01:20.3677423Z + exit 0 2024-12-18T00:01:20.3710998Z ##[group]Run set -eux 2024-12-18T00:01:20.3711322Z set -eux 2024-12-18T00:01:20.3712303Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin 2024-12-18T00:01:20.3719423Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:20.3719820Z env: 2024-12-18T00:01:20.3720041Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:20.3720318Z ##[endgroup] 2024-12-18T00:01:20.3748088Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2024-12-18T00:01:20.3748666Z + jq --raw-output .SecretString 2024-12-18T00:01:20.3750121Z + jq -r .docker_hub_readonly_token 2024-12-18T00:01:20.3750574Z + docker login --username pytorchbot --password-stdin 2024-12-18T00:01:21.0006190Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:01:21.0006995Z Configure a credential helper to remove this warning. See 2024-12-18T00:01:21.0007683Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:01:21.0008158Z 2024-12-18T00:01:21.0008281Z Login Succeeded 2024-12-18T00:01:21.0101866Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-12-18T00:01:21.0102249Z tag=${ECR_DOCKER_IMAGE##*/} 2024-12-18T00:01:21.0102681Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-12-18T00:01:21.0108503Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:21.0108909Z env: 2024-12-18T00:01:21.0109142Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:21.0109952Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:21.0110789Z ##[endgroup] 2024-12-18T00:01:21.0137803Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9-45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:21.0193632Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@release/2.6 2024-12-18T00:01:21.0194130Z with: 2024-12-18T00:01:21.0194871Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:21.0195808Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:21.0196225Z env: 2024-12-18T00:01:21.0196445Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:21.0196730Z ##[endgroup] 2024-12-18T00:01:21.0219798Z ##[group]Run set -x 2024-12-18T00:01:21.0220085Z set -x 2024-12-18T00:01:21.0220329Z set +e 2024-12-18T00:01:21.0220554Z  2024-12-18T00:01:21.0220786Z login() { 2024-12-18T00:01:21.0221292Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-12-18T00:01:21.0221852Z } 2024-12-18T00:01:21.0222076Z  2024-12-18T00:01:21.0222334Z retry () { 2024-12-18T00:01:21.0222626Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-12-18T00:01:21.0222953Z } 2024-12-18T00:01:21.0223177Z  2024-12-18T00:01:21.0223432Z retry login "${DOCKER_REGISTRY}" 2024-12-18T00:01:21.0223767Z  2024-12-18T00:01:21.0223992Z set -e 2024-12-18T00:01:21.0224342Z # ignore output since only exit code is used for conditional 2024-12-18T00:01:21.0224863Z # only pull docker image if it's not available locally 2024-12-18T00:01:21.0225438Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-12-18T00:01:21.0225971Z  retry docker pull "${DOCKER_IMAGE}" 2024-12-18T00:01:21.0226312Z fi 2024-12-18T00:01:21.0231435Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:21.0231961Z env: 2024-12-18T00:01:21.0232195Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:21.0232997Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:21.0233923Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:21.0234336Z ##[endgroup] 2024-12-18T00:01:21.0256554Z + set +e 2024-12-18T00:01:21.0256957Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:21.0257459Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:21.0260487Z + aws ecr get-login-password --region us-east-1 2024-12-18T00:01:21.0261535Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:21.5839637Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:01:21.5840369Z Configure a credential helper to remove this warning. See 2024-12-18T00:01:21.5841084Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:01:21.5841549Z 2024-12-18T00:01:21.5841684Z Login Succeeded 2024-12-18T00:01:21.5856842Z + set -e 2024-12-18T00:01:21.5857854Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:21.5997729Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:21.5999215Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:21.8498912Z 45e1356b47a284893081276eff3000b7b534f3b1: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9 2024-12-18T00:01:21.8499758Z 7a2c55901189: Pulling fs layer 2024-12-18T00:01:21.8500150Z 09d415c238d7: Pulling fs layer 2024-12-18T00:01:21.8500662Z 9fe6e2e61518: Pulling fs layer 2024-12-18T00:01:21.8501000Z 41f16248e682: Pulling fs layer 2024-12-18T00:01:21.8501553Z 95d7b7817039: Pulling fs layer 2024-12-18T00:01:21.8501892Z 8f6c90485347: Pulling fs layer 2024-12-18T00:01:21.8502226Z ab17245097e4: Pulling fs layer 2024-12-18T00:01:21.8502571Z dfecd7e9912b: Pulling fs layer 2024-12-18T00:01:21.8502915Z 464a8f745445: Pulling fs layer 2024-12-18T00:01:21.8503235Z d67c111fa588: Pulling fs layer 2024-12-18T00:01:21.8503903Z 0723abde19c0: Pulling fs layer 2024-12-18T00:01:21.8504303Z 38b3bd834f10: Pulling fs layer 2024-12-18T00:01:21.8504742Z e45523662aa5: Pulling fs layer 2024-12-18T00:01:21.8505270Z 95d7b7817039: Waiting 2024-12-18T00:01:21.8505722Z 5a6d97ee6e47: Pulling fs layer 2024-12-18T00:01:21.8506199Z 8b9cee92f408: Pulling fs layer 2024-12-18T00:01:21.8506642Z 05a25e0e092f: Pulling fs layer 2024-12-18T00:01:21.8507071Z 32f9bd3454b2: Pulling fs layer 2024-12-18T00:01:21.8507533Z 7cef0be11b19: Pulling fs layer 2024-12-18T00:01:21.8507986Z 3a7f27881c57: Pulling fs layer 2024-12-18T00:01:21.8508406Z 7feada69620e: Pulling fs layer 2024-12-18T00:01:21.8508907Z 1f019fb4d944: Pulling fs layer 2024-12-18T00:01:21.8509356Z f900f3284582: Pulling fs layer 2024-12-18T00:01:21.8509725Z 03ef53c6b863: Pulling fs layer 2024-12-18T00:01:21.8510211Z 8c33d01f9e39: Pulling fs layer 2024-12-18T00:01:21.8510666Z 49be5867b6ab: Pulling fs layer 2024-12-18T00:01:21.8511115Z 2c921c2b9262: Pulling fs layer 2024-12-18T00:01:21.8511582Z f276569c48f3: Pulling fs layer 2024-12-18T00:01:21.8511973Z cac6488770e6: Pulling fs layer 2024-12-18T00:01:21.8512419Z 9695d05a4951: Pulling fs layer 2024-12-18T00:01:21.8512887Z 2857ab4d43aa: Pulling fs layer 2024-12-18T00:01:21.8513337Z fcdec1f5e2fd: Pulling fs layer 2024-12-18T00:01:21.8513820Z 8f6c90485347: Waiting 2024-12-18T00:01:21.8514179Z ab17245097e4: Waiting 2024-12-18T00:01:21.8514592Z ff1478fbefd9: Pulling fs layer 2024-12-18T00:01:21.8515065Z dfecd7e9912b: Waiting 2024-12-18T00:01:21.8515622Z f8a6d90f0b3e: Pulling fs layer 2024-12-18T00:01:21.8516042Z 011542937262: Pulling fs layer 2024-12-18T00:01:21.8516818Z 812caf3e4338: Pulling fs layer 2024-12-18T00:01:21.8517275Z 7e2aa240b70c: Pulling fs layer 2024-12-18T00:01:21.8517683Z 464a8f745445: Waiting 2024-12-18T00:01:21.8518150Z 41f16248e682: Waiting 2024-12-18T00:01:21.8518561Z a0e5fd2630de: Pulling fs layer 2024-12-18T00:01:21.8519013Z d67c111fa588: Waiting 2024-12-18T00:01:21.8519390Z a7253f4f1310: Pulling fs layer 2024-12-18T00:01:21.8519825Z 7cef0be11b19: Waiting 2024-12-18T00:01:21.8520239Z f900f3284582: Waiting 2024-12-18T00:01:21.8520653Z 0723abde19c0: Waiting 2024-12-18T00:01:21.8521233Z d4bbe33feb5a: Pulling fs layer 2024-12-18T00:01:21.8521624Z 3a7f27881c57: Waiting 2024-12-18T00:01:21.8522043Z c7da3e1fda20: Pulling fs layer 2024-12-18T00:01:21.8522491Z a1fa96c8688b: Pulling fs layer 2024-12-18T00:01:21.8522933Z 03ef53c6b863: Waiting 2024-12-18T00:01:21.8523442Z 600e7beb6ed6: Pulling fs layer 2024-12-18T00:01:21.8524044Z 7fee2a426d6e: Pulling fs layer 2024-12-18T00:01:21.8524541Z 2c921c2b9262: Waiting 2024-12-18T00:01:21.8524957Z 4f4fb700ef54: Pulling fs layer 2024-12-18T00:01:21.8525375Z 38b3bd834f10: Waiting 2024-12-18T00:01:21.8525834Z 1f019fb4d944: Waiting 2024-12-18T00:01:21.8526199Z 321081f536b0: Pulling fs layer 2024-12-18T00:01:21.8526610Z ff1478fbefd9: Waiting 2024-12-18T00:01:21.8527256Z b2671014f386: Pulling fs layer 2024-12-18T00:01:21.8527849Z 2857ab4d43aa: Waiting 2024-12-18T00:01:21.8528278Z 94a3028693ad: Pulling fs layer 2024-12-18T00:01:21.8528840Z f8a6d90f0b3e: Waiting 2024-12-18T00:01:21.8529269Z 838f3043a48a: Pulling fs layer 2024-12-18T00:01:21.8529736Z 3bf4f9678a11: Pulling fs layer 2024-12-18T00:01:21.8530192Z f276569c48f3: Waiting 2024-12-18T00:01:21.8530583Z c7da3e1fda20: Waiting 2024-12-18T00:01:21.8530956Z cbd527c6febd: Pulling fs layer 2024-12-18T00:01:21.8531417Z 7feada69620e: Waiting 2024-12-18T00:01:21.8531819Z 70c75642ca15: Pulling fs layer 2024-12-18T00:01:21.8532357Z fcdec1f5e2fd: Waiting 2024-12-18T00:01:21.8532766Z d4bbe33feb5a: Waiting 2024-12-18T00:01:21.8533133Z 975ffa249994: Pulling fs layer 2024-12-18T00:01:21.8533606Z a1fa96c8688b: Waiting 2024-12-18T00:01:21.8534126Z 321081f536b0: Waiting 2024-12-18T00:01:21.8534516Z 1de45071cffa: Pulling fs layer 2024-12-18T00:01:21.8535034Z 49be5867b6ab: Waiting 2024-12-18T00:01:21.8535492Z 5a6d97ee6e47: Waiting 2024-12-18T00:01:21.8535861Z 32f9bd3454b2: Waiting 2024-12-18T00:01:21.8536302Z 6f1f1c737991: Pulling fs layer 2024-12-18T00:01:21.8536735Z 8da387044f45: Pulling fs layer 2024-12-18T00:01:21.8537233Z 25b1aa58f5ca: Pulling fs layer 2024-12-18T00:01:21.8537626Z 3bf4f9678a11: Waiting 2024-12-18T00:01:21.8538129Z 7e2aa240b70c: Waiting 2024-12-18T00:01:21.8538534Z 8b9cee92f408: Waiting 2024-12-18T00:01:21.8538954Z f4c8ff23e407: Pulling fs layer 2024-12-18T00:01:21.8539461Z 05a25e0e092f: Waiting 2024-12-18T00:01:21.8539820Z d9a1d16ac9fb: Pulling fs layer 2024-12-18T00:01:21.8540261Z 7fee2a426d6e: Waiting 2024-12-18T00:01:21.8540730Z 4c2486120303: Pulling fs layer 2024-12-18T00:01:21.8541197Z 1c1a1b6b59a8: Pulling fs layer 2024-12-18T00:01:21.8541754Z 94a3028693ad: Waiting 2024-12-18T00:01:21.8542264Z cbd527c6febd: Waiting 2024-12-18T00:01:21.8557431Z 812caf3e4338: Waiting 2024-12-18T00:01:21.8557770Z 65ab75609fc9: Pulling fs layer 2024-12-18T00:01:21.8558105Z 41fe6d82c37a: Pulling fs layer 2024-12-18T00:01:21.8558428Z 4f4fb700ef54: Waiting 2024-12-18T00:01:21.8558714Z 6f1f1c737991: Waiting 2024-12-18T00:01:21.8558988Z b2671014f386: Waiting 2024-12-18T00:01:21.8559280Z 9456ad12e83f: Pulling fs layer 2024-12-18T00:01:21.8559585Z 600e7beb6ed6: Waiting 2024-12-18T00:01:21.8560108Z 7dd812a14957: Pulling fs layer 2024-12-18T00:01:21.8560444Z e47d2dd30bb9: Pulling fs layer 2024-12-18T00:01:21.8560775Z 9a255e925aee: Pulling fs layer 2024-12-18T00:01:21.8561184Z 266e2409e545: Pulling fs layer 2024-12-18T00:01:21.8561509Z 100d759b6098: Pulling fs layer 2024-12-18T00:01:21.8561822Z 9695d05a4951: Waiting 2024-12-18T00:01:21.8562110Z a3111847bfa2: Pulling fs layer 2024-12-18T00:01:21.8562579Z 4c2486120303: Waiting 2024-12-18T00:01:21.8562842Z 975ffa249994: Waiting 2024-12-18T00:01:21.8563124Z 1c1a1b6b59a8: Waiting 2024-12-18T00:01:21.8563523Z 7e6ea32f1b1c: Pulling fs layer 2024-12-18T00:01:21.8564021Z 70c75642ca15: Waiting 2024-12-18T00:01:21.8564367Z 38bf4ff6a6ac: Pulling fs layer 2024-12-18T00:01:21.8564692Z 231d0f857789: Pulling fs layer 2024-12-18T00:01:21.8565021Z 1de45071cffa: Waiting 2024-12-18T00:01:21.8565520Z 0fe4c18889ad: Pulling fs layer 2024-12-18T00:01:21.8565844Z 266e2409e545: Waiting 2024-12-18T00:01:21.8566144Z c5d910c7fb9c: Pulling fs layer 2024-12-18T00:01:21.8566458Z 9a255e925aee: Waiting 2024-12-18T00:01:21.8566750Z a0e5fd2630de: Waiting 2024-12-18T00:01:21.8567038Z 9456ad12e83f: Waiting 2024-12-18T00:01:21.8567322Z 41fe6d82c37a: Waiting 2024-12-18T00:01:21.8567610Z e47d2dd30bb9: Waiting 2024-12-18T00:01:21.8567886Z 25b1aa58f5ca: Waiting 2024-12-18T00:01:21.8568174Z 231d0f857789: Waiting 2024-12-18T00:01:21.8568458Z 38bf4ff6a6ac: Waiting 2024-12-18T00:01:21.8568755Z 7e6ea32f1b1c: Waiting 2024-12-18T00:01:21.8569031Z d9a1d16ac9fb: Waiting 2024-12-18T00:01:21.8569318Z 8da387044f45: Waiting 2024-12-18T00:01:21.8569608Z 838f3043a48a: Waiting 2024-12-18T00:01:21.8569896Z 65ab75609fc9: Waiting 2024-12-18T00:01:21.8570183Z e45523662aa5: Waiting 2024-12-18T00:01:21.8570454Z a7253f4f1310: Waiting 2024-12-18T00:01:21.8570740Z 7dd812a14957: Waiting 2024-12-18T00:01:21.8571027Z 100d759b6098: Waiting 2024-12-18T00:01:21.8571309Z a3111847bfa2: Waiting 2024-12-18T00:01:21.8571593Z c5d910c7fb9c: Waiting 2024-12-18T00:01:21.9981387Z 09d415c238d7: Verifying Checksum 2024-12-18T00:01:21.9981810Z 09d415c238d7: Download complete 2024-12-18T00:01:22.0832265Z 41f16248e682: Verifying Checksum 2024-12-18T00:01:22.0832748Z 41f16248e682: Download complete 2024-12-18T00:01:22.1851860Z 95d7b7817039: Verifying Checksum 2024-12-18T00:01:22.1852260Z 95d7b7817039: Download complete 2024-12-18T00:01:22.1913247Z 7a2c55901189: Verifying Checksum 2024-12-18T00:01:22.1913663Z 7a2c55901189: Download complete 2024-12-18T00:01:22.2716721Z ab17245097e4: Download complete 2024-12-18T00:01:22.3685849Z dfecd7e9912b: Download complete 2024-12-18T00:01:22.4498176Z 464a8f745445: Verifying Checksum 2024-12-18T00:01:22.4498701Z 464a8f745445: Download complete 2024-12-18T00:01:22.4803883Z 9fe6e2e61518: Verifying Checksum 2024-12-18T00:01:22.4804331Z 9fe6e2e61518: Download complete 2024-12-18T00:01:22.5796941Z 0723abde19c0: Verifying Checksum 2024-12-18T00:01:22.5797354Z 0723abde19c0: Download complete 2024-12-18T00:01:22.6550256Z 38b3bd834f10: Verifying Checksum 2024-12-18T00:01:22.6550972Z 38b3bd834f10: Download complete 2024-12-18T00:01:22.9226227Z 7a2c55901189: Pull complete 2024-12-18T00:01:23.1245145Z 09d415c238d7: Pull complete 2024-12-18T00:01:23.8822379Z 9fe6e2e61518: Pull complete 2024-12-18T00:01:23.9035909Z 41f16248e682: Pull complete 2024-12-18T00:01:23.9242405Z 95d7b7817039: Pull complete 2024-12-18T00:01:29.3008645Z e45523662aa5: Verifying Checksum 2024-12-18T00:01:29.3009115Z e45523662aa5: Download complete 2024-12-18T00:01:29.3987596Z 5a6d97ee6e47: Verifying Checksum 2024-12-18T00:01:29.3988004Z 5a6d97ee6e47: Download complete 2024-12-18T00:01:29.5447340Z 8b9cee92f408: Verifying Checksum 2024-12-18T00:01:29.5447778Z 8b9cee92f408: Download complete 2024-12-18T00:01:29.6401066Z 05a25e0e092f: Verifying Checksum 2024-12-18T00:01:29.6402046Z 05a25e0e092f: Download complete 2024-12-18T00:01:30.6240342Z 32f9bd3454b2: Verifying Checksum 2024-12-18T00:01:30.6240789Z 32f9bd3454b2: Download complete 2024-12-18T00:01:30.6931735Z 7cef0be11b19: Verifying Checksum 2024-12-18T00:01:30.6932176Z 7cef0be11b19: Download complete 2024-12-18T00:01:30.8202941Z 3a7f27881c57: Download complete 2024-12-18T00:01:30.9321911Z 7feada69620e: Verifying Checksum 2024-12-18T00:01:30.9322322Z 7feada69620e: Download complete 2024-12-18T00:01:31.0275878Z 1f019fb4d944: Download complete 2024-12-18T00:01:36.0878761Z 8f6c90485347: Verifying Checksum 2024-12-18T00:01:36.0879739Z 8f6c90485347: Download complete 2024-12-18T00:01:36.1859695Z 03ef53c6b863: Download complete 2024-12-18T00:01:36.2978563Z 8c33d01f9e39: Download complete 2024-12-18T00:01:36.3880622Z 49be5867b6ab: Verifying Checksum 2024-12-18T00:01:36.3881474Z 49be5867b6ab: Download complete 2024-12-18T00:01:36.4784497Z 2c921c2b9262: Verifying Checksum 2024-12-18T00:01:36.4784927Z 2c921c2b9262: Download complete 2024-12-18T00:01:36.5798321Z f276569c48f3: Download complete 2024-12-18T00:01:37.8919477Z cac6488770e6: Verifying Checksum 2024-12-18T00:01:37.8919956Z cac6488770e6: Download complete 2024-12-18T00:01:37.9695713Z 9695d05a4951: Download complete 2024-12-18T00:01:38.0638426Z 2857ab4d43aa: Verifying Checksum 2024-12-18T00:01:38.0638908Z 2857ab4d43aa: Download complete 2024-12-18T00:01:38.1471493Z fcdec1f5e2fd: Download complete 2024-12-18T00:01:38.2357957Z ff1478fbefd9: Verifying Checksum 2024-12-18T00:01:38.2358646Z ff1478fbefd9: Download complete 2024-12-18T00:01:38.3368248Z f8a6d90f0b3e: Verifying Checksum 2024-12-18T00:01:38.3368674Z f8a6d90f0b3e: Download complete 2024-12-18T00:01:43.3550244Z 011542937262: Verifying Checksum 2024-12-18T00:01:43.3550660Z 011542937262: Download complete 2024-12-18T00:01:43.4876168Z 812caf3e4338: Verifying Checksum 2024-12-18T00:01:43.4876635Z 812caf3e4338: Download complete 2024-12-18T00:01:43.5800222Z 7e2aa240b70c: Verifying Checksum 2024-12-18T00:01:43.5800672Z 7e2aa240b70c: Download complete 2024-12-18T00:01:44.0079331Z a0e5fd2630de: Verifying Checksum 2024-12-18T00:01:44.1129644Z a7253f4f1310: Verifying Checksum 2024-12-18T00:01:44.1130107Z a7253f4f1310: Download complete 2024-12-18T00:01:44.2149255Z d4bbe33feb5a: Download complete 2024-12-18T00:01:44.4998664Z c7da3e1fda20: Verifying Checksum 2024-12-18T00:01:44.6107453Z c7da3e1fda20: Download complete 2024-12-18T00:01:44.6107890Z a1fa96c8688b: Verifying Checksum 2024-12-18T00:01:44.6108235Z a1fa96c8688b: Download complete 2024-12-18T00:01:44.7204592Z 600e7beb6ed6: Verifying Checksum 2024-12-18T00:01:44.7205059Z 600e7beb6ed6: Download complete 2024-12-18T00:01:44.8032460Z 7fee2a426d6e: Verifying Checksum 2024-12-18T00:01:44.8032948Z 7fee2a426d6e: Download complete 2024-12-18T00:01:44.8121787Z 4f4fb700ef54: Verifying Checksum 2024-12-18T00:01:44.8122565Z 4f4fb700ef54: Download complete 2024-12-18T00:01:44.8890766Z 321081f536b0: Verifying Checksum 2024-12-18T00:01:44.8891203Z 321081f536b0: Download complete 2024-12-18T00:01:44.9641451Z b2671014f386: Download complete 2024-12-18T00:01:45.4606361Z 94a3028693ad: Verifying Checksum 2024-12-18T00:01:45.4606792Z 94a3028693ad: Download complete 2024-12-18T00:01:45.5987642Z 838f3043a48a: Verifying Checksum 2024-12-18T00:01:45.5988083Z 838f3043a48a: Download complete 2024-12-18T00:01:45.7095297Z 3bf4f9678a11: Verifying Checksum 2024-12-18T00:01:45.7095850Z 3bf4f9678a11: Download complete 2024-12-18T00:01:45.7766416Z cbd527c6febd: Verifying Checksum 2024-12-18T00:01:45.7766893Z cbd527c6febd: Download complete 2024-12-18T00:01:45.8725748Z 70c75642ca15: Verifying Checksum 2024-12-18T00:01:45.8726181Z 70c75642ca15: Download complete 2024-12-18T00:01:46.9823297Z d67c111fa588: Verifying Checksum 2024-12-18T00:01:46.9823752Z d67c111fa588: Download complete 2024-12-18T00:01:47.0576518Z 1de45071cffa: Verifying Checksum 2024-12-18T00:01:47.0576989Z 1de45071cffa: Download complete 2024-12-18T00:01:47.1371043Z 6f1f1c737991: Verifying Checksum 2024-12-18T00:01:47.1371483Z 6f1f1c737991: Download complete 2024-12-18T00:01:47.2470474Z 8da387044f45: Download complete 2024-12-18T00:01:47.3258443Z 25b1aa58f5ca: Verifying Checksum 2024-12-18T00:01:47.3258870Z 25b1aa58f5ca: Download complete 2024-12-18T00:01:47.4046455Z f4c8ff23e407: Verifying Checksum 2024-12-18T00:01:49.4227651Z f4c8ff23e407: Download complete 2024-12-18T00:01:49.4228343Z d9a1d16ac9fb: Verifying Checksum 2024-12-18T00:01:49.4228964Z d9a1d16ac9fb: Download complete 2024-12-18T00:01:49.5241014Z 4c2486120303: Verifying Checksum 2024-12-18T00:01:49.5241697Z 4c2486120303: Download complete 2024-12-18T00:01:49.6006538Z 1c1a1b6b59a8: Download complete 2024-12-18T00:01:49.6720627Z 65ab75609fc9: Verifying Checksum 2024-12-18T00:01:49.6721457Z 65ab75609fc9: Download complete 2024-12-18T00:01:49.7472722Z 41fe6d82c37a: Verifying Checksum 2024-12-18T00:01:49.7473170Z 41fe6d82c37a: Download complete 2024-12-18T00:01:49.8691685Z 8f6c90485347: Pull complete 2024-12-18T00:01:49.8991676Z 9456ad12e83f: Verifying Checksum 2024-12-18T00:01:49.8992396Z 9456ad12e83f: Download complete 2024-12-18T00:01:50.0071630Z 7dd812a14957: Download complete 2024-12-18T00:01:50.2394476Z ab17245097e4: Pull complete 2024-12-18T00:01:50.6141956Z dfecd7e9912b: Pull complete 2024-12-18T00:01:50.6439008Z e47d2dd30bb9: Verifying Checksum 2024-12-18T00:01:50.6439542Z e47d2dd30bb9: Download complete 2024-12-18T00:01:50.7408852Z 9a255e925aee: Download complete 2024-12-18T00:01:51.0306619Z 464a8f745445: Pull complete 2024-12-18T00:02:00.6314073Z f900f3284582: Verifying Checksum 2024-12-18T00:02:00.6314501Z f900f3284582: Download complete 2024-12-18T00:02:00.7162449Z 100d759b6098: Verifying Checksum 2024-12-18T00:02:00.7162855Z 100d759b6098: Download complete 2024-12-18T00:02:00.8390748Z a3111847bfa2: Verifying Checksum 2024-12-18T00:02:00.8391286Z a3111847bfa2: Download complete 2024-12-18T00:02:01.3018874Z 7e6ea32f1b1c: Verifying Checksum 2024-12-18T00:02:01.3019641Z 7e6ea32f1b1c: Download complete 2024-12-18T00:02:01.4016316Z 38bf4ff6a6ac: Verifying Checksum 2024-12-18T00:02:01.4017027Z 38bf4ff6a6ac: Download complete 2024-12-18T00:02:01.4907265Z 231d0f857789: Verifying Checksum 2024-12-18T00:02:01.4907738Z 231d0f857789: Download complete 2024-12-18T00:02:01.5830344Z 0fe4c18889ad: Download complete 2024-12-18T00:02:01.6579122Z c5d910c7fb9c: Verifying Checksum 2024-12-18T00:02:01.6579529Z c5d910c7fb9c: Download complete 2024-12-18T00:02:04.5282503Z 266e2409e545: Verifying Checksum 2024-12-18T00:02:04.5283393Z 266e2409e545: Download complete 2024-12-18T00:02:16.3154291Z d67c111fa588: Pull complete 2024-12-18T00:02:16.7627362Z 0723abde19c0: Pull complete 2024-12-18T00:02:17.1936002Z 38b3bd834f10: Pull complete 2024-12-18T00:02:26.7347428Z e45523662aa5: Pull complete 2024-12-18T00:02:27.1573254Z 5a6d97ee6e47: Pull complete 2024-12-18T00:02:27.5966586Z 8b9cee92f408: Pull complete 2024-12-18T00:02:28.0552361Z 05a25e0e092f: Pull complete 2024-12-18T00:02:30.3246130Z 32f9bd3454b2: Pull complete 2024-12-18T00:02:30.7535151Z 7cef0be11b19: Pull complete 2024-12-18T00:02:31.2406789Z 3a7f27881c57: Pull complete 2024-12-18T00:02:31.6971885Z 7feada69620e: Pull complete 2024-12-18T00:02:32.1404666Z 1f019fb4d944: Pull complete 2024-12-18T00:02:35.0036566Z 975ffa249994: Verifying Checksum 2024-12-18T00:02:35.0036991Z 975ffa249994: Download complete 2024-12-18T00:03:14.3954254Z f900f3284582: Pull complete 2024-12-18T00:03:14.8760561Z 03ef53c6b863: Pull complete 2024-12-18T00:03:15.3573489Z 8c33d01f9e39: Pull complete 2024-12-18T00:03:15.8062773Z 49be5867b6ab: Pull complete 2024-12-18T00:03:16.1968418Z 2c921c2b9262: Pull complete 2024-12-18T00:03:16.5099552Z f276569c48f3: Pull complete 2024-12-18T00:03:19.5283576Z cac6488770e6: Pull complete 2024-12-18T00:03:19.7526309Z 9695d05a4951: Pull complete 2024-12-18T00:03:19.8574716Z 2857ab4d43aa: Pull complete 2024-12-18T00:03:19.9746625Z fcdec1f5e2fd: Pull complete 2024-12-18T00:03:20.1738113Z ff1478fbefd9: Pull complete 2024-12-18T00:03:20.2540074Z f8a6d90f0b3e: Pull complete 2024-12-18T00:03:27.3897813Z 011542937262: Pull complete 2024-12-18T00:03:27.8943851Z 812caf3e4338: Pull complete 2024-12-18T00:03:28.3634517Z 7e2aa240b70c: Pull complete 2024-12-18T00:03:29.3633107Z a0e5fd2630de: Pull complete 2024-12-18T00:03:29.6432652Z a7253f4f1310: Pull complete 2024-12-18T00:03:30.0983743Z d4bbe33feb5a: Pull complete 2024-12-18T00:03:30.8142627Z c7da3e1fda20: Pull complete 2024-12-18T00:03:31.2738411Z a1fa96c8688b: Pull complete 2024-12-18T00:03:32.1128002Z 600e7beb6ed6: Pull complete 2024-12-18T00:03:32.5207042Z 7fee2a426d6e: Pull complete 2024-12-18T00:03:32.9723517Z 4f4fb700ef54: Pull complete 2024-12-18T00:03:33.4251874Z 321081f536b0: Pull complete 2024-12-18T00:03:33.8544807Z b2671014f386: Pull complete 2024-12-18T00:03:35.9585961Z 94a3028693ad: Pull complete 2024-12-18T00:03:36.4133866Z 838f3043a48a: Pull complete 2024-12-18T00:03:36.8706722Z 3bf4f9678a11: Pull complete 2024-12-18T00:03:37.7980641Z cbd527c6febd: Pull complete 2024-12-18T00:03:38.2397705Z 70c75642ca15: Pull complete 2024-12-18T00:04:51.4128042Z 975ffa249994: Pull complete 2024-12-18T00:04:51.7183897Z 1de45071cffa: Pull complete 2024-12-18T00:04:51.9777186Z 6f1f1c737991: Pull complete 2024-12-18T00:04:52.8494662Z 8da387044f45: Pull complete 2024-12-18T00:04:53.7732252Z 25b1aa58f5ca: Pull complete 2024-12-18T00:04:54.2427225Z f4c8ff23e407: Pull complete 2024-12-18T00:05:00.0547028Z d9a1d16ac9fb: Pull complete 2024-12-18T00:05:00.4855794Z 4c2486120303: Pull complete 2024-12-18T00:05:00.9705427Z 1c1a1b6b59a8: Pull complete 2024-12-18T00:05:01.3496714Z 65ab75609fc9: Pull complete 2024-12-18T00:05:01.6885686Z 41fe6d82c37a: Pull complete 2024-12-18T00:05:02.8896121Z 9456ad12e83f: Pull complete 2024-12-18T00:05:03.2647024Z 7dd812a14957: Pull complete 2024-12-18T00:05:05.2018598Z e47d2dd30bb9: Pull complete 2024-12-18T00:05:05.6886700Z 9a255e925aee: Pull complete 2024-12-18T00:05:18.4362165Z 266e2409e545: Pull complete 2024-12-18T00:05:18.8827699Z 100d759b6098: Pull complete 2024-12-18T00:05:19.2639432Z a3111847bfa2: Pull complete 2024-12-18T00:05:19.9898766Z 7e6ea32f1b1c: Pull complete 2024-12-18T00:05:20.4385824Z 38bf4ff6a6ac: Pull complete 2024-12-18T00:05:20.9195797Z 231d0f857789: Pull complete 2024-12-18T00:05:21.3645019Z 0fe4c18889ad: Pull complete 2024-12-18T00:05:21.8214934Z c5d910c7fb9c: Pull complete 2024-12-18T00:05:23.1005200Z Digest: sha256:52918bcdb0e370d1b9413099049464295e6c36e0a09cff6cff645b519e6da502 2024-12-18T00:05:23.1687350Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:05:23.2173092Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:05:23.2225470Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:05:23.2226538Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:05:23.2234555Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:05:23.2234994Z env: 2024-12-18T00:05:23.2235273Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:05:23.2235573Z ##[endgroup] 2024-12-18T00:05:23.2513983Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@release/2.6 2024-12-18T00:05:23.2514483Z with: 2024-12-18T00:05:23.2514733Z driver-version: 550.54.15 2024-12-18T00:05:23.2515015Z env: 2024-12-18T00:05:23.2515251Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:05:23.2515543Z ##[endgroup] 2024-12-18T00:05:23.2836595Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-12-18T00:05:23.2837055Z with: 2024-12-18T00:05:23.2837311Z timeout_minutes: 10 2024-12-18T00:05:23.2837593Z max_attempts: 3 2024-12-18T00:05:23.2868069Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-12-18T00:05:23.2899960Z retry_wait_seconds: 10 2024-12-18T00:05:23.2900278Z polling_interval_seconds: 1 2024-12-18T00:05:23.2900610Z warning_on_retry: true 2024-12-18T00:05:23.2900919Z continue_on_error: false 2024-12-18T00:05:23.2901205Z env: 2024-12-18T00:05:23.2901451Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:05:23.2901759Z DRIVER_VERSION: 550.54.15 2024-12-18T00:05:23.2902062Z ##[endgroup] 2024-12-18T00:05:23.4230907Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-12-18T00:05:23.4232241Z + pre_install_nvidia_driver_amzn2 2024-12-18T00:05:23.4233491Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-12-18T00:05:23.9023791Z No match for argument: nvidia-driver-latest-dkms 2024-12-18T00:05:23.9024269Z No packages marked for removal. 2024-12-18T00:05:23.9094014Z Dependencies resolved. 2024-12-18T00:05:23.9103677Z Nothing to do. 2024-12-18T00:05:23.9104365Z Complete! 2024-12-18T00:05:24.0490898Z + install_nvidia_driver_common 2024-12-18T00:05:24.0492806Z + echo 'Before installing NVIDIA driver' 2024-12-18T00:05:24.0494276Z + lspci 2024-12-18T00:05:24.0496189Z Before installing NVIDIA driver 2024-12-18T00:05:24.1815011Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-12-18T00:05:24.1816013Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-12-18T00:05:24.1817414Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-12-18T00:05:24.1818154Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-12-18T00:05:24.1818770Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-12-18T00:05:24.1819429Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-12-18T00:05:24.1820061Z 00:1b.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:24.1820650Z 00:1c.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:24.1821488Z 00:1d.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:24.1822076Z 00:1e.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:24.1822704Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-12-18T00:05:24.1823193Z + lsmod 2024-12-18T00:05:24.1840097Z Module Size Used by 2024-12-18T00:05:24.1840560Z xt_conntrack 16384 1 2024-12-18T00:05:24.1840951Z nft_chain_nat 16384 3 2024-12-18T00:05:24.1841282Z xt_MASQUERADE 20480 1 2024-12-18T00:05:24.1841663Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-12-18T00:05:24.1842070Z nf_conntrack_netlink 57344 0 2024-12-18T00:05:24.1842571Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-12-18T00:05:24.1843144Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-12-18T00:05:24.1843836Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-12-18T00:05:24.1844230Z xfrm_user 57344 1 2024-12-18T00:05:24.1844559Z xfrm_algo 16384 1 xfrm_user 2024-12-18T00:05:24.1844931Z xt_addrtype 16384 2 2024-12-18T00:05:24.1845264Z nft_compat 20480 4 2024-12-18T00:05:24.1845656Z nf_tables 311296 57 nft_compat,nft_chain_nat 2024-12-18T00:05:24.1846187Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-12-18T00:05:24.1846658Z br_netfilter 36864 0 2024-12-18T00:05:24.1847221Z bridge 323584 1 br_netfilter 2024-12-18T00:05:24.1847601Z stp 16384 1 bridge 2024-12-18T00:05:24.1847966Z llc 16384 2 bridge,stp 2024-12-18T00:05:24.1848329Z overlay 167936 0 2024-12-18T00:05:24.1848637Z tls 114688 0 2024-12-18T00:05:24.1848960Z nls_ascii 16384 1 2024-12-18T00:05:24.1849390Z nls_cp437 20480 1 2024-12-18T00:05:24.1849700Z vfat 24576 1 2024-12-18T00:05:24.1850125Z fat 86016 1 vfat 2024-12-18T00:05:24.1850435Z sunrpc 692224 1 2024-12-18T00:05:24.1850845Z ena 167936 0 2024-12-18T00:05:24.1851121Z i8042 45056 0 2024-12-18T00:05:24.1851414Z serio 28672 3 i8042 2024-12-18T00:05:24.1851727Z button 24576 0 2024-12-18T00:05:24.1852008Z skx_edac_common 28672 0 2024-12-18T00:05:24.1852306Z ghash_clmulni_intel 16384 0 2024-12-18T00:05:24.1852613Z sch_fq_codel 20480 33 2024-12-18T00:05:24.1852907Z dm_mod 188416 0 2024-12-18T00:05:24.1853194Z fuse 163840 1 2024-12-18T00:05:24.1853468Z configfs 57344 1 2024-12-18T00:05:24.1853754Z loop 36864 0 2024-12-18T00:05:24.1854047Z dax 45056 1 dm_mod 2024-12-18T00:05:24.1854362Z dmi_sysfs 20480 0 2024-12-18T00:05:24.1854655Z crc32_pclmul 16384 0 2024-12-18T00:05:24.1854942Z crc32c_intel 24576 0 2024-12-18T00:05:24.1855238Z efivarfs 24576 1 2024-12-18T00:05:24.1855704Z + modinfo nvidia 2024-12-18T00:05:24.1859044Z filename: /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-12-18T00:05:24.1860043Z alias: char-major-195-* 2024-12-18T00:05:24.1860438Z version: 550.54.15 2024-12-18T00:05:24.1860824Z supported: external 2024-12-18T00:05:24.1861117Z license: NVIDIA 2024-12-18T00:05:24.1861475Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-12-18T00:05:24.1861901Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-12-18T00:05:24.1862303Z srcversion: 833721318DA517F0C2FEC97 2024-12-18T00:05:24.1862702Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-12-18T00:05:24.1863110Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-12-18T00:05:24.1863525Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-12-18T00:05:24.1863899Z depends: i2c-core,drm 2024-12-18T00:05:24.1864212Z retpoline: Y 2024-12-18T00:05:24.1864638Z name: nvidia 2024-12-18T00:05:24.1865081Z vermagic: 6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-12-18T00:05:24.1865989Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-12-18T00:05:24.1866545Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-12-18T00:05:24.1867067Z parm: NVreg_ResmanDebugLevel:int 2024-12-18T00:05:24.1867446Z parm: NVreg_RmLogonRC:int 2024-12-18T00:05:24.1867828Z parm: NVreg_ModifyDeviceFiles:int 2024-12-18T00:05:24.1868209Z parm: NVreg_DeviceFileUID:int 2024-12-18T00:05:24.1868585Z parm: NVreg_DeviceFileGID:int 2024-12-18T00:05:24.1868964Z parm: NVreg_DeviceFileMode:int 2024-12-18T00:05:24.1869410Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-12-18T00:05:24.1869885Z parm: NVreg_UsePageAttributeTable:int 2024-12-18T00:05:24.1870287Z parm: NVreg_EnablePCIeGen3:int 2024-12-18T00:05:24.1870668Z parm: NVreg_EnableMSI:int 2024-12-18T00:05:24.1871028Z parm: NVreg_TCEBypassMode:int 2024-12-18T00:05:24.1871420Z parm: NVreg_EnableStreamMemOPs:int 2024-12-18T00:05:24.1871866Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-12-18T00:05:24.1872339Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-12-18T00:05:24.1872806Z parm: NVreg_EnableS0ixPowerManagement:int 2024-12-18T00:05:24.1873312Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-12-18T00:05:24.1873974Z parm: NVreg_DynamicPowerManagement:int 2024-12-18T00:05:24.1874490Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-12-18T00:05:24.1874986Z parm: NVreg_EnableGpuFirmware:int 2024-12-18T00:05:24.1875403Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-12-18T00:05:24.1875867Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-12-18T00:05:24.1876324Z parm: NVreg_EnableUserNUMAManagement:int 2024-12-18T00:05:24.1876750Z parm: NVreg_MemoryPoolSize:int 2024-12-18T00:05:24.1877139Z parm: NVreg_KMallocHeapMaxSize:int 2024-12-18T00:05:24.1877550Z parm: NVreg_VMallocHeapMaxSize:int 2024-12-18T00:05:24.1877953Z parm: NVreg_IgnoreMMIOCheck:int 2024-12-18T00:05:24.1878339Z parm: NVreg_NvLinkDisable:int 2024-12-18T00:05:24.1878767Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-12-18T00:05:24.1879207Z parm: NVreg_RegisterPCIDriver:int 2024-12-18T00:05:24.1879607Z parm: NVreg_EnableResizableBar:int 2024-12-18T00:05:24.1880014Z parm: NVreg_EnableDbgBreakpoint:int 2024-12-18T00:05:24.1880435Z parm: NVreg_EnableNonblockingOpen:int 2024-12-18T00:05:24.1880846Z parm: NVreg_RegistryDwords:charp 2024-12-18T00:05:24.1881251Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-12-18T00:05:24.1881660Z parm: NVreg_RmMsg:charp 2024-12-18T00:05:24.1882019Z parm: NVreg_GpuBlacklist:charp 2024-12-18T00:05:24.1882424Z parm: NVreg_TemporaryFilePath:charp 2024-12-18T00:05:24.1882827Z parm: NVreg_ExcludedGpus:charp 2024-12-18T00:05:24.1883281Z parm: NVreg_DmaRemapPeerMmio:int 2024-12-18T00:05:24.1883854Z parm: NVreg_RmNvlinkBandwidth:charp 2024-12-18T00:05:24.1884273Z parm: NVreg_ImexChannelCount:int 2024-12-18T00:05:24.1884669Z parm: rm_firmware_active:charp 2024-12-18T00:05:24.1885040Z + HAS_NVIDIA_DRIVER=0 2024-12-18T00:05:24.1885372Z ++ command -v nvidia-smi 2024-12-18T00:05:24.1885690Z + '[' -x /usr/bin/nvidia-smi ']' 2024-12-18T00:05:24.1886021Z + set +e 2024-12-18T00:05:24.1886410Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-12-18T00:05:29.0568547Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-12-18T00:05:29.0569005Z + NVIDIA_SMI_STATUS=0 2024-12-18T00:05:29.0569361Z + '[' 0 -ne 0 ']' 2024-12-18T00:05:29.0569643Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-12-18T00:05:29.0569971Z + HAS_NVIDIA_DRIVER=1 2024-12-18T00:05:29.0571279Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-12-18T00:05:29.0571930Z + set -e 2024-12-18T00:05:29.0572182Z + '[' 1 -eq 0 ']' 2024-12-18T00:05:29.0572673Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-12-18T00:05:29.0573260Z + post_install_nvidia_driver_common 2024-12-18T00:05:29.0573631Z + sudo modprobe nvidia 2024-12-18T00:05:29.1944511Z + echo 'After installing NVIDIA driver' 2024-12-18T00:05:29.1944967Z + lspci 2024-12-18T00:05:29.1945236Z After installing NVIDIA driver 2024-12-18T00:05:29.2070697Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-12-18T00:05:29.2071332Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-12-18T00:05:29.2072026Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-12-18T00:05:29.2072682Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-12-18T00:05:29.2073305Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-12-18T00:05:29.2073967Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-12-18T00:05:29.2074591Z 00:1b.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:29.2075155Z 00:1c.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:29.2075733Z 00:1d.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:29.2076544Z 00:1e.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:29.2077322Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-12-18T00:05:29.2077833Z + lsmod 2024-12-18T00:05:29.2094558Z Module Size Used by 2024-12-18T00:05:29.2094935Z nvidia_uvm 4706304 0 2024-12-18T00:05:29.2095419Z nvidia 54071296 1 nvidia_uvm 2024-12-18T00:05:29.2095814Z drm 602112 1 nvidia 2024-12-18T00:05:29.2096200Z drm_panel_orientation_quirks 32768 1 drm 2024-12-18T00:05:29.2096584Z backlight 24576 1 drm 2024-12-18T00:05:29.2096924Z i2c_core 106496 2 nvidia,drm 2024-12-18T00:05:29.2097289Z xt_conntrack 16384 1 2024-12-18T00:05:29.2097610Z nft_chain_nat 16384 3 2024-12-18T00:05:29.2097929Z xt_MASQUERADE 20480 1 2024-12-18T00:05:29.2098298Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-12-18T00:05:29.2098699Z nf_conntrack_netlink 57344 0 2024-12-18T00:05:29.2099205Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-12-18T00:05:29.2099747Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-12-18T00:05:29.2100141Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-12-18T00:05:29.2100504Z xfrm_user 57344 1 2024-12-18T00:05:29.2100837Z xfrm_algo 16384 1 xfrm_user 2024-12-18T00:05:29.2101182Z xt_addrtype 16384 2 2024-12-18T00:05:29.2101508Z nft_compat 20480 4 2024-12-18T00:05:29.2101885Z nf_tables 311296 57 nft_compat,nft_chain_nat 2024-12-18T00:05:29.2102395Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-12-18T00:05:29.2102857Z br_netfilter 36864 0 2024-12-18T00:05:29.2103186Z bridge 323584 1 br_netfilter 2024-12-18T00:05:29.2103551Z stp 16384 1 bridge 2024-12-18T00:05:29.2103903Z llc 16384 2 bridge,stp 2024-12-18T00:05:29.2104260Z overlay 167936 0 2024-12-18T00:05:29.2104571Z tls 114688 0 2024-12-18T00:05:29.2104869Z nls_ascii 16384 1 2024-12-18T00:05:29.2105180Z nls_cp437 20480 1 2024-12-18T00:05:29.2105488Z vfat 24576 1 2024-12-18T00:05:29.2105793Z fat 86016 1 vfat 2024-12-18T00:05:29.2106145Z sunrpc 692224 1 2024-12-18T00:05:29.2106454Z ena 167936 0 2024-12-18T00:05:29.2106901Z i8042 45056 0 2024-12-18T00:05:29.2107207Z serio 28672 3 i8042 2024-12-18T00:05:29.2107549Z button 24576 0 2024-12-18T00:05:29.2107870Z skx_edac_common 28672 0 2024-12-18T00:05:29.2108199Z ghash_clmulni_intel 16384 0 2024-12-18T00:05:29.2108534Z sch_fq_codel 20480 33 2024-12-18T00:05:29.2108846Z dm_mod 188416 0 2024-12-18T00:05:29.2109266Z fuse 163840 1 2024-12-18T00:05:29.2109576Z configfs 57344 1 2024-12-18T00:05:29.2109883Z loop 36864 0 2024-12-18T00:05:29.2110195Z dax 45056 1 dm_mod 2024-12-18T00:05:29.2110515Z dmi_sysfs 20480 0 2024-12-18T00:05:29.2110829Z crc32_pclmul 16384 0 2024-12-18T00:05:29.2111140Z crc32c_intel 24576 0 2024-12-18T00:05:29.2111449Z efivarfs 24576 1 2024-12-18T00:05:29.2111752Z + modinfo nvidia 2024-12-18T00:05:29.2113045Z filename: /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-12-18T00:05:29.2114040Z alias: char-major-195-* 2024-12-18T00:05:29.2114479Z version: 550.54.15 2024-12-18T00:05:29.2114793Z supported: external 2024-12-18T00:05:29.2115098Z license: NVIDIA 2024-12-18T00:05:29.2115431Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-12-18T00:05:29.2115850Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-12-18T00:05:29.2116234Z srcversion: 833721318DA517F0C2FEC97 2024-12-18T00:05:29.2116793Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-12-18T00:05:29.2117216Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-12-18T00:05:29.2117635Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-12-18T00:05:29.2118019Z depends: i2c-core,drm 2024-12-18T00:05:29.2118335Z retpoline: Y 2024-12-18T00:05:29.2118602Z name: nvidia 2024-12-18T00:05:29.2119092Z vermagic: 6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-12-18T00:05:29.2119692Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-12-18T00:05:29.2120241Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-12-18T00:05:29.2120745Z parm: NVreg_ResmanDebugLevel:int 2024-12-18T00:05:29.2121127Z parm: NVreg_RmLogonRC:int 2024-12-18T00:05:29.2121494Z parm: NVreg_ModifyDeviceFiles:int 2024-12-18T00:05:29.2121880Z parm: NVreg_DeviceFileUID:int 2024-12-18T00:05:29.2122249Z parm: NVreg_DeviceFileGID:int 2024-12-18T00:05:29.2122619Z parm: NVreg_DeviceFileMode:int 2024-12-18T00:05:29.2123060Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-12-18T00:05:29.2123840Z parm: NVreg_UsePageAttributeTable:int 2024-12-18T00:05:29.2124268Z parm: NVreg_EnablePCIeGen3:int 2024-12-18T00:05:29.2124659Z parm: NVreg_EnableMSI:int 2024-12-18T00:05:29.2125017Z parm: NVreg_TCEBypassMode:int 2024-12-18T00:05:29.2125426Z parm: NVreg_EnableStreamMemOPs:int 2024-12-18T00:05:29.2125895Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-12-18T00:05:29.2126395Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-12-18T00:05:29.2126873Z parm: NVreg_EnableS0ixPowerManagement:int 2024-12-18T00:05:29.2127374Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-12-18T00:05:29.2127886Z parm: NVreg_DynamicPowerManagement:int 2024-12-18T00:05:29.2128409Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-12-18T00:05:29.2129049Z parm: NVreg_EnableGpuFirmware:int 2024-12-18T00:05:29.2129463Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-12-18T00:05:29.2130018Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-12-18T00:05:29.2130622Z parm: NVreg_EnableUserNUMAManagement:int 2024-12-18T00:05:29.2131042Z parm: NVreg_MemoryPoolSize:int 2024-12-18T00:05:29.2131439Z parm: NVreg_KMallocHeapMaxSize:int 2024-12-18T00:05:29.2131938Z parm: NVreg_VMallocHeapMaxSize:int 2024-12-18T00:05:29.2132341Z parm: NVreg_IgnoreMMIOCheck:int 2024-12-18T00:05:29.2132714Z parm: NVreg_NvLinkDisable:int 2024-12-18T00:05:29.2133140Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-12-18T00:05:29.2133583Z parm: NVreg_RegisterPCIDriver:int 2024-12-18T00:05:29.2133988Z parm: NVreg_EnableResizableBar:int 2024-12-18T00:05:29.2134401Z parm: NVreg_EnableDbgBreakpoint:int 2024-12-18T00:05:29.2134816Z parm: NVreg_EnableNonblockingOpen:int 2024-12-18T00:05:29.2135231Z parm: NVreg_RegistryDwords:charp 2024-12-18T00:05:29.2135650Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-12-18T00:05:29.2136058Z parm: NVreg_RmMsg:charp 2024-12-18T00:05:29.2136417Z parm: NVreg_GpuBlacklist:charp 2024-12-18T00:05:29.2136805Z parm: NVreg_TemporaryFilePath:charp 2024-12-18T00:05:29.2137210Z parm: NVreg_ExcludedGpus:charp 2024-12-18T00:05:29.2137604Z parm: NVreg_DmaRemapPeerMmio:int 2024-12-18T00:05:29.2138009Z parm: NVreg_RmNvlinkBandwidth:charp 2024-12-18T00:05:29.2138414Z parm: NVreg_ImexChannelCount:int 2024-12-18T00:05:29.2138793Z parm: rm_firmware_active:charp 2024-12-18T00:05:29.2139151Z + set +e 2024-12-18T00:05:29.2139392Z + nvidia-smi 2024-12-18T00:05:31.7368431Z Wed Dec 18 00:05:31 2024 2024-12-18T00:05:31.7368924Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:05:31.7369839Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-12-18T00:05:31.7370462Z |-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:05:31.7371108Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-12-18T00:05:31.7371801Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-12-18T00:05:31.7372373Z | | | MIG M. | 2024-12-18T00:05:31.7372805Z |=========================================+========================+======================| 2024-12-18T00:05:31.7462625Z | 0 Tesla T4 Off | 00000000:00:1B.0 Off | 0 | 2024-12-18T00:05:31.7463223Z | N/A 29C P0 25W / 70W | 0MiB / 15360MiB | 0% Default | 2024-12-18T00:05:31.7463720Z | | | N/A | 2024-12-18T00:05:31.7464223Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:05:31.7565533Z | 1 Tesla T4 Off | 00000000:00:1C.0 Off | 0 | 2024-12-18T00:05:31.7566129Z | N/A 29C P0 25W / 70W | 0MiB / 15360MiB | 6% Default | 2024-12-18T00:05:31.7566652Z | | | N/A | 2024-12-18T00:05:31.7567171Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:05:31.7654636Z | 2 Tesla T4 Off | 00000000:00:1D.0 Off | 0 | 2024-12-18T00:05:31.7655224Z | N/A 28C P0 25W / 70W | 0MiB / 15360MiB | 0% Default | 2024-12-18T00:05:31.7655839Z | | | N/A | 2024-12-18T00:05:31.7656357Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:05:31.7765088Z | 3 Tesla T4 Off | 00000000:00:1E.0 Off | 0 | 2024-12-18T00:05:31.7765865Z | N/A 29C P0 24W / 70W | 0MiB / 15360MiB | 0% Default | 2024-12-18T00:05:31.7766372Z | | | N/A | 2024-12-18T00:05:31.7767106Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:05:31.7770003Z 2024-12-18T00:05:31.7770525Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:05:31.7771091Z | Processes: | 2024-12-18T00:05:31.7771685Z | GPU GI CI PID Type Process name GPU Memory | 2024-12-18T00:05:31.7772237Z | ID ID Usage | 2024-12-18T00:05:31.7772683Z |=========================================================================================| 2024-12-18T00:05:31.7789608Z | No running processes found | 2024-12-18T00:05:31.7790255Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:05:34.1593940Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-12-18T00:05:36.6771942Z Tesla T4 2024-12-18T00:05:38.7326709Z + NVIDIA_SMI_STATUS=0 2024-12-18T00:05:38.7327089Z + '[' 0 -eq 0 ']' 2024-12-18T00:05:38.7327408Z + echo 'INFO: Ignoring allowed status 0' 2024-12-18T00:05:38.7327783Z + set -e 2024-12-18T00:05:38.7328038Z INFO: Ignoring allowed status 0 2024-12-18T00:05:38.7333110Z == Installing nvidia container toolkit for amzn2023 == 2024-12-18T00:05:38.7336855Z + sudo yum install -y yum-utils 2024-12-18T00:05:39.1840538Z Last metadata expiration check: 0:17:00 ago on Tue Dec 17 23:48:39 2024. 2024-12-18T00:05:39.2100033Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-12-18T00:05:39.2491305Z Dependencies resolved. 2024-12-18T00:05:39.2670760Z Nothing to do. 2024-12-18T00:05:39.2671077Z Complete! 2024-12-18T00:05:39.4212363Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-12-18T00:05:39.4213180Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-12-18T00:05:39.4214275Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-12-18T00:05:39.7285956Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-12-18T00:05:39.7968017Z + sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 2024-12-18T00:05:40.3328472Z nvidia-container-toolkit 10 kB/s | 833 B 00:00 2024-12-18T00:05:40.3592691Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-12-18T00:05:40.3996054Z Dependencies resolved. 2024-12-18T00:05:40.4179613Z ================================================================================ 2024-12-18T00:05:40.4180548Z Package Arch Version Repository Size 2024-12-18T00:05:40.4181239Z ================================================================================ 2024-12-18T00:05:40.4181630Z Downgrading: 2024-12-18T00:05:40.4182091Z nvidia-container-toolkit x86_64 1.16.2-1 nvidia-container-toolkit 1.2 M 2024-12-18T00:05:40.4182844Z nvidia-container-toolkit-base x86_64 1.16.2-1 nvidia-container-toolkit 5.6 M 2024-12-18T00:05:40.4183285Z 2024-12-18T00:05:40.4183394Z Transaction Summary 2024-12-18T00:05:40.4183710Z ================================================================================ 2024-12-18T00:05:40.4184116Z Downgrade 2 Packages 2024-12-18T00:05:40.4184308Z 2024-12-18T00:05:40.4184426Z Total download size: 6.8 M 2024-12-18T00:05:40.4184747Z Downloading Packages: 2024-12-18T00:05:40.5305621Z (1/2): nvidia-container-toolkit-1.16.2-1.x86_64 11 MB/s | 1.2 MB 00:00 2024-12-18T00:05:40.6921248Z (2/2): nvidia-container-toolkit-base-1.16.2-1.x 21 MB/s | 5.6 MB 00:00 2024-12-18T00:05:40.6932021Z -------------------------------------------------------------------------------- 2024-12-18T00:05:40.6932850Z Total 25 MB/s | 6.8 MB 00:00 2024-12-18T00:05:40.6933575Z Running transaction check 2024-12-18T00:05:40.7045123Z Transaction check succeeded. 2024-12-18T00:05:40.7045535Z Running transaction test 2024-12-18T00:05:40.7358045Z Transaction test succeeded. 2024-12-18T00:05:40.7358577Z Running transaction 2024-12-18T00:05:41.6530815Z Preparing : 1/1 2024-12-18T00:05:41.8701476Z Downgrading : nvidia-container-toolkit-base-1.16.2-1.x86_64 1/4 2024-12-18T00:05:41.9229913Z Downgrading : nvidia-container-toolkit-1.16.2-1.x86_64 2/4 2024-12-18T00:05:42.0067318Z Running scriptlet: nvidia-container-toolkit-1.16.2-1.x86_64 2/4 2024-12-18T00:05:42.0068046Z Cleanup : nvidia-container-toolkit-1.17.3-1.x86_64 3/4 2024-12-18T00:05:42.0645020Z Running scriptlet: nvidia-container-toolkit-1.17.3-1.x86_64 3/4 2024-12-18T00:05:42.1215453Z Cleanup : nvidia-container-toolkit-base-1.17.3-1.x86_64 4/4 2024-12-18T00:07:12.1270697Z Running scriptlet: nvidia-container-toolkit-1.16.2-1.x86_64 4/4 2024-12-18T00:07:12.1272719Z Verifying : nvidia-container-toolkit-1.16.2-1.x86_64 1/4 2024-12-18T00:07:12.1273400Z Verifying : nvidia-container-toolkit-1.17.3-1.x86_64 2/4 2024-12-18T00:07:12.1274066Z Verifying : nvidia-container-toolkit-base-1.16.2-1.x86_64 3/4 2024-12-18T00:07:12.2644793Z Verifying : nvidia-container-toolkit-base-1.17.3-1.x86_64 4/4================================================================================ 2024-12-18T00:07:12.2645571Z WARNING: 2024-12-18T00:07:12.2645884Z A newer release of "Amazon Linux" is available. 2024-12-18T00:07:12.2646181Z 2024-12-18T00:07:12.2646311Z Available Versions: 2024-12-18T00:07:12.2646500Z 2024-12-18T00:07:12.2646629Z Version 2023.6.20241010: 2024-12-18T00:07:12.2647053Z Run the following command to upgrade to 2023.6.20241010: 2024-12-18T00:07:12.2647392Z 2024-12-18T00:07:12.2647545Z dnf upgrade --releasever=2023.6.20241010 2024-12-18T00:07:12.2647832Z 2024-12-18T00:07:12.2647938Z Release notes: 2024-12-18T00:07:12.2648463Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241010.html 2024-12-18T00:07:12.2648939Z 2024-12-18T00:07:12.2649062Z Version 2023.6.20241028: 2024-12-18T00:07:12.2649468Z Run the following command to upgrade to 2023.6.20241028: 2024-12-18T00:07:12.2649789Z 2024-12-18T00:07:12.2649932Z dnf upgrade --releasever=2023.6.20241028 2024-12-18T00:07:12.2650211Z 2024-12-18T00:07:12.2650317Z Release notes: 2024-12-18T00:07:12.2650820Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241028.html 2024-12-18T00:07:12.2651302Z 2024-12-18T00:07:12.2651413Z Version 2023.6.20241031: 2024-12-18T00:07:12.2651804Z Run the following command to upgrade to 2023.6.20241031: 2024-12-18T00:07:12.2652135Z 2024-12-18T00:07:12.2652290Z dnf upgrade --releasever=2023.6.20241031 2024-12-18T00:07:12.2652557Z 2024-12-18T00:07:12.2652663Z Release notes: 2024-12-18T00:07:12.2653167Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241031.html 2024-12-18T00:07:12.2653645Z 2024-12-18T00:07:12.2653755Z Version 2023.6.20241111: 2024-12-18T00:07:12.2654148Z Run the following command to upgrade to 2023.6.20241111: 2024-12-18T00:07:12.2654473Z 2024-12-18T00:07:12.2654628Z dnf upgrade --releasever=2023.6.20241111 2024-12-18T00:07:12.2654893Z 2024-12-18T00:07:12.2655012Z Release notes: 2024-12-18T00:07:12.2655501Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241111.html 2024-12-18T00:07:12.2656112Z 2024-12-18T00:07:12.2656219Z Version 2023.6.20241121: 2024-12-18T00:07:12.2656597Z Run the following command to upgrade to 2023.6.20241121: 2024-12-18T00:07:12.2656918Z 2024-12-18T00:07:12.2657328Z dnf upgrade --releasever=2023.6.20241121 2024-12-18T00:07:12.2657602Z 2024-12-18T00:07:12.2657721Z Release notes: 2024-12-18T00:07:12.2658211Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241121.html 2024-12-18T00:07:12.2658662Z 2024-12-18T00:07:12.2658768Z Version 2023.6.20241212: 2024-12-18T00:07:12.2659153Z Run the following command to upgrade to 2023.6.20241212: 2024-12-18T00:07:12.2659476Z 2024-12-18T00:07:12.2659622Z dnf upgrade --releasever=2023.6.20241212 2024-12-18T00:07:12.2659894Z 2024-12-18T00:07:12.2659997Z Release notes: 2024-12-18T00:07:12.2660592Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241212.html 2024-12-18T00:07:12.2661033Z 2024-12-18T00:07:12.2661178Z ================================================================================ 2024-12-18T00:07:12.2993822Z 2024-12-18T00:07:12.2994216Z 2024-12-18T00:07:12.2994385Z Downgraded: 2024-12-18T00:07:12.2994863Z nvidia-container-toolkit-1.16.2-1.x86_64 2024-12-18T00:07:12.2995589Z nvidia-container-toolkit-base-1.16.2-1.x86_64 2024-12-18T00:07:12.2996011Z 2024-12-18T00:07:12.2996130Z Complete! 2024-12-18T00:07:12.3447242Z + sudo systemctl restart docker 2024-12-18T00:07:21.0951909Z Wed Dec 18 00:07:21 2024 2024-12-18T00:07:21.0952468Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:07:21.0953446Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-12-18T00:07:21.0954063Z |-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:07:21.0954696Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-12-18T00:07:21.0955365Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-12-18T00:07:21.0955914Z | | | MIG M. | 2024-12-18T00:07:21.0956339Z |=========================================+========================+======================| 2024-12-18T00:07:21.1061288Z | 0 Tesla T4 On | 00000000:00:1B.0 Off | 0 | 2024-12-18T00:07:21.1062394Z | N/A 25C P0 24W / 70W | 0MiB / 15360MiB | 0% Default | 2024-12-18T00:07:21.1062939Z | | | N/A | 2024-12-18T00:07:21.1063477Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:07:21.1166137Z | 1 Tesla T4 On | 00000000:00:1C.0 Off | 0 | 2024-12-18T00:07:21.1166735Z | N/A 25C P0 25W / 70W | 0MiB / 15360MiB | 0% Default | 2024-12-18T00:07:21.1167242Z | | | N/A | 2024-12-18T00:07:21.1167767Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:07:21.1261493Z | 2 Tesla T4 On | 00000000:00:1D.0 Off | 0 | 2024-12-18T00:07:21.1262126Z | N/A 24C P0 25W / 70W | 0MiB / 15360MiB | 0% Default | 2024-12-18T00:07:21.1262621Z | | | N/A | 2024-12-18T00:07:21.1263146Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:07:21.1352979Z | 3 Tesla T4 On | 00000000:00:1E.0 Off | 0 | 2024-12-18T00:07:21.1353559Z | N/A 25C P0 24W / 70W | 0MiB / 15360MiB | 6% Default | 2024-12-18T00:07:21.1354119Z | | | N/A | 2024-12-18T00:07:21.1354878Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:07:21.1356874Z 2024-12-18T00:07:21.1357373Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:07:21.1357917Z | Processes: | 2024-12-18T00:07:21.1358484Z | GPU GI CI PID Type Process name GPU Memory | 2024-12-18T00:07:21.1359025Z | ID ID Usage | 2024-12-18T00:07:21.1359442Z |=========================================================================================| 2024-12-18T00:07:21.1375939Z | No running processes found | 2024-12-18T00:07:21.1376546Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:07:21.5348223Z Command completed after 1 attempt(s). 2024-12-18T00:07:21.5440560Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-12-18T00:07:21.5441183Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-12-18T00:07:21.5441731Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-12-18T00:07:21.5442232Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:07:21.5452694Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:21.5453148Z env: 2024-12-18T00:07:21.5453413Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:21.5453813Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:21.5454231Z ##[endgroup] 2024-12-18T00:07:21.8624610Z Defaulting to user installation because normal site-packages is not writeable 2024-12-18T00:07:22.2729892Z Collecting psutil==5.9.1 2024-12-18T00:07:22.3125827Z Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-12-18T00:07:22.3657507Z Collecting nvidia-ml-py==11.525.84 2024-12-18T00:07:22.3695319Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-12-18T00:07:22.4676257Z Installing collected packages: psutil, nvidia-ml-py 2024-12-18T00:07:22.6343300Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-12-18T00:07:22.8294603Z Prepare all required actions 2024-12-18T00:07:22.8295783Z Getting action download info 2024-12-18T00:07:22.9610822Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-12-18T00:07:23.1726586Z Download action repository 'actions/download-artifact@v4' (SHA:fa0a91b85d4f404e444e00e005971372dc801d16) 2024-12-18T00:07:23.4344638Z ##[group]Run ./.github/actions/download-build-artifacts 2024-12-18T00:07:23.4345061Z with: 2024-12-18T00:07:23.4345343Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-12-18T00:07:23.4345730Z s3-bucket: gha-artifacts 2024-12-18T00:07:23.4346035Z env: 2024-12-18T00:07:23.4346276Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:23.4346667Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:23.4347086Z ##[endgroup] 2024-12-18T00:07:23.4382736Z ##[group]Run seemethere/download-artifact-s3@v4 2024-12-18T00:07:23.4383138Z with: 2024-12-18T00:07:23.4383483Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-12-18T00:07:23.4383866Z s3-bucket: gha-artifacts 2024-12-18T00:07:23.4384181Z region: us-east-1 2024-12-18T00:07:23.4384452Z env: 2024-12-18T00:07:23.4384702Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:23.4385092Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:23.4385496Z ##[endgroup] 2024-12-18T00:07:23.9642309Z (node:57243) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-12-18T00:07:23.9643179Z 2024-12-18T00:07:23.9643583Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-12-18T00:07:23.9644238Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-12-18T00:07:23.9644899Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-12-18T00:07:24.2612148Z Found 1 objects with prefix pytorch/pytorch/12383255652/linux-focal-cuda11.8-py3.10-gcc9/ 2024-12-18T00:07:24.2613064Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-12-18T00:07:38.4365813Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-12-18T00:07:38.4372022Z Artifact download has finished successfully 2024-12-18T00:07:38.4596977Z ##[group]Run unzip -o artifacts.zip 2024-12-18T00:07:38.4597345Z unzip -o artifacts.zip 2024-12-18T00:07:38.4604062Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:38.4604500Z env: 2024-12-18T00:07:38.4604755Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:38.4605153Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:38.4605580Z ##[endgroup] 2024-12-18T00:07:38.4684392Z Archive: artifacts.zip 2024-12-18T00:07:38.4684754Z creating: dist/ 2024-12-18T00:07:40.5039964Z inflating: dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl 2024-12-18T00:07:40.5040755Z creating: build/custom_test_artifacts/ 2024-12-18T00:07:40.5041224Z creating: build/custom_test_artifacts/custom-op-build/ 2024-12-18T00:07:40.5041830Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-12-18T00:07:40.5042869Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-12-18T00:07:40.5047502Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T00:07:40.5048317Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-12-18T00:07:40.5049110Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T00:07:40.5049963Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T00:07:40.5050778Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T00:07:40.5051734Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T00:07:40.5053188Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T00:07:40.5054061Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T00:07:40.5054920Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T00:07:40.5056331Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T00:07:40.5057752Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T00:07:40.5059403Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T00:07:40.5060361Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T00:07:40.5061481Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T00:07:40.5062525Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T00:07:40.5063399Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-12-18T00:07:40.5064242Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-12-18T00:07:40.5112097Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-12-18T00:07:40.5113499Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-12-18T00:07:40.5114726Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-12-18T00:07:40.5115966Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-12-18T00:07:40.5117222Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-12-18T00:07:40.5118408Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-12-18T00:07:40.5119593Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-12-18T00:07:40.5120784Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-12-18T00:07:40.5121959Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-12-18T00:07:40.5158218Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-12-18T00:07:40.5199414Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-12-18T00:07:40.5200660Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-12-18T00:07:40.5202080Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-12-18T00:07:40.5203255Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-12-18T00:07:40.5204505Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-12-18T00:07:40.5205585Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-12-18T00:07:40.5206627Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-12-18T00:07:40.5207690Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-12-18T00:07:40.5267590Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-12-18T00:07:40.5330933Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-12-18T00:07:40.5332022Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-12-18T00:07:40.5332891Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-12-18T00:07:40.5333639Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-12-18T00:07:40.5334418Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-12-18T00:07:40.5335321Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-12-18T00:07:40.5336180Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-12-18T00:07:40.5337145Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-12-18T00:07:40.5338091Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-12-18T00:07:40.5338978Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-12-18T00:07:40.5339882Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-12-18T00:07:40.5340794Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-12-18T00:07:40.5341702Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-12-18T00:07:40.5342608Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-12-18T00:07:40.5343493Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-12-18T00:07:40.5359721Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-12-18T00:07:40.5506674Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-12-18T00:07:40.5507705Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-12-18T00:07:40.5508633Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-12-18T00:07:40.5509676Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-12-18T00:07:40.5510675Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-12-18T00:07:40.5511612Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-12-18T00:07:40.5512575Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-12-18T00:07:40.5513536Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-12-18T00:07:40.5514503Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-12-18T00:07:40.5515656Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-12-18T00:07:40.5516608Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-12-18T00:07:40.5532311Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-12-18T00:07:40.5619652Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-12-18T00:07:40.5620858Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T00:07:40.5621766Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-12-18T00:07:40.5622789Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-12-18T00:07:40.5623563Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-12-18T00:07:40.5624326Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-12-18T00:07:40.5625080Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-12-18T00:07:40.5625788Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-12-18T00:07:40.5626426Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-12-18T00:07:40.5627088Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-12-18T00:07:40.5751801Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-12-18T00:07:40.5817364Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-12-18T00:07:40.5817997Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-12-18T00:07:40.5818571Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-12-18T00:07:40.5819240Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-12-18T00:07:40.5823960Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T00:07:40.5824732Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-12-18T00:07:40.5825487Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T00:07:40.5826305Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T00:07:40.5827092Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T00:07:40.5828009Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T00:07:40.5829047Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T00:07:40.5829872Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T00:07:40.5830675Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T00:07:40.5832496Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T00:07:40.5833947Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T00:07:40.5835659Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T00:07:40.5836607Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T00:07:40.5837802Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T00:07:40.5838783Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T00:07:40.5839645Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-12-18T00:07:40.5840629Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-12-18T00:07:40.5888564Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-12-18T00:07:40.5889822Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-12-18T00:07:40.5891078Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-12-18T00:07:40.5892357Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-12-18T00:07:40.5893629Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-12-18T00:07:40.5895050Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-12-18T00:07:40.5896362Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-12-18T00:07:40.5897529Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-12-18T00:07:40.5898695Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-12-18T00:07:40.5934192Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-12-18T00:07:40.5976143Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-12-18T00:07:40.5977362Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-12-18T00:07:40.5978465Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-12-18T00:07:40.5979523Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-12-18T00:07:40.5980552Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-12-18T00:07:40.5981578Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-12-18T00:07:40.5982581Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-12-18T00:07:40.5983599Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-12-18T00:07:40.6044702Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-12-18T00:07:40.6109475Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-12-18T00:07:40.6110506Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-12-18T00:07:40.6111340Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-12-18T00:07:40.6112046Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-12-18T00:07:40.6112769Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-12-18T00:07:40.6113552Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-12-18T00:07:40.6114445Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-12-18T00:07:40.6115460Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-12-18T00:07:40.6116435Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-12-18T00:07:40.6117529Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-12-18T00:07:40.6118462Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-12-18T00:07:40.6119390Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-12-18T00:07:40.6120334Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-12-18T00:07:40.6121272Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-12-18T00:07:40.6122192Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-12-18T00:07:40.6137871Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-12-18T00:07:40.6203342Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-12-18T00:07:40.6204596Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T00:07:40.6205513Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-12-18T00:07:40.6206352Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-12-18T00:07:40.6207122Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-12-18T00:07:40.6207890Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-12-18T00:07:40.6208655Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-12-18T00:07:40.6209371Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-12-18T00:07:40.6210018Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-12-18T00:07:40.6210696Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-12-18T00:07:40.6261688Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-12-18T00:07:40.6262358Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-12-18T00:07:40.6262982Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-12-18T00:07:40.6263718Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-12-18T00:07:40.6269099Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T00:07:40.6269926Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-12-18T00:07:40.6270755Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T00:07:40.6271644Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T00:07:40.6272525Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T00:07:40.6273513Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T00:07:40.6274520Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T00:07:40.6275412Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T00:07:40.6276287Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T00:07:40.6277314Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T00:07:40.6278436Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T00:07:40.6280036Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T00:07:40.6281068Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T00:07:40.6282773Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T00:07:40.6284103Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T00:07:40.6285065Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-12-18T00:07:40.6285971Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-12-18T00:07:40.6332841Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-12-18T00:07:40.6334167Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-12-18T00:07:40.6335798Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-12-18T00:07:40.6337130Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-12-18T00:07:40.6338436Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-12-18T00:07:40.6339682Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-12-18T00:07:40.6340935Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-12-18T00:07:40.6342196Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-12-18T00:07:40.6343436Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-12-18T00:07:40.6380776Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-12-18T00:07:40.6422990Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-12-18T00:07:40.6424269Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-12-18T00:07:40.6425435Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-12-18T00:07:40.6426560Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-12-18T00:07:40.6427661Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-12-18T00:07:40.6428791Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-12-18T00:07:40.6429862Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-12-18T00:07:40.6430954Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-12-18T00:07:40.6492351Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-12-18T00:07:40.6555789Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-12-18T00:07:40.6556909Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-12-18T00:07:40.6557818Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-12-18T00:07:40.6558597Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-12-18T00:07:40.6559411Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-12-18T00:07:40.6560434Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-12-18T00:07:40.6561407Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-12-18T00:07:40.6562510Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-12-18T00:07:40.6563863Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-12-18T00:07:40.6564875Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-12-18T00:07:40.6566079Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-12-18T00:07:40.6567246Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-12-18T00:07:40.6568287Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-12-18T00:07:40.6569337Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-12-18T00:07:40.6570371Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-12-18T00:07:40.6571483Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-12-18T00:07:40.6689388Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-12-18T00:07:40.6690456Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-12-18T00:07:40.6691513Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-12-18T00:07:40.6692700Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-12-18T00:07:40.6693843Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-12-18T00:07:40.6694908Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-12-18T00:07:40.6696096Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-12-18T00:07:40.6697183Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-12-18T00:07:40.6698254Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-12-18T00:07:40.6699326Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-12-18T00:07:40.6700371Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-12-18T00:07:40.6716810Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-12-18T00:07:40.6772632Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-12-18T00:07:40.6773821Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T00:07:40.6774831Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-12-18T00:07:40.6775835Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-12-18T00:07:40.6776652Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-12-18T00:07:40.6777453Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-12-18T00:07:40.6778262Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-12-18T00:07:40.6779259Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-12-18T00:07:40.6780103Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-12-18T00:07:40.6780819Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-12-18T00:07:40.6884894Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-12-18T00:07:40.6928155Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-12-18T00:07:40.6928773Z creating: build/lib/ 2024-12-18T00:07:40.6938634Z inflating: build/lib/libpthreadpool.a 2024-12-18T00:07:40.6947043Z inflating: build/lib/libcpuinfo.a 2024-12-18T00:07:40.6955006Z inflating: build/lib/libcpuinfo_internals.a 2024-12-18T00:07:40.6955843Z inflating: build/lib/libclog.a 2024-12-18T00:07:40.7046920Z inflating: build/lib/libprotobuf-lite.a 2024-12-18T00:07:40.7048772Z inflating: build/lib/libnnpack_reference_layers.a 2024-12-18T00:07:40.7518020Z inflating: build/lib/libprotobuf.a 2024-12-18T00:07:40.7588812Z inflating: build/lib/libgtest.a 2024-12-18T00:07:40.7666640Z inflating: build/lib/libbenchmark.a 2024-12-18T00:07:40.7673563Z inflating: build/lib/libittnotify.a 2024-12-18T00:07:40.7704891Z inflating: build/lib/libtensorpipe_uv.a 2024-12-18T00:07:40.7770183Z inflating: build/lib/libasmjit.a 2024-12-18T00:07:40.7790515Z inflating: build/lib/libnnpack.a 2024-12-18T00:07:40.7922487Z inflating: build/lib/libgloo.a 2024-12-18T00:07:40.7945038Z inflating: build/lib/libfmt.a 2024-12-18T00:07:40.7945714Z inflating: build/lib/libcaffe2_nvrtc.so 2024-12-18T00:07:40.8049077Z inflating: build/lib/libc10.so 2024-12-18T00:07:40.8242104Z inflating: build/lib/libmicrokernels-prod.a 2024-12-18T00:07:40.8242856Z inflating: build/lib/libtorch_global_deps.so 2024-12-18T00:07:40.8760095Z inflating: build/lib/libprotoc.a 2024-12-18T00:07:40.8781494Z inflating: build/lib/libpytorch_qnnpack.a 2024-12-18T00:07:40.8800011Z inflating: build/lib/libgmock.a 2024-12-18T00:07:40.8800821Z inflating: build/lib/libgtest_main.a 2024-12-18T00:07:40.8801293Z inflating: build/lib/libbenchmark_main.a 2024-12-18T00:07:41.9118091Z inflating: build/lib/libdnnl.a 2024-12-18T00:07:41.9699098Z inflating: build/lib/libtensorpipe.a 2024-12-18T00:07:41.9760366Z inflating: build/lib/libc10_cuda.so 2024-12-18T00:07:42.1018822Z inflating: build/lib/libfbgemm.a 2024-12-18T00:07:42.1019409Z inflating: build/lib/libgmock_main.a 2024-12-18T00:07:42.1544001Z inflating: build/lib/libkineto.a 2024-12-18T00:07:42.1802700Z inflating: build/lib/libtensorpipe_cuda.a 2024-12-18T00:07:42.1894310Z inflating: build/lib/libXNNPACK.a 2024-12-18T00:07:42.2280033Z inflating: build/lib/libgloo_cuda.a 2024-12-18T00:07:42.2323078Z inflating: build/lib/libonnx_proto.a 2024-12-18T00:07:42.3066385Z inflating: build/lib/libonnx.a 2024-12-18T00:07:42.3952034Z inflating: build/lib/libmicrokernels-all.a 2024-12-18T00:07:44.9606501Z inflating: build/lib/libtorch_cpu.so 2024-12-18T00:07:44.9611374Z inflating: build/lib/libunbox_lib.a 2024-12-18T00:07:44.9615902Z inflating: build/lib/libshm.so 2024-12-18T00:07:46.6732559Z inflating: build/lib/libtorch_cuda.so 2024-12-18T00:07:46.6733412Z inflating: build/lib/libtorch.so 2024-12-18T00:07:46.6736007Z inflating: build/lib/libc10d_cuda_test.so 2024-12-18T00:07:47.6283943Z inflating: build/lib/libtorch_cuda_linalg.so 2024-12-18T00:07:47.6304824Z inflating: build/lib/libjitbackend_test.so 2024-12-18T00:07:47.6375886Z inflating: build/lib/libtorchbind_test.so 2024-12-18T00:07:47.6402671Z inflating: build/lib/libbackend_with_compiler.so 2024-12-18T00:07:47.6428907Z inflating: build/lib/libaoti_custom_ops.so 2024-12-18T00:07:47.8561173Z inflating: build/lib/libtorch_python.so 2024-12-18T00:07:47.8597129Z inflating: build/lib/libnnapi_backend.so 2024-12-18T00:07:47.8598290Z creating: build/bin/ 2024-12-18T00:07:47.8652430Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-12-18T00:07:47.8711696Z inflating: build/bin/c10_DeviceGuard_test 2024-12-18T00:07:47.8768114Z inflating: build/bin/c10_Device_test 2024-12-18T00:07:47.8833715Z inflating: build/bin/c10_DispatchKeySet_test 2024-12-18T00:07:47.8893874Z inflating: build/bin/c10_Scalar_test 2024-12-18T00:07:47.8949036Z inflating: build/bin/c10_StreamGuard_test 2024-12-18T00:07:47.9005467Z inflating: build/bin/c10_SymInt_test 2024-12-18T00:07:47.9066501Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-12-18T00:07:47.9127297Z inflating: build/bin/c10_InlineStreamGuard_test 2024-12-18T00:07:47.9191124Z inflating: build/bin/c10_SizesAndStrides_test 2024-12-18T00:07:47.9244843Z inflating: build/bin/c10_ArrayRef_test 2024-12-18T00:07:47.9324192Z inflating: build/bin/c10_cow_test 2024-12-18T00:07:47.9377064Z inflating: build/bin/c10_ConstexprCrc_test 2024-12-18T00:07:47.9437299Z inflating: build/bin/c10_Bitset_test 2024-12-18T00:07:47.9493319Z inflating: build/bin/c10_DeadlockDetection_test 2024-12-18T00:07:47.9550449Z inflating: build/bin/c10_Half_test 2024-12-18T00:07:47.9612972Z inflating: build/bin/c10_LeftRight_test 2024-12-18T00:07:47.9674143Z inflating: build/bin/c10_Metaprogramming_test 2024-12-18T00:07:47.9734466Z inflating: build/bin/c10_NetworkFlow_test 2024-12-18T00:07:47.9791628Z inflating: build/bin/c10_Synchronized_test 2024-12-18T00:07:47.9851926Z inflating: build/bin/c10_ThreadLocal_test 2024-12-18T00:07:47.9911870Z inflating: build/bin/c10_TypeIndex_test 2024-12-18T00:07:47.9967406Z inflating: build/bin/c10_TypeList_test 2024-12-18T00:07:48.0023188Z inflating: build/bin/c10_TypeTraits_test 2024-12-18T00:07:48.0082335Z inflating: build/bin/c10_accumulate_test 2024-12-18T00:07:48.0145390Z inflating: build/bin/c10_bfloat16_test 2024-12-18T00:07:48.0201900Z inflating: build/bin/c10_bit_cast_test 2024-12-18T00:07:48.0264491Z inflating: build/bin/c10_complex_math_test 2024-12-18T00:07:48.0324534Z inflating: build/bin/c10_complex_test 2024-12-18T00:07:48.0380768Z inflating: build/bin/c10_error_test 2024-12-18T00:07:48.0438593Z inflating: build/bin/c10_exception_test 2024-12-18T00:07:48.0497214Z inflating: build/bin/c10_flags_test 2024-12-18T00:07:48.0553113Z inflating: build/bin/c10_generic_math_test 2024-12-18T00:07:48.0729297Z inflating: build/bin/c10_intrusive_ptr_test 2024-12-18T00:07:48.0787442Z inflating: build/bin/c10_irange_test 2024-12-18T00:07:48.0844654Z inflating: build/bin/c10_lazy_test 2024-12-18T00:07:48.0910211Z inflating: build/bin/c10_logging_test 2024-12-18T00:07:48.0992653Z inflating: build/bin/c10_optional_test 2024-12-18T00:07:48.1062033Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-12-18T00:07:48.1121776Z inflating: build/bin/c10_registry_test 2024-12-18T00:07:48.1284741Z inflating: build/bin/c10_small_vector_test 2024-12-18T00:07:48.1342746Z inflating: build/bin/c10_ssize_test 2024-12-18T00:07:48.1400436Z inflating: build/bin/c10_string_util_test 2024-12-18T00:07:48.1457119Z inflating: build/bin/c10_tempfile_test 2024-12-18T00:07:48.1522854Z inflating: build/bin/c10_string_view_test 2024-12-18T00:07:48.1585523Z inflating: build/bin/c10_typeid_test 2024-12-18T00:07:48.1636918Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-12-18T00:07:48.2103151Z inflating: build/bin/protoc-3.13.0.0 2024-12-18T00:07:48.2565519Z inflating: build/bin/protoc 2024-12-18T00:07:48.2625874Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-12-18T00:07:48.2684596Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-12-18T00:07:48.2743736Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-12-18T00:07:48.2802761Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-12-18T00:07:48.2857886Z inflating: build/bin/c10_cuda_CUDATest 2024-12-18T00:07:48.2916471Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-12-18T00:07:48.2974113Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-12-18T00:07:48.3032888Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-12-18T00:07:48.3440140Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-12-18T00:07:48.3860700Z inflating: build/bin/vec_test_all_types_AVX512 2024-12-18T00:07:48.4294880Z inflating: build/bin/vec_test_all_types_AVX2 2024-12-18T00:07:48.4354588Z inflating: build/bin/HashStoreTest 2024-12-18T00:07:48.4412276Z inflating: build/bin/FileStoreTest 2024-12-18T00:07:48.4473629Z inflating: build/bin/TCPStoreTest 2024-12-18T00:07:48.4532298Z inflating: build/bin/BackoffTest 2024-12-18T00:07:48.4548221Z inflating: build/bin/ProcessGroupMPITest 2024-12-18T00:07:48.4551358Z inflating: build/bin/torch_shm_manager 2024-12-18T00:07:48.4611706Z inflating: build/bin/test_edge_op_registration 2024-12-18T00:07:48.4614323Z inflating: build/bin/example_allreduce 2024-12-18T00:07:48.4672633Z inflating: build/bin/Dimname_test 2024-12-18T00:07:48.4754528Z inflating: build/bin/Dict_test 2024-12-18T00:07:48.4826357Z inflating: build/bin/MaybeOwned_test 2024-12-18T00:07:48.4891239Z inflating: build/bin/NamedTensor_test 2024-12-18T00:07:48.4956595Z inflating: build/bin/apply_utils_test 2024-12-18T00:07:48.5022660Z inflating: build/bin/atest 2024-12-18T00:07:48.5092357Z inflating: build/bin/basic 2024-12-18T00:07:48.5155628Z inflating: build/bin/broadcast_test 2024-12-18T00:07:48.5212257Z inflating: build/bin/cpu_allocator_test 2024-12-18T00:07:48.5280031Z inflating: build/bin/cpu_generator_test 2024-12-18T00:07:48.5341098Z inflating: build/bin/cpu_profiling_allocator_test 2024-12-18T00:07:48.5439673Z inflating: build/bin/cpu_rng_test 2024-12-18T00:07:48.5498113Z inflating: build/bin/dispatch_key_set_test 2024-12-18T00:07:48.5553441Z inflating: build/bin/dlconvertor_test 2024-12-18T00:07:48.5618931Z inflating: build/bin/extension_backend_test 2024-12-18T00:07:48.5681734Z inflating: build/bin/half_test 2024-12-18T00:07:48.5785985Z inflating: build/bin/ivalue_test 2024-12-18T00:07:48.5840884Z inflating: build/bin/lazy_tensor_test 2024-12-18T00:07:48.5902536Z inflating: build/bin/math_kernel_test 2024-12-18T00:07:48.5961601Z inflating: build/bin/memory_format_test 2024-12-18T00:07:48.6021232Z inflating: build/bin/memory_overlapping_test 2024-12-18T00:07:48.6080897Z inflating: build/bin/mobile_memory_cleanup 2024-12-18T00:07:48.6138303Z inflating: build/bin/operator_name_test 2024-12-18T00:07:48.6200302Z inflating: build/bin/native_test 2024-12-18T00:07:48.6259046Z inflating: build/bin/packedtensoraccessor_test 2024-12-18T00:07:48.6316400Z inflating: build/bin/operators_test 2024-12-18T00:07:48.6390220Z inflating: build/bin/pow_test 2024-12-18T00:07:48.6452907Z inflating: build/bin/quantized_test 2024-12-18T00:07:48.6510841Z inflating: build/bin/reduce_ops_test 2024-12-18T00:07:48.6566811Z inflating: build/bin/reportMemoryUsage_test 2024-12-18T00:07:48.6625734Z inflating: build/bin/StorageUtils_test 2024-12-18T00:07:48.6689337Z inflating: build/bin/scalar_tensor_test 2024-12-18T00:07:48.6755638Z inflating: build/bin/scalar_test 2024-12-18T00:07:48.6813470Z inflating: build/bin/stride_properties_test 2024-12-18T00:07:48.6901967Z inflating: build/bin/tensor_iterator_test 2024-12-18T00:07:48.6961469Z inflating: build/bin/test_parallel 2024-12-18T00:07:48.7023488Z inflating: build/bin/type_ptr_test 2024-12-18T00:07:48.7024928Z inflating: build/bin/thread_init_test 2024-12-18T00:07:48.7092549Z inflating: build/bin/type_test 2024-12-18T00:07:48.7152431Z inflating: build/bin/undefined_tensor_test 2024-12-18T00:07:48.7153141Z inflating: build/bin/verify_api_visibility 2024-12-18T00:07:48.7210361Z inflating: build/bin/wrapdim_test 2024-12-18T00:07:48.7288117Z inflating: build/bin/legacy_vmap_test 2024-12-18T00:07:48.7346160Z inflating: build/bin/weakref_test 2024-12-18T00:07:48.7404076Z inflating: build/bin/xla_tensor_test 2024-12-18T00:07:48.7469190Z inflating: build/bin/IListRef_test 2024-12-18T00:07:48.7583554Z inflating: build/bin/List_test 2024-12-18T00:07:48.7714889Z inflating: build/bin/kernel_function_legacy_test 2024-12-18T00:07:48.7819280Z inflating: build/bin/kernel_function_test 2024-12-18T00:07:48.7892134Z inflating: build/bin/KernelFunction_test 2024-12-18T00:07:48.8002653Z inflating: build/bin/kernel_lambda_test 2024-12-18T00:07:48.8139905Z inflating: build/bin/kernel_lambda_legacy_test 2024-12-18T00:07:48.8206113Z inflating: build/bin/kernel_stackbased_test 2024-12-18T00:07:48.8263819Z inflating: build/bin/CppSignature_test 2024-12-18T00:07:48.8365909Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-12-18T00:07:48.8421381Z inflating: build/bin/op_allowlist_test 2024-12-18T00:07:48.8733121Z inflating: build/bin/op_registration_test 2024-12-18T00:07:48.8795726Z inflating: build/bin/backend_fallback_test 2024-12-18T00:07:48.8866146Z inflating: build/bin/inline_container_test 2024-12-18T00:07:48.8924765Z inflating: build/bin/cuda_apply_test 2024-12-18T00:07:48.8983133Z inflating: build/bin/cuda_allocator_test 2024-12-18T00:07:48.9046287Z inflating: build/bin/cuda_atomic_ops_test 2024-12-18T00:07:48.9109370Z inflating: build/bin/cuda_caching_host_allocator_test 2024-12-18T00:07:48.9188229Z inflating: build/bin/cuda_complex_math_test 2024-12-18T00:07:48.9252320Z inflating: build/bin/cuda_complex_test 2024-12-18T00:07:48.9310101Z inflating: build/bin/cuda_device_test 2024-12-18T00:07:48.9374181Z inflating: build/bin/cuda_cub_test 2024-12-18T00:07:48.9431918Z inflating: build/bin/cuda_dlconvertor_test 2024-12-18T00:07:48.9489575Z inflating: build/bin/cuda_integer_divider_test 2024-12-18T00:07:48.9561158Z inflating: build/bin/cuda_distributions_test 2024-12-18T00:07:48.9624223Z inflating: build/bin/cuda_generator_test 2024-12-18T00:07:48.9681293Z inflating: build/bin/cuda_half_test 2024-12-18T00:07:48.9740197Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-12-18T00:07:48.9796220Z inflating: build/bin/cuda_optional_test 2024-12-18T00:07:48.9850418Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-12-18T00:07:48.9920077Z inflating: build/bin/cuda_stream_test 2024-12-18T00:07:48.9979406Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-12-18T00:07:49.0034454Z inflating: build/bin/cuda_cudnn_test 2024-12-18T00:07:49.0092616Z inflating: build/bin/cuda_vectorized_test 2024-12-18T00:07:49.0164932Z inflating: build/bin/ProcessGroupGlooTest 2024-12-18T00:07:49.0182358Z inflating: build/bin/tutorial_tensorexpr 2024-12-18T00:07:49.0244815Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-12-18T00:07:49.0316599Z inflating: build/bin/ProcessGroupNCCLTest 2024-12-18T00:07:49.0385497Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-12-18T00:07:49.0443125Z inflating: build/bin/ProcessGroupUCCTest 2024-12-18T00:07:49.0505133Z inflating: build/bin/test_dist_autograd 2024-12-18T00:07:49.0506113Z inflating: build/bin/parallel_benchmark 2024-12-18T00:07:49.0584152Z inflating: build/bin/test_cpp_rpc 2024-12-18T00:07:49.0657818Z inflating: build/bin/test_mobile_nnc 2024-12-18T00:07:49.0666080Z inflating: build/bin/aot_model_compiler_test 2024-12-18T00:07:49.1026725Z inflating: build/bin/test_lazy 2024-12-18T00:07:49.1890994Z inflating: build/bin/test_tensorexpr 2024-12-18T00:07:49.3115674Z inflating: build/bin/test_api 2024-12-18T00:07:49.3720244Z inflating: build/bin/test_jit 2024-12-18T00:07:49.3720700Z creating: .additional_ci_files/ 2024-12-18T00:07:49.3812394Z inflating: .additional_ci_files/test-times.json 2024-12-18T00:07:49.4186460Z inflating: .additional_ci_files/test-class-times.json 2024-12-18T00:07:49.4222311Z ##[group]Run rm artifacts.zip 2024-12-18T00:07:49.4222784Z rm artifacts.zip 2024-12-18T00:07:49.4229496Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:49.4230018Z env: 2024-12-18T00:07:49.4230405Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:49.4230877Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:49.4231341Z ##[endgroup] 2024-12-18T00:07:49.4866556Z ##[group]Run df -H 2024-12-18T00:07:49.4866858Z df -H 2024-12-18T00:07:49.4872497Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:49.4872925Z env: 2024-12-18T00:07:49.4873177Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:49.4873566Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:49.4874120Z ##[endgroup] 2024-12-18T00:07:49.4915484Z Filesystem Size Used Avail Use% Mounted on 2024-12-18T00:07:49.4915991Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-12-18T00:07:49.4916406Z tmpfs 101G 0 101G 0% /dev/shm 2024-12-18T00:07:49.4916954Z tmpfs 41G 697k 41G 1% /run 2024-12-18T00:07:49.4917514Z /dev/nvme0n1p1 161G 53G 109G 33% / 2024-12-18T00:07:49.4917948Z tmpfs 101G 13k 101G 1% /tmp 2024-12-18T00:07:49.4918355Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-12-18T00:07:49.4918761Z tmpfs 21G 0 21G 0% /run/user/0 2024-12-18T00:07:49.4954015Z Prepare all required actions 2024-12-18T00:07:49.4954424Z Getting action download info 2024-12-18T00:07:49.6237905Z ##[group]Run ./.github/actions/download-td-artifacts 2024-12-18T00:07:49.6238319Z with: 2024-12-18T00:07:49.6238556Z env: 2024-12-18T00:07:49.6238828Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:49.6239327Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:49.6239722Z ##[endgroup] 2024-12-18T00:07:49.6276088Z ##[group]Run seemethere/download-artifact-s3@v4 2024-12-18T00:07:49.6276488Z with: 2024-12-18T00:07:49.6276734Z name: td_results 2024-12-18T00:07:49.6277019Z s3-bucket: gha-artifacts 2024-12-18T00:07:49.6277318Z region: us-east-1 2024-12-18T00:07:49.6277588Z env: 2024-12-18T00:07:49.6277832Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:49.6278339Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:49.6278723Z ##[endgroup] 2024-12-18T00:07:50.1103444Z (node:57263) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-12-18T00:07:50.1104038Z 2024-12-18T00:07:50.1104281Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-12-18T00:07:50.1104896Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-12-18T00:07:50.1105546Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-12-18T00:07:50.2351233Z Found 1 objects with prefix pytorch/pytorch/12383255652/td_results/ 2024-12-18T00:07:50.2352623Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-12-18T00:07:50.2984191Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-12-18T00:07:50.2988416Z Artifact download has finished successfully 2024-12-18T00:07:50.3181024Z ##[group]Run mkdir -p .additional_ci_files 2024-12-18T00:07:50.3181452Z mkdir -p .additional_ci_files 2024-12-18T00:07:50.3181962Z mv td_results.json .additional_ci_files/td_results.json || true 2024-12-18T00:07:50.3188465Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:50.3188875Z env: 2024-12-18T00:07:50.3189109Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:50.3189472Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:50.3189854Z ##[endgroup] 2024-12-18T00:07:50.3299512Z ##[group]Run .github/scripts/parse_ref.py 2024-12-18T00:07:50.3299922Z .github/scripts/parse_ref.py 2024-12-18T00:07:50.3305409Z shell: /usr/bin/bash -e {0} 2024-12-18T00:07:50.3305705Z env: 2024-12-18T00:07:50.3305938Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:50.3306300Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:50.3306671Z ##[endgroup] 2024-12-18T00:07:50.3585353Z Prepare all required actions 2024-12-18T00:07:50.3621017Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-12-18T00:07:50.3621408Z with: 2024-12-18T00:07:50.3621835Z github-token: *** 2024-12-18T00:07:50.3622104Z env: 2024-12-18T00:07:50.3622348Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:50.3622729Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:50.3623262Z ##[endgroup] 2024-12-18T00:07:50.3641464Z ##[group]Run set -eux 2024-12-18T00:07:50.3641765Z set -eux 2024-12-18T00:07:50.3642255Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-12-18T00:07:50.3652609Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:50.3653087Z env: 2024-12-18T00:07:50.3653345Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:50.3653744Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:50.3654362Z GITHUB_TOKEN: *** 2024-12-18T00:07:50.3654643Z ##[endgroup] 2024-12-18T00:07:50.3678478Z + python3 .github/scripts/get_workflow_job_id.py 12383255652 i-0c6125b7fa38a3d09 2024-12-18T00:07:52.5240206Z setting job-id=34566463052 2024-12-18T00:07:52.5241141Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:07:52.5435341Z Prepare all required actions 2024-12-18T00:07:52.5435759Z Getting action download info 2024-12-18T00:07:52.6718381Z ##[group]Run ./.github/actions/filter-test-configs 2024-12-18T00:07:52.6718753Z with: 2024-12-18T00:07:52.6719171Z github-token: *** 2024-12-18T00:07:52.6720399Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]} 2024-12-18T00:07:52.6721954Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:07:52.6722529Z env: 2024-12-18T00:07:52.6722747Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:52.6723276Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:52.6723865Z ##[endgroup] 2024-12-18T00:07:52.6766229Z ##[group]Run nick-fields/retry@v3.0.0 2024-12-18T00:07:52.6766605Z with: 2024-12-18T00:07:52.6766843Z shell: bash 2024-12-18T00:07:52.6767135Z timeout_minutes: 10 2024-12-18T00:07:52.6767429Z max_attempts: 5 2024-12-18T00:07:52.6767719Z retry_wait_seconds: 30 2024-12-18T00:07:52.6768665Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-12-18T00:07:52.6769658Z polling_interval_seconds: 1 2024-12-18T00:07:52.6769997Z warning_on_retry: true 2024-12-18T00:07:52.6770332Z continue_on_error: false 2024-12-18T00:07:52.6770639Z env: 2024-12-18T00:07:52.6770888Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:52.6771266Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:52.6771885Z GITHUB_TOKEN: *** 2024-12-18T00:07:52.6772169Z ##[endgroup] 2024-12-18T00:07:52.7767983Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-12-18T00:07:53.0306896Z Defaulting to user installation because normal site-packages is not writeable 2024-12-18T00:07:53.1765391Z Collecting requests==2.27.1 2024-12-18T00:07:53.2132539Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-12-18T00:07:53.4014327Z Collecting pyyaml==6.0.1 2024-12-18T00:07:53.4057685Z Downloading PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB) 2024-12-18T00:07:53.4383285Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-12-18T00:07:53.7575298Z Collecting charset-normalizer~=2.0.0 2024-12-18T00:07:53.7631522Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-12-18T00:07:53.7665409Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-12-18T00:07:53.8132142Z Collecting certifi>=2017.4.17 2024-12-18T00:07:53.8177006Z Downloading certifi-2024.12.14-py3-none-any.whl (164 kB) 2024-12-18T00:07:53.9043988Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2024-12-18T00:07:54.0244674Z Successfully installed certifi-2024.12.14 charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 2024-12-18T00:07:54.7595861Z Command completed after 1 attempt(s). 2024-12-18T00:07:54.7651353Z ##[group]Run set -x 2024-12-18T00:07:54.7651674Z set -x 2024-12-18T00:07:54.7651951Z  2024-12-18T00:07:54.7652419Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-12-18T00:07:54.7652993Z # in runner workspace 2024-12-18T00:07:54.7653453Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-12-18T00:07:54.7660511Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:54.7660918Z env: 2024-12-18T00:07:54.7661156Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:54.7661505Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:54.7661895Z ##[endgroup] 2024-12-18T00:07:54.7687889Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-12-18T00:07:54.7940147Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-12-18T00:07:54.7940595Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-12-18T00:07:54.7940949Z echo "Job name: ${JOB_NAME}" 2024-12-18T00:07:54.7941267Z  2024-12-18T00:07:54.7941695Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-12-18T00:07:54.7942225Z # in runner workspace 2024-12-18T00:07:54.7942672Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-12-18T00:07:54.7943187Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-12-18T00:07:54.7943557Z  --job-name "${JOB_NAME}" \ 2024-12-18T00:07:54.7944878Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]}" \ 2024-12-18T00:07:54.7946220Z  --selected-test-configs "" \ 2024-12-18T00:07:54.7946587Z  --pr-number "${PR_NUMBER}" \ 2024-12-18T00:07:54.7946928Z  --tag "${TAG}" \ 2024-12-18T00:07:54.7947231Z  --event-name "${EVENT_NAME}" \ 2024-12-18T00:07:54.7947593Z  --schedule "${SCHEDULE}" \ 2024-12-18T00:07:54.7947932Z  --branch "${HEAD_BRANCH}" 2024-12-18T00:07:54.7953340Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:54.7953749Z env: 2024-12-18T00:07:54.7953972Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:54.7954340Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:54.7954944Z GITHUB_TOKEN: *** 2024-12-18T00:07:54.7955471Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:07:54.7956051Z PR_NUMBER: 2024-12-18T00:07:54.7956294Z TAG: 2024-12-18T00:07:54.7956511Z EVENT_NAME: push 2024-12-18T00:07:54.7956762Z SCHEDULE: 2024-12-18T00:07:54.7956995Z HEAD_BRANCH: 2024-12-18T00:07:54.7957236Z ##[endgroup] 2024-12-18T00:07:54.7981374Z Workflow: pull 2024-12-18T00:07:54.7981973Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:07:55.0607144Z ##[group]Run echo "Filtered matrix:" 2024-12-18T00:07:55.0607561Z echo "Filtered matrix:" 2024-12-18T00:07:55.0609000Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]}" 2024-12-18T00:07:55.0610440Z  2024-12-18T00:07:55.0610707Z echo 2024-12-18T00:07:55.0611037Z echo "Is the current job unstable? False" 2024-12-18T00:07:55.0611443Z  2024-12-18T00:07:55.0611686Z echo 2024-12-18T00:07:55.0611986Z echo "Is keep-going label set? False" 2024-12-18T00:07:55.0612368Z  2024-12-18T00:07:55.0612609Z echo 2024-12-18T00:07:55.0612889Z echo "Renabled issues? " 2024-12-18T00:07:55.0618803Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:55.0619212Z env: 2024-12-18T00:07:55.0619448Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:55.0619809Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:55.0620196Z ##[endgroup] 2024-12-18T00:07:55.0643940Z Filtered matrix: 2024-12-18T00:07:55.0645551Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.g4dn.12xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: linux.g4dn.12xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: linux.g4dn.12xlarge.nvidia.gpu}]} 2024-12-18T00:07:55.0646880Z 2024-12-18T00:07:55.0647017Z Is the current job unstable? False 2024-12-18T00:07:55.0647280Z 2024-12-18T00:07:55.0647411Z Is keep-going label set? False 2024-12-18T00:07:55.0647633Z 2024-12-18T00:07:55.0647759Z Renabled issues? 2024-12-18T00:07:55.0698257Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-12-18T00:07:55.0698801Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-12-18T00:07:55.0704371Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:55.0704780Z env: 2024-12-18T00:07:55.0705015Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:55.0705378Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:55.0705769Z JOB_TIMEOUT: 360 2024-12-18T00:07:55.0706007Z ##[endgroup] 2024-12-18T00:07:55.0777134Z ##[group]Run set -x 2024-12-18T00:07:55.0777494Z set -x 2024-12-18T00:07:55.0777752Z  2024-12-18T00:07:55.0778038Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-12-18T00:07:55.0778593Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-12-18T00:07:55.0779008Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-12-18T00:07:55.0779397Z  TEST_COMMAND=.ci/onnx/test.sh 2024-12-18T00:07:55.0779727Z else 2024-12-18T00:07:55.0779998Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-12-18T00:07:55.0780334Z fi 2024-12-18T00:07:55.0780560Z  2024-12-18T00:07:55.0780923Z # detached container should get cleaned up by teardown_ec2_linux 2024-12-18T00:07:55.0781501Z # TODO: Stop building test binaries as part of the build phase 2024-12-18T00:07:55.0782007Z # Used for GPU_FLAG since that doesn't play nice 2024-12-18T00:07:55.0782452Z # shellcheck disable=SC2086,SC2090 2024-12-18T00:07:55.0782821Z container_name=$(docker run \ 2024-12-18T00:07:55.0783157Z  ${GPU_FLAG:-} \ 2024-12-18T00:07:55.0783484Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2024-12-18T00:07:55.0783856Z  -e BUILD_ENVIRONMENT \ 2024-12-18T00:07:55.0784162Z  -e PR_NUMBER \ 2024-12-18T00:07:55.0784457Z  -e GITHUB_ACTIONS \ 2024-12-18T00:07:55.0784767Z  -e GITHUB_REPOSITORY \ 2024-12-18T00:07:55.0785089Z  -e GITHUB_WORKFLOW \ 2024-12-18T00:07:55.0785398Z  -e GITHUB_JOB \ 2024-12-18T00:07:55.0785801Z  -e GITHUB_RUN_ID \ 2024-12-18T00:07:55.0786106Z  -e GITHUB_RUN_NUMBER \ 2024-12-18T00:07:55.0786431Z  -e GITHUB_RUN_ATTEMPT \ 2024-12-18T00:07:55.0786752Z  -e JOB_ID \ 2024-12-18T00:07:55.0787031Z  -e JOB_NAME \ 2024-12-18T00:07:55.0787302Z  -e BASE_SHA \ 2024-12-18T00:07:55.0787584Z  -e BRANCH \ 2024-12-18T00:07:55.0787856Z  -e SHA1 \ 2024-12-18T00:07:55.0788131Z  -e AWS_DEFAULT_REGION \ 2024-12-18T00:07:55.0788449Z  -e IN_WHEEL_TEST \ 2024-12-18T00:07:55.0788735Z  -e SHARD_NUMBER \ 2024-12-18T00:07:55.0789034Z  -e TEST_CONFIG \ 2024-12-18T00:07:55.0789335Z  -e NUM_TEST_SHARDS \ 2024-12-18T00:07:55.0789652Z  -e REENABLED_ISSUES \ 2024-12-18T00:07:55.0789979Z  -e CONTINUE_THROUGH_ERROR \ 2024-12-18T00:07:55.0790316Z  -e VERBOSE_TEST_LOGS \ 2024-12-18T00:07:55.0790621Z  -e TEST_SHOWLOCALS \ 2024-12-18T00:07:55.0790936Z  -e NO_TEST_TIMEOUT \ 2024-12-18T00:07:55.0791238Z  -e NO_TD \ 2024-12-18T00:07:55.0791509Z  -e TD_DISTRIBUTED \ 2024-12-18T00:07:55.0791814Z  -e PR_LABELS \ 2024-12-18T00:07:55.0792118Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-12-18T00:07:55.0792475Z  -e SCCACHE_BUCKET \ 2024-12-18T00:07:55.0792782Z  -e SCCACHE_REGION \ 2024-12-18T00:07:55.0793093Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-12-18T00:07:55.0793417Z  -e XLA_CUDA \ 2024-12-18T00:07:55.0793716Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-12-18T00:07:55.0794101Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-12-18T00:07:55.0794495Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-12-18T00:07:55.0794891Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-12-18T00:07:55.0795256Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-12-18T00:07:55.0795601Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-12-18T00:07:55.0795958Z  -e DASHBOARD_TAG \ 2024-12-18T00:07:55.0796257Z  -e IS_A100_RUNNER \ 2024-12-18T00:07:55.0796571Z  -e ARTIFACTS_FILE_SUFFIX \ 2024-12-18T00:07:55.0796964Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-12-18T00:07:55.0797388Z  --security-opt seccomp=unconfined \ 2024-12-18T00:07:55.0797844Z  --cap-add=SYS_PTRACE \ 2024-12-18T00:07:55.0798164Z  --ipc=host \ 2024-12-18T00:07:55.0798454Z  --shm-size="${SHM_SIZE}" \ 2024-12-18T00:07:55.0798771Z  --tty \ 2024-12-18T00:07:55.0799025Z  --detach \ 2024-12-18T00:07:55.0799298Z  --name="${container_name}" \ 2024-12-18T00:07:55.0799634Z  --user jenkins \ 2024-12-18T00:07:55.0800007Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-12-18T00:07:55.0800433Z  -w /var/lib/jenkins/workspace \ 2024-12-18T00:07:55.0800775Z  "${DOCKER_IMAGE}" 2024-12-18T00:07:55.0801050Z ) 2024-12-18T00:07:55.0801535Z # Propagate download.pytorch.org IP to container 2024-12-18T00:07:55.0802289Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-12-18T00:07:55.0803199Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-12-18T00:07:55.0804207Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-12-18T00:07:55.0810293Z shell: /usr/bin/bash -e {0} 2024-12-18T00:07:55.0810623Z env: 2024-12-18T00:07:55.0810869Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:55.0811272Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:55.0811786Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-12-18T00:07:55.0812212Z PR_NUMBER: 2024-12-18T00:07:55.0812502Z GITHUB_REPOSITORY: pytorch/pytorch 2024-12-18T00:07:55.0812860Z GITHUB_WORKFLOW: pull 2024-12-18T00:07:55.0813162Z GITHUB_JOB: test 2024-12-18T00:07:55.0813537Z GITHUB_RUN_ID: 12383255652 2024-12-18T00:07:55.0813868Z GITHUB_RUN_NUMBER: 276594 2024-12-18T00:07:55.0814191Z GITHUB_RUN_ATTEMPT: 1 2024-12-18T00:07:55.0814474Z JOB_ID: 34566463052 2024-12-18T00:07:55.0815061Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:07:55.0815934Z BRANCH: release/2.6 2024-12-18T00:07:55.0816237Z SHA1: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:07:55.0816637Z BASE_SHA: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:07:55.0816996Z TEST_CONFIG: distributed 2024-12-18T00:07:55.0817277Z SHARD_NUMBER: 3 2024-12-18T00:07:55.0817527Z NUM_TEST_SHARDS: 3 2024-12-18T00:07:55.0817789Z REENABLED_ISSUES: 2024-12-18T00:07:55.0818059Z CONTINUE_THROUGH_ERROR: False 2024-12-18T00:07:55.0818348Z VERBOSE_TEST_LOGS: False 2024-12-18T00:07:55.0818633Z TEST_SHOWLOCALS: False 2024-12-18T00:07:55.0818911Z NO_TEST_TIMEOUT: False 2024-12-18T00:07:55.0819184Z NO_TD: False 2024-12-18T00:07:55.0819428Z TD_DISTRIBUTED: False 2024-12-18T00:07:55.0819746Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-12-18T00:07:55.0820132Z SCCACHE_REGION: us-east-1 2024-12-18T00:07:55.0820433Z SCCACHE_S3_KEY_PREFIX: pull 2024-12-18T00:07:55.0820731Z SHM_SIZE: 2g 2024-12-18T00:07:55.0821480Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:07:55.0822300Z XLA_CUDA: 2024-12-18T00:07:55.0822666Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-12-18T00:07:55.0823149Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-12-18T00:07:55.0823490Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-12-18T00:07:55.0823815Z DASHBOARD_TAG: 2024-12-18T00:07:55.0824276Z HUGGING_FACE_HUB_TOKEN: *** 2024-12-18T00:07:55.0824722Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2024-12-18T00:07:55.0825043Z IS_A100_RUNNER: 0 2024-12-18T00:07:55.0825525Z ARTIFACTS_FILE_SUFFIX: test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_34566463052 2024-12-18T00:07:55.0826073Z ##[endgroup] 2024-12-18T00:07:55.0848803Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-12-18T00:07:55.0849298Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-12-18T00:07:55.0849736Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-12-18T00:07:55.0858924Z +++ nproc --ignore=2 2024-12-18T00:07:55.0892870Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=46 -e SCCACHE_BUCKET -e SCCACHE_REGION -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e IS_A100_RUNNER -e ARTIFACTS_FILE_SUFFIX --env-file=/tmp/github_env_12383255652 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:08:06.4875403Z + container_name=ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T00:08:06.4879908Z + grep download.pytorch.org /etc/hosts 2024-12-18T00:08:06.4881615Z + docker exec -i ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f sudo bash -c '/bin/cat >> /etc/hosts' 2024-12-18T00:08:06.6689367Z + echo DOCKER_CONTAINER_ID=ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T00:08:06.6692788Z ++ echo dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl 2024-12-18T00:08:06.6695337Z + docker exec -t ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f sh -c 'python3 -m pip install dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-12-18T00:08:07.0604398Z Processing ./dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl (from torch==2.6.0a0+git0cdf8b1) 2024-12-18T00:08:07.9462494Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.16.1) 2024-12-18T00:08:07.9464092Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (4.12.2) 2024-12-18T00:08:07.9465859Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (2.8.8) 2024-12-18T00:08:07.9468599Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.1.4) 2024-12-18T00:08:07.9471859Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (2024.10.0) 2024-12-18T00:08:07.9477496Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.13.1) 2024-12-18T00:08:07.9502126Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.3.0) 2024-12-18T00:08:07.9517856Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.3.0) 2024-12-18T00:08:07.9531774Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.22.4) 2024-12-18T00:08:07.9929858Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.0.2) 2024-12-18T00:08:08.3556348Z Installing collected packages: torch 2024-12-18T00:08:19.1805639Z Successfully installed torch-2.6.0a0+git0cdf8b1 2024-12-18T00:08:19.2537164Z + export TERM=vt100 2024-12-18T00:08:19.2537672Z + TERM=vt100 2024-12-18T00:08:19.2538175Z ++ dirname .ci/pytorch/test.sh 2024-12-18T00:08:19.2545643Z + source .ci/pytorch/common.sh 2024-12-18T00:08:19.2549454Z +++ dirname .ci/pytorch/common.sh 2024-12-18T00:08:19.2557384Z ++ source .ci/pytorch/common_utils.sh 2024-12-18T00:08:19.2570622Z +++ declare -f -t trap_add 2024-12-18T00:08:19.2570982Z ++ set -ex 2024-12-18T00:08:19.2571294Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-12-18T00:08:19.2571729Z ++ BUILD_TEST_LIBTORCH=0 2024-12-18T00:08:19.2572100Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-12-18T00:08:19.2572533Z + [[ -d /var/lib/jenkins/workspace ]] 2024-12-18T00:08:19.2572937Z ++ stat -c %u /var/lib/jenkins/workspace 2024-12-18T00:08:19.2588385Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-12-18T00:08:19.2588811Z + trap_add cleanup_workspace EXIT 2024-12-18T00:08:19.2589180Z + trap_add_cmd=cleanup_workspace 2024-12-18T00:08:19.2589509Z + shift 2024-12-18T00:08:19.2589772Z + for trap_add_name in "$@" 2024-12-18T00:08:19.2593582Z +++ trap -p EXIT 2024-12-18T00:08:19.2597097Z ++ eval 'extract_trap_cmd ' 2024-12-18T00:08:19.2597417Z +++ extract_trap_cmd 2024-12-18T00:08:19.2597692Z +++ printf '%s\n' '' 2024-12-18T00:08:19.2597981Z ++ printf '%s\n' cleanup_workspace 2024-12-18T00:08:19.2598606Z + trap -- ' 2024-12-18T00:08:19.2598847Z cleanup_workspace' EXIT 2024-12-18T00:08:19.2599195Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-12-18T00:08:19.7482671Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-12-18T00:08:19.7506674Z + echo 'Environment variables:' 2024-12-18T00:08:19.7507075Z Environment variables: 2024-12-18T00:08:19.7507356Z + env 2024-12-18T00:08:19.7520018Z INSTALLED_DB=yes 2024-12-18T00:08:19.7520409Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-12-18T00:08:19.7520763Z NVIDIA_VISIBLE_DEVICES=all 2024-12-18T00:08:19.7521082Z NV_NVML_DEV_VERSION=11.8.86-1 2024-12-18T00:08:19.7521560Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T00:08:19.7522062Z CONTINUE_THROUGH_ERROR=False 2024-12-18T00:08:19.7522443Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-12-18T00:08:19.7522878Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-12-18T00:08:19.7523611Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-12-18T00:08:19.7524058Z HOSTNAME=ca3abfad5faf 2024-12-18T00:08:19.7524734Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_87941e69-ff5c-4592-9633-4a06ca1ea7ea 2024-12-18T00:08:19.7525484Z GITHUB_ACTION=__self 2024-12-18T00:08:19.7525802Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-12-18T00:08:19.7527728Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-12-18T00:08:19.7529725Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-12-18T00:08:19.7530166Z NV_NVTX_VERSION=11.8.86-1 2024-12-18T00:08:19.7530485Z GITHUB_RUN_NUMBER=276594 2024-12-18T00:08:19.7530792Z TEST_CONFIG=distributed 2024-12-18T00:08:19.7531113Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-12-18T00:08:19.7531508Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-12-18T00:08:19.7531882Z IS_A100_RUNNER=0 2024-12-18T00:08:19.7532175Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-12-18T00:08:19.7532532Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-12-18T00:08:19.7533159Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2024-12-18T00:08:19.7533514Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-12-18T00:08:19.7534093Z GITHUB_TRIGGERING_ACTOR=malfet 2024-12-18T00:08:19.7534496Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-12-18T00:08:19.7535008Z GITHUB_REF_TYPE=branch 2024-12-18T00:08:19.7535309Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-12-18T00:08:19.7535626Z NCCL_VERSION=2.16.2-1 2024-12-18T00:08:19.7535965Z BASE_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:19.7536350Z XLA_CUDA= 2024-12-18T00:08:19.7536710Z HUGGING_FACE_HUB_TOKEN=*** 2024-12-18T00:08:19.7537109Z *** 2024-12-18T00:08:19.7537367Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-12-18T00:08:19.7537719Z GITHUB_REPOSITORY_ID=65600975 2024-12-18T00:08:19.7538081Z GITHUB_ACTIONS=true 2024-12-18T00:08:19.7538373Z NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:08:19.7538737Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-12-18T00:08:19.7539158Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-12-18T00:08:19.7539566Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:19.7539974Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-12-18T00:08:19.7540392Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:19.7541001Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/heads/release/2.6 2024-12-18T00:08:19.7541573Z UCC_HOME=/usr 2024-12-18T00:08:19.7541853Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-12-18T00:08:19.7542202Z VERBOSE_TEST_LOGS=False 2024-12-18T00:08:19.7542505Z NVIDIA_PRODUCT_NAME=CUDA 2024-12-18T00:08:19.7542850Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-12-18T00:08:19.7543261Z GITHUB_REF=refs/heads/release/2.6 2024-12-18T00:08:19.7543609Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-12-18T00:08:19.7544075Z SHARD_NUMBER=3 2024-12-18T00:08:19.7544352Z GITHUB_REF_PROTECTED=true 2024-12-18T00:08:19.7544645Z HOME=/var/lib/jenkins 2024-12-18T00:08:19.7544969Z GITHUB_API_URL=https://api.github.com 2024-12-18T00:08:19.7545353Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-12-18T00:08:19.7545757Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-12-18T00:08:19.7546167Z SCCACHE_S3_KEY_PREFIX=pull 2024-12-18T00:08:19.7546467Z CUDA_VERSION=11.8.0 2024-12-18T00:08:19.7546791Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-12-18T00:08:19.7547175Z NUM_TEST_SHARDS=3 2024-12-18T00:08:19.7547446Z UCX_HOME=/usr 2024-12-18T00:08:19.7547828Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-12-18T00:08:19.7548703Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_87941e69-ff5c-4592-9633-4a06ca1ea7ea 2024-12-18T00:08:19.7549715Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:08:19.7550705Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_87941e69-ff5c-4592-9633-4a06ca1ea7ea 2024-12-18T00:08:19.7551630Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-12-18T00:08:19.7552212Z GITHUB_EVENT_NAME=push 2024-12-18T00:08:19.7552494Z DASHBOARD_TAG= 2024-12-18T00:08:19.7552764Z GITHUB_RUN_ID=12383255652 2024-12-18T00:08:19.7553125Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-12-18T00:08:19.7553554Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-12-18T00:08:19.7554346Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_87941e69-ff5c-4592-9633-4a06ca1ea7ea 2024-12-18T00:08:19.7555125Z GITHUB_ACTOR=malfet 2024-12-18T00:08:19.7555407Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-12-18T00:08:19.7555738Z PR_NUMBER= 2024-12-18T00:08:19.7555992Z GITHUB_RUN_ATTEMPT=1 2024-12-18T00:08:19.7556288Z ANACONDA_PYTHON_VERSION=3.10 2024-12-18T00:08:19.7556666Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-12-18T00:08:19.7557071Z TERM=vt100 2024-12-18T00:08:19.7557342Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-12-18T00:08:19.7557699Z INSTALLED_VISION=yes 2024-12-18T00:08:19.7557983Z BRANCH=release/2.6 2024-12-18T00:08:19.7558252Z SCCACHE_REGION=us-east-1 2024-12-18T00:08:19.7558564Z OPENSSL_ROOT_DIR=/opt/openssl 2024-12-18T00:08:19.7558910Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-12-18T00:08:19.7559355Z CUDA_PATH=/usr/local/cuda 2024-12-18T00:08:19.7559965Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-12-18T00:08:19.7560629Z GITHUB_SERVER_URL=https://github.com 2024-12-18T00:08:19.7561049Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-12-18T00:08:19.7561454Z REENABLED_ISSUES= 2024-12-18T00:08:19.7561721Z SHLVL=1 2024-12-18T00:08:19.7561959Z MAX_JOBS=46 2024-12-18T00:08:19.7562205Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-12-18T00:08:19.7562512Z NVARCH=x86_64 2024-12-18T00:08:19.7562774Z GITHUB_ACTOR_ID=2453524 2024-12-18T00:08:19.7563263Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:19.7563895Z GITHUB_REF_NAME=release/2.6 2024-12-18T00:08:19.7564234Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-12-18T00:08:19.7564765Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-12-18T00:08:19.7565504Z GITHUB_JOB=test 2024-12-18T00:08:19.7565848Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-12-18T00:08:19.7566340Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-12-18T00:08:19.7566803Z NO_TEST_TIMEOUT=False 2024-12-18T00:08:19.7567091Z TD_DISTRIBUTED=False 2024-12-18T00:08:19.7567416Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-12-18T00:08:19.7567805Z GITHUB_REPOSITORY=pytorch/pytorch 2024-12-18T00:08:19.7568166Z NV_NVPROF_VERSION=11.8.87-1 2024-12-18T00:08:19.7568481Z GITHUB_RETENTION_DAYS=90 2024-12-18T00:08:19.7568798Z OPENSSL_DIR=/opt/openssl 2024-12-18T00:08:19.7569115Z GITHUB_ACTION_REPOSITORY= 2024-12-18T00:08:19.7570045Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:08:19.7571143Z GITHUB_BASE_REF= 2024-12-18T00:08:19.7571670Z ARTIFACTS_FILE_SUFFIX=test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_34566463052 2024-12-18T00:08:19.7572284Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-12-18T00:08:19.7572619Z CI=true 2024-12-18T00:08:19.7572890Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-12-18T00:08:19.7573259Z GITHUB_REPOSITORY_OWNER=pytorch 2024-12-18T00:08:19.7573587Z JOB_ID=34566463052 2024-12-18T00:08:19.7573859Z INSTALLED_PROTOBUF=yes 2024-12-18T00:08:19.7574156Z GITHUB_HEAD_REF= 2024-12-18T00:08:19.7574430Z GITHUB_ACTION_REF= 2024-12-18T00:08:19.7574773Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-12-18T00:08:19.7575178Z TEST_SHOWLOCALS=False 2024-12-18T00:08:19.7575483Z GITHUB_WORKFLOW=pull 2024-12-18T00:08:19.7575794Z DEBIAN_FRONTEND=noninteractive 2024-12-18T00:08:19.7576536Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_87941e69-ff5c-4592-9633-4a06ca1ea7ea 2024-12-18T00:08:19.7577393Z NO_TD=False 2024-12-18T00:08:19.7577766Z SKIP_SCCACHE_INITIALIZATION=1 2024-12-18T00:08:19.7578063Z _=/usr/bin/env 2024-12-18T00:08:19.7578418Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-12-18T00:08:19.7762749Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-12-18T00:08:19.7763881Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T00:08:19.7764598Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-12-18T00:08:19.7765468Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-12-18T00:08:19.7766017Z + BUILD_DIR=build 2024-12-18T00:08:19.7766321Z + BUILD_RENAMED_DIR=build_renamed 2024-12-18T00:08:19.7766679Z + BUILD_BIN_DIR=build/bin 2024-12-18T00:08:19.7766994Z + SHARD_NUMBER=3 2024-12-18T00:08:19.7767258Z + NUM_TEST_SHARDS=3 2024-12-18T00:08:19.7767557Z + export VALGRIND=ON 2024-12-18T00:08:19.7767845Z + VALGRIND=ON 2024-12-18T00:08:19.7768172Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-12-18T00:08:19.7768647Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-12-18T00:08:19.7769044Z + [[ 0 == \1 ]] 2024-12-18T00:08:19.7769294Z + [[ False == \1 ]] 2024-12-18T00:08:19.7769826Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-12-18T00:08:19.7770273Z ++ realpath build/custom_test_artifacts 2024-12-18T00:08:19.7775791Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-12-18T00:08:19.7776386Z + [[ -n '' ]] 2024-12-18T00:08:19.7776670Z + echo 'Environment variables' 2024-12-18T00:08:19.7777111Z Environment variables 2024-12-18T00:08:19.7777391Z + env 2024-12-18T00:08:19.7786021Z INSTALLED_DB=yes 2024-12-18T00:08:19.7786348Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-12-18T00:08:19.7786709Z NVIDIA_VISIBLE_DEVICES=all 2024-12-18T00:08:19.7787032Z NV_NVML_DEV_VERSION=11.8.86-1 2024-12-18T00:08:19.7787496Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T00:08:19.7788005Z CONTINUE_THROUGH_ERROR=False 2024-12-18T00:08:19.7788394Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-12-18T00:08:19.7788823Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-12-18T00:08:19.7789249Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-12-18T00:08:19.7789654Z HOSTNAME=ca3abfad5faf 2024-12-18T00:08:19.7790314Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_87941e69-ff5c-4592-9633-4a06ca1ea7ea 2024-12-18T00:08:19.7791035Z GITHUB_ACTION=__self 2024-12-18T00:08:19.7791342Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-12-18T00:08:19.7793189Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-12-18T00:08:19.7795240Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-12-18T00:08:19.7795654Z NV_NVTX_VERSION=11.8.86-1 2024-12-18T00:08:19.7795970Z GITHUB_RUN_NUMBER=276594 2024-12-18T00:08:19.7796279Z TEST_CONFIG=distributed 2024-12-18T00:08:19.7796601Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-12-18T00:08:19.7796978Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-12-18T00:08:19.7797322Z IS_A100_RUNNER=0 2024-12-18T00:08:19.7797610Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-12-18T00:08:19.7797971Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-12-18T00:08:19.7798542Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2024-12-18T00:08:19.7798884Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-12-18T00:08:19.7799207Z GITHUB_TRIGGERING_ACTOR=malfet 2024-12-18T00:08:19.7799593Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-12-18T00:08:19.7800005Z GITHUB_REF_TYPE=branch 2024-12-18T00:08:19.7800306Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-12-18T00:08:19.7800627Z NCCL_VERSION=2.16.2-1 2024-12-18T00:08:19.7800950Z BASE_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:19.7801335Z XLA_CUDA= 2024-12-18T00:08:19.7801710Z HUGGING_FACE_HUB_TOKEN=*** 2024-12-18T00:08:19.7802108Z *** 2024-12-18T00:08:19.7802366Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-12-18T00:08:19.7802715Z GITHUB_REPOSITORY_ID=65600975 2024-12-18T00:08:19.7803131Z GITHUB_ACTIONS=true 2024-12-18T00:08:19.7803425Z NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:08:19.7803985Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-12-18T00:08:19.7804423Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-12-18T00:08:19.7804839Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:19.7805241Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-12-18T00:08:19.7805666Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:19.7806305Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/heads/release/2.6 2024-12-18T00:08:19.7806920Z UCC_HOME=/usr 2024-12-18T00:08:19.7807209Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-12-18T00:08:19.7807568Z VERBOSE_TEST_LOGS=False 2024-12-18T00:08:19.7807876Z NVIDIA_PRODUCT_NAME=CUDA 2024-12-18T00:08:19.7808231Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-12-18T00:08:19.7808658Z GITHUB_REF=refs/heads/release/2.6 2024-12-18T00:08:19.7809024Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-12-18T00:08:19.7809451Z SHARD_NUMBER=3 2024-12-18T00:08:19.7809738Z GITHUB_REF_PROTECTED=true 2024-12-18T00:08:19.7810042Z HOME=/var/lib/jenkins 2024-12-18T00:08:19.7810374Z GITHUB_API_URL=https://api.github.com 2024-12-18T00:08:19.7810770Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-12-18T00:08:19.7811189Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-12-18T00:08:19.7811599Z SCCACHE_S3_KEY_PREFIX=pull 2024-12-18T00:08:19.7811915Z CUDA_VERSION=11.8.0 2024-12-18T00:08:19.7812241Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-12-18T00:08:19.7812636Z NUM_TEST_SHARDS=3 2024-12-18T00:08:19.7812910Z UCX_HOME=/usr 2024-12-18T00:08:19.7813311Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-12-18T00:08:19.7814210Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_87941e69-ff5c-4592-9633-4a06ca1ea7ea 2024-12-18T00:08:19.7815253Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:08:19.7816363Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_87941e69-ff5c-4592-9633-4a06ca1ea7ea 2024-12-18T00:08:19.7817287Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-12-18T00:08:19.7817875Z GITHUB_EVENT_NAME=push 2024-12-18T00:08:19.7818157Z DASHBOARD_TAG= 2024-12-18T00:08:19.7818427Z GITHUB_RUN_ID=12383255652 2024-12-18T00:08:19.7818788Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-12-18T00:08:19.7819217Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-12-18T00:08:19.7820009Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_87941e69-ff5c-4592-9633-4a06ca1ea7ea 2024-12-18T00:08:19.7820870Z GITHUB_ACTOR=malfet 2024-12-18T00:08:19.7821149Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-12-18T00:08:19.7821480Z PR_NUMBER= 2024-12-18T00:08:19.7821733Z GITHUB_RUN_ATTEMPT=1 2024-12-18T00:08:19.7822015Z VALGRIND=ON 2024-12-18T00:08:19.7822268Z ANACONDA_PYTHON_VERSION=3.10 2024-12-18T00:08:19.7822663Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-12-18T00:08:19.7823066Z TERM=vt100 2024-12-18T00:08:19.7823342Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-12-18T00:08:19.7823700Z INSTALLED_VISION=yes 2024-12-18T00:08:19.7823973Z BRANCH=release/2.6 2024-12-18T00:08:19.7824253Z SCCACHE_REGION=us-east-1 2024-12-18T00:08:19.7824569Z OPENSSL_ROOT_DIR=/opt/openssl 2024-12-18T00:08:19.7824917Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-12-18T00:08:19.7825281Z CUDA_PATH=/usr/local/cuda 2024-12-18T00:08:19.7825868Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-12-18T00:08:19.7826543Z GITHUB_SERVER_URL=https://github.com 2024-12-18T00:08:19.7827075Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-12-18T00:08:19.7827563Z REENABLED_ISSUES= 2024-12-18T00:08:19.7828082Z SHLVL=1 2024-12-18T00:08:19.7828427Z MAX_JOBS=46 2024-12-18T00:08:19.7828807Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-12-18T00:08:19.7829204Z NVARCH=x86_64 2024-12-18T00:08:19.7829585Z GITHUB_ACTOR_ID=2453524 2024-12-18T00:08:19.7830095Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:19.7830675Z GITHUB_REF_NAME=release/2.6 2024-12-18T00:08:19.7831124Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-12-18T00:08:19.7831778Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-12-18T00:08:19.7832334Z GITHUB_JOB=test 2024-12-18T00:08:19.7832742Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-12-18T00:08:19.7833357Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-12-18T00:08:19.7833910Z NO_TEST_TIMEOUT=False 2024-12-18T00:08:19.7834294Z TD_DISTRIBUTED=False 2024-12-18T00:08:19.7834711Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-12-18T00:08:19.7835198Z GITHUB_REPOSITORY=pytorch/pytorch 2024-12-18T00:08:19.7835641Z NV_NVPROF_VERSION=11.8.87-1 2024-12-18T00:08:19.7836109Z GITHUB_RETENTION_DAYS=90 2024-12-18T00:08:19.7836530Z OPENSSL_DIR=/opt/openssl 2024-12-18T00:08:19.7836879Z GITHUB_ACTION_REPOSITORY= 2024-12-18T00:08:19.7838006Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:08:19.7839052Z GITHUB_BASE_REF= 2024-12-18T00:08:19.7839690Z ARTIFACTS_FILE_SUFFIX=test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_34566463052 2024-12-18T00:08:19.7840436Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-12-18T00:08:19.7840868Z CI=true 2024-12-18T00:08:19.7841203Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-12-18T00:08:19.7841691Z GITHUB_REPOSITORY_OWNER=pytorch 2024-12-18T00:08:19.7842117Z JOB_ID=34566463052 2024-12-18T00:08:19.7842530Z INSTALLED_PROTOBUF=yes 2024-12-18T00:08:19.7843012Z GITHUB_HEAD_REF= 2024-12-18T00:08:19.7843529Z GITHUB_ACTION_REF= 2024-12-18T00:08:19.7844023Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-12-18T00:08:19.7844561Z TEST_SHOWLOCALS=False 2024-12-18T00:08:19.7844949Z GITHUB_WORKFLOW=pull 2024-12-18T00:08:19.7845423Z DEBIAN_FRONTEND=noninteractive 2024-12-18T00:08:19.7846232Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_87941e69-ff5c-4592-9633-4a06ca1ea7ea 2024-12-18T00:08:19.7847079Z NO_TD=False 2024-12-18T00:08:19.7847519Z SKIP_SCCACHE_INITIALIZATION=1 2024-12-18T00:08:19.7847959Z _=/usr/bin/env 2024-12-18T00:08:19.7848347Z + echo 'Testing pytorch' 2024-12-18T00:08:19.7848740Z Testing pytorch 2024-12-18T00:08:19.7849213Z + export LANG=C.UTF-8 2024-12-18T00:08:19.7849632Z + LANG=C.UTF-8 2024-12-18T00:08:19.7850029Z + PR_NUMBER= 2024-12-18T00:08:19.7850418Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-12-18T00:08:19.7850876Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-12-18T00:08:19.7851527Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-12-18T00:08:19.7852068Z + [[ distributed == \s\l\o\w ]] 2024-12-18T00:08:19.7852628Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-12-18T00:08:19.7853237Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-12-18T00:08:19.7853785Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-12-18T00:08:19.7854327Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-12-18T00:08:19.7854814Z + [[ distributed == *crossref* ]] 2024-12-18T00:08:19.7855418Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-12-18T00:08:19.7856020Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-12-18T00:08:19.7856591Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-12-18T00:08:19.7857054Z + pip_install --user ninja==1.10.2 2024-12-18T00:08:19.7857665Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2024-12-18T00:08:19.7858349Z + python3 -m pip install --progress-bar off --user ninja==1.10.2 2024-12-18T00:08:21.6244529Z Collecting ninja==1.10.2 2024-12-18T00:08:21.6415786Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-12-18T00:08:21.6513124Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-12-18T00:08:22.0046910Z Installing collected packages: ninja 2024-12-18T00:08:22.0126905Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-12-18T00:08:22.0128074Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-12-18T00:08:22.0170680Z Successfully installed ninja-1.10.2 2024-12-18T00:08:22.0842586Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:08:22.0844786Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:08:22.0846006Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-12-18T00:08:22.0846434Z + install_tlparse 2024-12-18T00:08:22.0846756Z + pip_install --user tlparse==0.3.25 2024-12-18T00:08:22.0847511Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2024-12-18T00:08:22.0848125Z + python3 -m pip install --progress-bar off --user tlparse==0.3.25 2024-12-18T00:08:22.4825633Z Collecting tlparse==0.3.25 2024-12-18T00:08:22.5005451Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-12-18T00:08:22.5123422Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-12-18T00:08:22.8850892Z Installing collected packages: tlparse 2024-12-18T00:08:22.9223776Z Successfully installed tlparse-0.3.25 2024-12-18T00:08:22.9891359Z ++ python -m site --user-base 2024-12-18T00:08:23.0142615Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:08:23.0143920Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-12-18T00:08:23.0144424Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-12-18T00:08:23.0144895Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-12-18T00:08:23.0145544Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-12-18T00:08:23.0146344Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-12-18T00:08:23.0146905Z + cd test 2024-12-18T00:08:23.0147294Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-12-18T00:08:24.7510920Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-12-18T00:08:24.7511923Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-12-18T00:08:24.7512466Z + DYNAMO_BENCHMARK_FLAGS=() 2024-12-18T00:08:24.7513661Z + [[ distributed == *pr_time_benchmarks* ]] 2024-12-18T00:08:24.7514062Z + [[ distributed == *dynamo_eager* ]] 2024-12-18T00:08:24.7514443Z + [[ distributed == *aot_eager* ]] 2024-12-18T00:08:24.7514798Z + [[ distributed == *aot_inductor* ]] 2024-12-18T00:08:24.7515162Z + [[ distributed == *inductor* ]] 2024-12-18T00:08:24.7515508Z + [[ distributed == *dynamic* ]] 2024-12-18T00:08:24.7515843Z + [[ distributed == *cpu* ]] 2024-12-18T00:08:24.7516203Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-12-18T00:08:24.7553128Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-12-18T00:08:24.7553657Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-12-18T00:08:24.7554065Z + cd test 2024-12-18T00:08:24.7554411Z + python -c 'import torch; print(torch.__config__.show())' 2024-12-18T00:08:26.7522655Z PyTorch built with: 2024-12-18T00:08:26.7523173Z - GCC 9.4 2024-12-18T00:08:26.7523520Z - C++ Version: 201703 2024-12-18T00:08:26.7524344Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-12-18T00:08:26.7525216Z - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-12-18T00:08:26.7525748Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-12-18T00:08:26.7526168Z - LAPACK is enabled (usually provided by MKL) 2024-12-18T00:08:26.7526571Z - NNPACK is enabled 2024-12-18T00:08:26.7526872Z - CPU capability usage: AVX512 2024-12-18T00:08:26.7527216Z - CUDA Runtime 11.8 2024-12-18T00:08:26.7527636Z - NVCC architecture flags: -gencode;arch=compute_75,code=sm_75 2024-12-18T00:08:26.7528101Z - CuDNN 90.1 2024-12-18T00:08:26.7528377Z - Magma 2.6.1 2024-12-18T00:08:26.7534931Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-12-18T00:08:26.7540796Z 2024-12-18T00:08:27.1921641Z + cd test 2024-12-18T00:08:27.1922256Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-12-18T00:08:28.6705757Z ATen/Parallel: 2024-12-18T00:08:28.6706174Z at::get_num_threads() : 24 2024-12-18T00:08:28.6706547Z at::get_num_interop_threads() : 24 2024-12-18T00:08:28.6706973Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-12-18T00:08:28.6707314Z omp_get_max_threads() : 24 2024-12-18T00:08:28.6707978Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-12-18T00:08:28.6708679Z mkl_get_max_threads() : 24 2024-12-18T00:08:28.6709134Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-12-18T00:08:28.6709658Z std::thread::hardware_concurrency() : 48 2024-12-18T00:08:28.6710017Z Environment variables: 2024-12-18T00:08:28.6710321Z OMP_NUM_THREADS : [not set] 2024-12-18T00:08:28.6710957Z MKL_NUM_THREADS : [not set] 2024-12-18T00:08:28.6711290Z ATen parallel backend: OpenMP 2024-12-18T00:08:28.6711505Z 2024-12-18T00:08:28.9131571Z + [[ distributed == *numpy_2* ]] 2024-12-18T00:08:28.9132097Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-12-18T00:08:28.9132569Z + [[ distributed == *backward* ]] 2024-12-18T00:08:28.9132932Z + [[ distributed == *xla* ]] 2024-12-18T00:08:28.9133315Z + [[ distributed == *executorch* ]] 2024-12-18T00:08:28.9133687Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-12-18T00:08:28.9134138Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-12-18T00:08:28.9134593Z + [[ distributed == distributed ]] 2024-12-18T00:08:28.9134942Z + test_distributed 2024-12-18T00:08:28.9135256Z + echo 'Testing distributed python tests' 2024-12-18T00:08:28.9135753Z Testing distributed python tests 2024-12-18T00:08:28.9136232Z + python test/run_test.py --distributed-tests --shard 3 3 --verbose 2024-12-18T00:08:29.0194398Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:08:29.0195465Z import pkg_resources 2024-12-18T00:08:34.1066047Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:08:34.1576235Z Ignoring disabled issues: [''] 2024-12-18T00:08:34.1747201Z Found test times from artifacts 2024-12-18T00:08:34.2390616Z Found test times from artifacts 2024-12-18T00:08:34.2407259Z Running all tests 2024-12-18T00:08:34.2528785Z Running parallel tests on 2 processes 2024-12-18T00:08:34.2532045Z Name: tests to run (est. time: 110.34min) 2024-12-18T00:08:34.2532533Z Serial tests (73): 2024-12-18T00:08:34.2532917Z distributed/test_inductor_collectives 1/1 2024-12-18T00:08:34.2533429Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 2024-12-18T00:08:34.2533925Z distributed/_tensor/test_common_rules 1/1 2024-12-18T00:08:34.2534373Z distributed/_tensor/test_init 1/1 2024-12-18T00:08:34.2534781Z distributed/_tensor/test_xla_integration 1/1 2024-12-18T00:08:34.2535235Z distributed/_tensor/test_convolution_ops 1/1 2024-12-18T00:08:34.2535768Z distributed/_tensor/test_math_ops 1/1 2024-12-18T00:08:34.2536160Z distributed/_tensor/test_pointwise_ops 1/1 2024-12-18T00:08:34.2536881Z distributed/_composable/fsdp/test_fully_shard_init 1/1 2024-12-18T00:08:34.2537364Z distributed/_tensor/test_embedding_ops 1/1 2024-12-18T00:08:34.2537776Z distributed/_tensor/test_matrix_ops 1/1 2024-12-18T00:08:34.2538203Z distributed/rpc/cuda/test_tensorpipe_agent 2/2 2024-12-18T00:08:34.2538684Z distributed/_shard/sharded_tensor/ops/test_embedding 1/1 2024-12-18T00:08:34.2539192Z distributed/flight_recorder/test_fr_analysis 1/1 2024-12-18T00:08:34.2539647Z distributed/_tensor/test_dtensor_compile 1/1 2024-12-18T00:08:34.2540136Z distributed/_composable/fsdp/test_fully_shard_compile 1/1 2024-12-18T00:08:34.2540617Z distributed/_tools/test_mem_tracker 1/1 2024-12-18T00:08:34.2541008Z distributed/fsdp/test_fsdp_fx 1/1 2024-12-18T00:08:34.2541365Z distributed/_tensor/test_api 1/1 2024-12-18T00:08:34.2541729Z distributed/test_backends 1/1 2024-12-18T00:08:34.2542164Z distributed/_composable/fsdp/test_fully_shard_logging 1/1 2024-12-18T00:08:34.2542637Z distributed/test_symmetric_memory 1/1 2024-12-18T00:08:34.2543028Z distributed/pipelining/test_stage 1/1 2024-12-18T00:08:34.2543417Z distributed/_tools/test_memory_tracker 1/1 2024-12-18T00:08:34.2543950Z distributed/_composable/test_composability/test_pp_composability 1/1 2024-12-18T00:08:34.2544487Z distributed/test_distributed_spawn 3/9 2024-12-18T00:08:34.2544882Z distributed/test_distributed_spawn 6/9 2024-12-18T00:08:34.2545277Z distributed/test_distributed_spawn 9/9 2024-12-18T00:08:34.2545750Z distributed/algorithms/quantization/test_quantization 1/1 2024-12-18T00:08:34.2547402Z distributed/elastic/test_control_plane 1/1 2024-12-18T00:08:34.2547827Z distributed/checkpoint/test_tp_checkpoint 1/1 2024-12-18T00:08:34.2548317Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 2024-12-18T00:08:34.2548776Z distributed/checkpoint/test_planner 1/1 2024-12-18T00:08:34.2549259Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 2024-12-18T00:08:34.2549784Z distributed/checkpoint/test_state_dict_utils 1/1 2024-12-18T00:08:34.2550206Z distributed/fsdp/test_shard_utils 1/1 2024-12-18T00:08:34.2550614Z distributed/checkpoint/test_compatibility 1/1 2024-12-18T00:08:34.2551054Z distributed/checkpoint/test_hsdp_checkpoint 1/1 2024-12-18T00:08:34.2551444Z distributed/test_nccl 1/1 2024-12-18T00:08:34.2551779Z distributed/test_functional_api 1/1 2024-12-18T00:08:34.2552161Z distributed/_tensor/test_optimizers 1/1 2024-12-18T00:08:34.2552638Z distributed/_composable/fsdp/test_fully_shard_grad_scaler 1/1 2024-12-18T00:08:34.2553117Z distributed/checkpoint/test_utils 1/1 2024-12-18T00:08:34.2553478Z distributed/_tensor/test_utils 1/1 2024-12-18T00:08:34.2553835Z distributed/test_c10d_nccl 1/3 2024-12-18T00:08:34.2554196Z distributed/fsdp/test_fsdp_overlap 1/1 2024-12-18T00:08:34.2554593Z distributed/fsdp/test_fsdp_optim_state 1/1 2024-12-18T00:08:34.2555060Z distributed/checkpoint/test_file_system_checkpoint 1/1 2024-12-18T00:08:34.2555509Z distributed/fsdp/test_fsdp_apply 1/1 2024-12-18T00:08:34.2555896Z distributed/fsdp/test_fsdp_flatten_params 1/1 2024-12-18T00:08:34.2556355Z distributed/optim/test_zero_redundancy_optimizer 1/1 2024-12-18T00:08:34.2556853Z distributed/_composable/fsdp/test_fully_shard_memory 1/1 2024-12-18T00:08:34.2557324Z distributed/fsdp/test_checkpoint_wrapper 1/1 2024-12-18T00:08:34.2557720Z distributed/fsdp/test_utils 1/1 2024-12-18T00:08:34.2558107Z distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 2024-12-18T00:08:34.2558552Z distributed/fsdp/test_fsdp_tp_integration 1/1 2024-12-18T00:08:34.2559138Z distributed/fsdp/test_fsdp_checkpoint 1/1 2024-12-18T00:08:34.2559542Z distributed/algorithms/test_join 1/1 2024-12-18T00:08:34.2559947Z distributed/fsdp/test_fsdp_hybrid_shard 1/1 2024-12-18T00:08:34.2560427Z distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 2024-12-18T00:08:34.2560991Z distributed/test_data_parallel 1/1 2024-12-18T00:08:34.2561371Z distributed/fsdp/test_fsdp_core 1/2 2024-12-18T00:08:34.2561753Z distributed/rpc/test_faulty_agent 1/1 2024-12-18T00:08:34.2562152Z distributed/fsdp/test_fsdp_fine_tune 1/1 2024-12-18T00:08:34.2562634Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 2024-12-18T00:08:34.2563202Z distributed/fsdp/test_fsdp_unshard_params 1/1 2024-12-18T00:08:34.2563833Z distributed/fsdp/test_fsdp_mixed_precision 1/1 2024-12-18T00:08:34.2564272Z distributed/test_c10d_spawn_gloo 1/1 2024-12-18T00:08:34.2564666Z distributed/test_c10d_spawn_ucc 1/1 2024-12-18T00:08:34.2565064Z distributed/test_c10d_spawn_nccl 1/1 2024-12-18T00:08:34.2565645Z distributed/test_store 1/1 2024-12-18T00:08:34.2566055Z distributed/elastic/timer/local_timer_example 1/1 2024-12-18T00:08:34.2566543Z distributed/elastic/utils/distributed_test 1/1 2024-12-18T00:08:34.2566969Z Parallel tests (0): 2024-12-18T00:08:34.2567295Z Name: excluded (est. time: 0.0min) 2024-12-18T00:08:34.2567634Z Serial tests (0): 2024-12-18T00:08:34.2567926Z Parallel tests (0): 2024-12-18T00:08:34.2597928Z Running distributed/test_inductor_collectives 1/1 ... [2024-12-18 00:08:34.259274] 2024-12-18T00:08:34.2598597Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:08:34.2599917Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_inductor_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:08:34.259616] 2024-12-18T00:11:07.0077214Z 2024-12-18T00:11:07.0078550Z distributed/test_inductor_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_inductor_collectives_1.1_63740d089a9e43f8_.log 2024-12-18T00:11:07.0107359Z Running 48 items in this shard: test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_contiguous_input, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_into_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_output_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor_cudagraph_trees, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_input_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_broadcast_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_c10d_functional_tagged_pt2_compliant, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_async_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_inductor_allreduce_eager_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_permute_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_reduce_scatter_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_backwards, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_GroupMember_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source__get_default_group, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_group_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_graphbreaks_unsupported_async_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_pg_var, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_args_match, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_list, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_to_all_single, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_unspecified, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op0, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op1, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op2, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op3, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op4, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_reduce_scatter, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_support_collective_op_with_async_op_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_pg, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allgather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allreduce, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_reduce_scatter_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_all_gather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_doesnt_mutate_shared, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_reduce_scatter_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_single_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_steal_buffer, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_meta 2024-12-18T00:11:07.0134611Z 2024-12-18T00:11:07.0135012Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 ... [2024-12-18 00:11:07.007919] 2024-12-18T00:11:07.0135789Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:11:07.0137144Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:11:07.008310] 2024-12-18T00:11:52.9874304Z 2024-12-18T00:11:52.9876075Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_binary_cmp_1.1_ee9735d507b5c4ac_.log 2024-12-18T00:11:52.9879800Z Running 4 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_allclose, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_allclose_tensor_specs, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_equal, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_equal_tensor_specs 2024-12-18T00:11:52.9882298Z 2024-12-18T00:11:52.9882603Z Running distributed/_tensor/test_common_rules 1/1 ... [2024-12-18 00:11:52.987929] 2024-12-18T00:11:52.9883351Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:11:52.9885225Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_common_rules.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:11:52.988284] 2024-12-18T00:13:05.0418927Z 2024-12-18T00:13:05.0422279Z distributed/_tensor/test_common_rules 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_common_rules_1.1_4a28379d84814dad_.log 2024-12-18T00:13:05.0427900Z Running 10 items in this shard: test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_basic_propagation, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_errors, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_linearity, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_merge_sharding, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_pointwise_propagation, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_enforce_sharding_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_broadcasting, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_suggestion 2024-12-18T00:13:05.0433021Z 2024-12-18T00:13:05.0433303Z Running distributed/_tensor/test_init 1/1 ... [2024-12-18 00:13:05.042000] 2024-12-18T00:13:05.0433825Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:13:05.0435068Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:13:05.042356] 2024-12-18T00:13:56.9885925Z 2024-12-18T00:13:56.9887117Z distributed/_tensor/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_init_1.1_1383db0fc1b52b4c_.log 2024-12-18T00:13:56.9890779Z Running 7 items in this shard: test/distributed/_tensor/test_init.py::DTensorInitOpsTest::test_init_ops, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_empty, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_full, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_ones, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros_full_mesh, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros_submesh 2024-12-18T00:13:56.9893671Z 2024-12-18T00:13:56.9894016Z Running distributed/_tensor/test_xla_integration 1/1 ... [2024-12-18 00:13:56.988650] 2024-12-18T00:13:56.9894621Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:13:56.9896406Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_xla_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:13:56.989018] 2024-12-18T00:14:01.3143987Z 2024-12-18T00:14:01.3145341Z distributed/_tensor/test_xla_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_xla_integration_1.1_135b28974c321750_.log 2024-12-18T00:14:01.3148089Z Running 3 items in this shard: test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_replicate, test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_shard, test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_2d 2024-12-18T00:14:01.3149967Z 2024-12-18T00:14:01.3150324Z Running distributed/_tensor/test_convolution_ops 1/1 ... [2024-12-18 00:14:01.314393] 2024-12-18T00:14:01.3150914Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:14:01.3152234Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_convolution_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:14:01.314769] 2024-12-18T00:14:36.8198572Z 2024-12-18T00:14:36.8199811Z distributed/_tensor/test_convolution_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_convolution_ops_1.1_2449f6e121be4711_.log 2024-12-18T00:14:36.8201856Z Running 2 items in this shard: test/distributed/_tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_depthwise_convolution, test/distributed/_tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_downsampling_convolution 2024-12-18T00:14:36.8203644Z 2024-12-18T00:14:36.8204133Z Running distributed/_tensor/test_math_ops 1/1 ... [2024-12-18 00:14:36.819803] 2024-12-18T00:14:36.8204699Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:14:36.8206004Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_math_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:14:36.820173] 2024-12-18T00:16:20.6589263Z 2024-12-18T00:16:20.6590497Z distributed/_tensor/test_math_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_math_ops_1.1_4f12d41d9d4b04b7_.log 2024-12-18T00:16:20.6596779Z Running 14 items in this shard: test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_foreach_norm, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_bwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_bwd_req_grad, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_fwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_linalg_eigh, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_linear_op_reductions, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_mean, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_nll_loss_and_cross_entropy, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_shard0_svd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_shard_math_ops, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_softmax_fwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_softmax_with_bwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_topk, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_upsampling 2024-12-18T00:16:20.6602463Z 2024-12-18T00:16:20.6602792Z Running distributed/_tensor/test_pointwise_ops 1/1 ... [2024-12-18 00:16:20.659095] 2024-12-18T00:16:20.6603504Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:16:20.6605360Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_pointwise_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:16:20.659496] 2024-12-18T00:16:25.6864543Z 2024-12-18T00:16:25.6866026Z distributed/_tensor/test_pointwise_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_pointwise_ops_1.1_48c6bf496cc72b65_.log 2024-12-18T00:16:25.6870112Z Running 7 items in this shard: test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_activations, test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_dropout, test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_dropout_backward, test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_dropout_errors, test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_mul_out, test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_partial_add, test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_partial_mul 2024-12-18T00:16:25.6873360Z 2024-12-18T00:16:25.6873730Z Running distributed/_composable/fsdp/test_fully_shard_init 1/1 ... [2024-12-18 00:16:25.686344] 2024-12-18T00:16:25.6874368Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:16:25.6875790Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:16:25.686762] 2024-12-18T00:16:36.6244488Z 2024-12-18T00:16:36.6245805Z distributed/_composable/fsdp/test_fully_shard_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_init_1.1_2650a7d3e24133c7_.log 2024-12-18T00:16:36.6271027Z Running 38 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardDeviceTensor::test_move_states_to_device_tensor, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardDeviceDTensor::test_move_states_to_device_dtensor_invalid, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardDeviceDTensor::test_move_states_to_device_dtensor_valid, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMeshArg::test_2d_mesh_without_mesh_dim_names, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMeshArg::test_invalid_mesh_ndim, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_duplicate, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_list_of_mlps, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_nested, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_nested_fully_shard_and_replicate, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_single, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_states_list_of_mlps, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_states_nested_fully_shard, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_states_shared_params_and_buffers, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardParamModuleInfos::test_get_param_module_infos_duplicates, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardParamModuleInfos::test_get_param_module_infos_list_of_mlps, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardParamModuleInfos::test_get_param_module_infos_shared_params, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardedParameterTensor::test_raise_noncontiguous_parameter, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardedParameterTensor::test_raise_scalar_parameter, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardedParameterTensor::test_shard_tensor_parameters, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardedParameterDTensor::test_shard_dtensor_parameters, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_fully_shard_double_lazy_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_fully_shard_is_root, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_fully_shard_module_and_param_fqns, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_fully_shard_multi_module_root, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_reset_sharded_param_in_lazy_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMetaDeviceInit::test_invalid_meta_device_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMetaDeviceInit::test_meta_device_1d_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMetaDeviceInit::test_meta_device_2d_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMetaDeviceInit::test_rank0_broadcast_meta_device_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardProcessGroupInit::test_1d_process_group_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardProcessGroupInit::test_2d_process_group_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardHSDPBroadcast::test_hsdp_broadcast_across_replicas, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardPlacementFn::test_init_1d_transformer_shard_dim_neg1, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardPlacementFn::test_init_1d_transformer_shard_largest_dim, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardPlacementFn::test_init_1d_uneven_shard_largest_dim, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardPlacementFn::test_init_2d_transformer_shard_diff_dim, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardPlacementFn::test_invalid_shard_dim, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardOldImport::test_old_import_training 2024-12-18T00:16:36.6295614Z 2024-12-18T00:16:36.6296035Z Running distributed/_tensor/test_embedding_ops 1/1 ... [2024-12-18 00:16:36.624483] 2024-12-18T00:16:36.6296596Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:16:36.6297853Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_embedding_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:16:36.624883] 2024-12-18T00:17:08.2220721Z 2024-12-18T00:17:08.2221957Z distributed/_tensor/test_embedding_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_embedding_ops_1.1_28c25ffea555a0c4_.log 2024-12-18T00:17:08.2224915Z Running 4 items in this shard: test/distributed/_tensor/test_embedding_ops.py::TestEmbeddingOp::test_multiple_embeddings_rowwise, test/distributed/_tensor/test_embedding_ops.py::TestEmbeddingOp::test_sharded_embedding_colwise, test/distributed/_tensor/test_embedding_ops.py::TestEmbeddingOp::test_sharded_embedding_colwise_max_norm_errors, test/distributed/_tensor/test_embedding_ops.py::TestEmbeddingOp::test_sharded_embedding_rowwise 2024-12-18T00:17:08.2227104Z 2024-12-18T00:17:08.2227729Z Running distributed/_tensor/test_matrix_ops 1/1 ... [2024-12-18 00:17:08.222059] 2024-12-18T00:17:08.2228356Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:17:08.2229626Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_matrix_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:17:08.222419] 2024-12-18T00:18:31.1654402Z 2024-12-18T00:18:31.1655580Z distributed/_tensor/test_matrix_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_matrix_ops_1.1_8f913be26be6f2d6_.log 2024-12-18T00:18:31.1660806Z Running 11 items in this shard: test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm_auto_redistribute, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm_empty_operand, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_baddbmm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_bmm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_dtensor_mm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_matmul, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_mm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_scaled_dot_product_attention, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_t, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_t_partial 2024-12-18T00:18:31.1666059Z 2024-12-18T00:18:31.1666448Z Running distributed/rpc/cuda/test_tensorpipe_agent 2/2 ... [2024-12-18 00:18:31.165206] 2024-12-18T00:18:31.1667062Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:18:31.1668498Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '--shard-id=2', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:18:31.165576] 2024-12-18T00:29:55.3957800Z 2024-12-18T00:29:55.3959074Z distributed/rpc/cuda/test_tensorpipe_agent 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.cuda.test_tensorpipe_agent_2.2_ac43784569642ac7_.log 2024-12-18T00:29:55.3997120Z Running 52 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_input_moved_to_cuda_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_input_moved_to_cuda_device_script, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_async_execution_nested_with_cuda_future, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_callback_changes_devices, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_cuda_sparse_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_custom_class_with_cuda_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_int, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_str, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_modify_tensor_inplace, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_replace_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_value_on_bad_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_multi, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_nested_multi, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu_to_gpu_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu_to_gpu_non_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_7, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_8, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_7, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_non_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_to_cpu_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_gpu, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_max_local_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_max_remote_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_many_to_one, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_loop, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_not_timeout, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_remote_response, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_response, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_multi_gpu, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_multi_gpu_self, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_remote, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_devices_option_mismatch, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization5, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_with_unpickleable_attributes, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_tensor_view_as_return_value, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_device_maps_backward_pass 2024-12-18T00:29:55.4032271Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_input_moved_to_cuda_device 2024-12-18T00:29:55.4033693Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_input_moved_to_cuda_device_script 2024-12-18T00:29:55.4035189Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_async_execution_nested_with_cuda_future 2024-12-18T00:29:55.4036736Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_callback_changes_devices 2024-12-18T00:29:55.4038364Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_cuda_sparse_tensor 2024-12-18T00:29:55.4040024Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor 2024-12-18T00:29:55.4041742Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_custom_class_with_cuda_tensor 2024-12-18T00:29:55.4043368Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_device 2024-12-18T00:29:55.4045046Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_int 2024-12-18T00:29:55.4046558Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_str 2024-12-18T00:29:55.4048110Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_modify_tensor_inplace 2024-12-18T00:29:55.4049664Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_replace_tensor 2024-12-18T00:29:55.4051215Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_value_on_bad_device 2024-12-18T00:29:55.4052697Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream 2024-12-18T00:29:55.4055466Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_multi 2024-12-18T00:29:55.4057039Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_nested_multi 2024-12-18T00:29:55.4058539Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu_to_gpu_default 2024-12-18T00:29:55.4060150Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu_to_gpu_non_default 2024-12-18T00:29:55.4061648Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_7 2024-12-18T00:29:55.4063198Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_8 2024-12-18T00:29:55.4064615Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_1 2024-12-18T00:29:55.4066448Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_7 2024-12-18T00:29:55.4067978Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_non_default 2024-12-18T00:29:55.4069520Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_to_cpu_default 2024-12-18T00:29:55.4071007Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_gpu 2024-12-18T00:29:55.4072519Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_max_local_device 2024-12-18T00:29:55.4074252Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_max_remote_device 2024-12-18T00:29:55.4075823Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_many_to_one 2024-12-18T00:29:55.4077337Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config 2024-12-18T00:29:55.4079039Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_loop 2024-12-18T00:29:55.4080562Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_not_timeout 2024-12-18T00:29:55.4082132Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_remote_response 2024-12-18T00:29:55.4084129Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_response 2024-12-18T00:29:55.4085687Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_multi_gpu 2024-12-18T00:29:55.4087189Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_multi_gpu_self 2024-12-18T00:29:55.4088662Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_remote 2024-12-18T00:29:55.4090133Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_devices_option_mismatch 2024-12-18T00:29:55.4091687Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization2 2024-12-18T00:29:55.4093314Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization4 2024-12-18T00:29:55.4094992Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization1 2024-12-18T00:29:55.4096693Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization2 2024-12-18T00:29:55.4098073Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization3 2024-12-18T00:29:55.4099452Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization4 2024-12-18T00:29:55.4101072Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization5 2024-12-18T00:29:55.4102525Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization3 2024-12-18T00:29:55.4104002Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization4 2024-12-18T00:29:55.4105473Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization1 2024-12-18T00:29:55.4106986Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization2 2024-12-18T00:29:55.4108448Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization4 2024-12-18T00:29:55.4110001Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_with_unpickleable_attributes 2024-12-18T00:29:55.4111474Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_tensor_view_as_return_value 2024-12-18T00:29:55.4112934Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_device_maps_backward_pass 2024-12-18T00:29:55.4113727Z 2024-12-18T00:29:55.4114101Z Running distributed/_shard/sharded_tensor/ops/test_embedding 1/1 ... [2024-12-18 00:29:55.396958] 2024-12-18T00:29:55.4114707Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:29:55.4116026Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:29:55.397326] 2024-12-18T00:30:13.6061685Z 2024-12-18T00:30:13.6063266Z distributed/_shard/sharded_tensor/ops/test_embedding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_embedding_1.1_3e1aaec7d8b82ad0_.log 2024-12-18T00:30:13.6066030Z Running 2 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_embedding.py::TestShardedEmbedding::test_sharded_embedding_colwise, test/distributed/_shard/sharded_tensor/ops/test_embedding.py::TestShardedEmbedding::test_sharded_embedding_rowwise 2024-12-18T00:30:13.6067410Z 2024-12-18T00:30:13.6067786Z Running distributed/flight_recorder/test_fr_analysis 1/1 ... [2024-12-18 00:30:13.606237] 2024-12-18T00:30:13.6068399Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:30:13.6069782Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/flight_recorder/test_fr_analysis.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:30:13.606636] 2024-12-18T00:30:17.8824920Z 2024-12-18T00:30:17.8828210Z distributed/flight_recorder/test_fr_analysis 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.flight_recorder.test_fr_analysis_1.1_6c9cda0b572ecd26_.log 2024-12-18T00:30:17.8829914Z Running 1 items in this shard: test/distributed/flight_recorder/test_fr_analysis.py::FlightRecorderEventTest::test_match_one_event 2024-12-18T00:30:17.8833365Z 2024-12-18T00:30:17.8833701Z Running distributed/_tensor/test_dtensor_compile 1/1 ... [2024-12-18 00:30:17.882930] 2024-12-18T00:30:17.8834274Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:30:17.8835579Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:30:17.883306] 2024-12-18T00:31:26.0132166Z 2024-12-18T00:31:26.0133457Z distributed/_tensor/test_dtensor_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_compile_1.1_cc99e17d88e4c2d6_.log 2024-12-18T00:31:26.0152377Z Running 30 items in this shard: test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_dynamo_disable, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_graph_break, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_contiguous_dtensor_noncontiguous_local_as_tangent, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_different_gradient_placement, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dont_recompile_on_same_placement_devicemesh, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamo_device_mesh_attrs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_noncontiguous_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_graph_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_redistribute_unbalanced_correct_strides, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_dynamic_shapes, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute_async, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_recompile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs_forward_hook, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_graph_input_is_async, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_unwrap_async_collective_tensor_tangent, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True 2024-12-18T00:31:26.0169251Z 2024-12-18T00:31:26.0169661Z Running distributed/_composable/fsdp/test_fully_shard_compile 1/1 ... [2024-12-18 00:31:26.013507] 2024-12-18T00:31:26.0170335Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:31:26.0171749Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:31:26.013883] 2024-12-18T00:35:00.9050478Z 2024-12-18T00:35:00.9051909Z distributed/_composable/fsdp/test_fully_shard_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_compile_1.1_e0e054511287f3aa_.log 2024-12-18T00:35:00.9063361Z Running 16 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompileCompute::test_disable_compiling_hooks, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_compiled_autograd_ctx, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_dynamo_recompiles_on_fsdp_layers, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_dynamo_trace_use_training_state, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_inductor_fullgraph_False, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_inductor_fullgraph_True, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_inductor, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_trace_fsdp_copy_, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor_fullgraph_False, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor_fullgraph_True 2024-12-18T00:35:00.9074969Z 2024-12-18T00:35:00.9075291Z Running distributed/_tools/test_mem_tracker 1/1 ... [2024-12-18 00:35:00.905426] 2024-12-18T00:35:00.9075867Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:35:00.9077263Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_mem_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:35:00.905793] 2024-12-18T00:35:06.1326423Z 2024-12-18T00:35:06.1328038Z distributed/_tools/test_mem_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_mem_tracker_1.1_7c59a2b4b6fdcba4_.log 2024-12-18T00:35:06.1330605Z Running 3 items in this shard: test/distributed/_tools/test_mem_tracker.py::TestMemTracker::test_cuda_tracker_equivalence, test/distributed/_tools/test_mem_tracker.py::TestMemTracker::test_tracker_attribution, test/distributed/_tools/test_mem_tracker.py::TestMemTracker::test_tracker_with_activation_checkpointing 2024-12-18T00:35:06.1332154Z 2024-12-18T00:35:06.1332909Z Running distributed/fsdp/test_fsdp_fx 1/1 ... [2024-12-18 00:35:06.132898] 2024-12-18T00:35:06.1333450Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:35:06.1335093Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:35:06.133288] 2024-12-18T00:35:11.8617822Z 2024-12-18T00:35:11.8618871Z distributed/fsdp/test_fsdp_fx 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_fx_1.1_e2bee0f2efd5a070_.log 2024-12-18T00:35:11.8620321Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_fx.py::TestSymbolicTracingCUDA::test_symbolic_tracing_outputs_cuda 2024-12-18T00:35:11.8620998Z 2024-12-18T00:35:11.8621263Z Running distributed/_tensor/test_api 1/1 ... [2024-12-18 00:35:11.861826] 2024-12-18T00:35:11.8621793Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:35:11.8623608Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:35:11.862192] 2024-12-18T00:36:11.6345925Z 2024-12-18T00:36:11.6347061Z distributed/_tensor/test_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_api_1.1_69d830f6f3e7705a_.log 2024-12-18T00:36:11.6351507Z Running 8 items in this shard: test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_casting, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn_warning, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_meta, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor_errors, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor_uneven_sharding 2024-12-18T00:36:11.6355339Z 2024-12-18T00:36:11.6355632Z Running distributed/test_backends 1/1 ... [2024-12-18 00:36:11.634650] 2024-12-18T00:36:11.6356162Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:36:11.6357419Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_backends.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:36:11.635013] 2024-12-18T00:36:17.1139819Z 2024-12-18T00:36:17.1140964Z distributed/test_backends 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_backends_1.1_913e5a0b16a41327_.log 2024-12-18T00:36:17.1142835Z Running 2 items in this shard: test/distributed/test_backends.py::TestMiscCollectiveUtilsCUDA::test_create_pg_cuda, test/distributed/test_backends.py::TestMiscCollectiveUtilsCUDA::test_device_to_backend_mapping_cuda 2024-12-18T00:36:17.1144005Z 2024-12-18T00:36:17.1144391Z Running distributed/_composable/fsdp/test_fully_shard_logging 1/1 ... [2024-12-18 00:36:17.113916] 2024-12-18T00:36:17.1145043Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:36:17.1146767Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_logging.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:36:17.114282] 2024-12-18T00:36:21.2503934Z 2024-12-18T00:36:21.2505394Z distributed/_composable/fsdp/test_fully_shard_logging 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_logging_1.1_d5e95c2e90d518d6_.log 2024-12-18T00:36:21.2506749Z Running 0 items in this shard: 2024-12-18T00:36:21.2506972Z 2024-12-18T00:36:21.2507275Z Running distributed/test_symmetric_memory 1/1 ... [2024-12-18 00:36:21.250239] 2024-12-18T00:36:21.2507859Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:36:21.2509121Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_symmetric_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:36:21.250588] 2024-12-18T00:36:26.0270964Z 2024-12-18T00:36:26.0272141Z distributed/test_symmetric_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_symmetric_memory_1.1_835a17eca150a31e_.log 2024-12-18T00:36:26.0341546Z Running 113 items in this shard: test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_allow_overlapping_devices, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_barrier_timeout, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_cuda_nvlink_connectivity_detection, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p_persistent, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_gather_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_gather_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_native_symm_mem_input_False_is_b_row_major_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_native_symm_mem_input_False_is_b_row_major_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_native_symm_mem_input_True_is_b_row_major_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_native_symm_mem_input_True_is_b_row_major_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_0_scale_mode_row-wise-replicated, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_0_scale_mode_row-wise-sharded, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_0_scale_mode_tensor-wise, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_1_scale_mode_row-wise-replicated, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_1_scale_mode_row-wise-sharded, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_1_scale_mode_tensor-wise, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_matmul_reduce_scatter_scatter_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_matmul_reduce_scatter_scatter_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_0_rowwise_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_0_rowwise_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_1_rowwise_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_1_rowwise_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_get_signal_pad, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_has_multicast_support, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_large_alloc, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_all_gather_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_all_gather_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_avg_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_avg_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_sum_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_sum_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_2, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_put_signal_timeout, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_wait_signal_timeout, test/distributed/test_symmetric_memory.py::SubgroupTest::test_subgroup, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::LoweringTest::test_lowering_one_shot_all_reduce, test/distributed/test_symmetric_memory.py::SymmMemSingleProcTest::test_memset32, test/distributed/test_symmetric_memory.py::SymmMemSingleProcTest::test_stream_write_value32 2024-12-18T00:36:26.0406505Z 2024-12-18T00:36:26.0406833Z Running distributed/pipelining/test_stage 1/1 ... [2024-12-18 00:36:26.027473] 2024-12-18T00:36:26.0407416Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:36:26.0408733Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_stage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:36:26.027840] 2024-12-18T00:36:37.4223190Z 2024-12-18T00:36:37.4224508Z distributed/pipelining/test_stage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_stage_1.1_0d605cb6674317dc_.log 2024-12-18T00:36:37.4228160Z Running 6 items in this shard: test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_errors, test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_with_fb_schedule, test/distributed/pipelining/test_stage.py::StageTest::test_manual, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass0, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass1, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_kwargs_ModelClass0 2024-12-18T00:36:37.4233104Z Running 6 items in this shard: test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_errors, test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_with_fb_schedule, test/distributed/pipelining/test_stage.py::StageTest::test_manual, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass0, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass1, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_kwargs_ModelClass0 2024-12-18T00:36:37.4235503Z 2024-12-18T00:36:37.4235807Z Running distributed/_tools/test_memory_tracker 1/1 ... [2024-12-18 00:36:37.422227] 2024-12-18T00:36:37.4236362Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:36:37.4237606Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_memory_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:36:37.422595] 2024-12-18T00:36:42.4005576Z 2024-12-18T00:36:42.4006876Z distributed/_tools/test_memory_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_memory_tracker_1.1_be743280618cb84b_.log 2024-12-18T00:36:42.4008846Z Running 1 items in this shard: test/distributed/_tools/test_memory_tracker.py::TestMemoryTracker::test_local_model 2024-12-18T00:36:42.4009520Z 2024-12-18T00:36:42.4010082Z Running distributed/_composable/test_composability/test_pp_composability 1/1 ... [2024-12-18 00:36:42.400789] 2024-12-18T00:36:42.4010804Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:36:42.4013673Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_composability/test_pp_composability.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:36:42.401174] 2024-12-18T00:39:17.4780348Z 2024-12-18T00:39:17.4781902Z distributed/_composable/test_composability/test_pp_composability 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_composability.test_pp_composability_1.1_2cd743d308019d00_.log 2024-12-18T00:39:17.4800648Z Running 21 items in this shard: test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass0_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass0_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass1_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass1_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass2_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass2_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass3_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass3_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass4_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass4_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass0_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass0_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass1_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass1_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass2_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass2_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass3_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass3_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass4_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass4_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_pp_and_dcp 2024-12-18T00:39:17.4818590Z 2024-12-18T00:39:17.4818887Z Running distributed/test_distributed_spawn 3/9 ... [2024-12-18 00:39:17.478063] 2024-12-18T00:39:17.4819440Z /usr/bin/mpiexec 2024-12-18T00:39:17.4819802Z MPI not available -- MPI backend tests will be skipped 2024-12-18T00:39:17.4820359Z Running distributed tests for the test backend with env init_method 2024-12-18T00:39:17.4820877Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:39:17.4822211Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:39:17.480138] 2024-12-18T00:39:21.6890788Z 2024-12-18T00:39:21.6891988Z distributed/test_distributed_spawn 3/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.9_eaa7f13919c45ec7_.log 2024-12-18T00:39:21.6893104Z Running 0 items in this shard: 2024-12-18T00:39:21.6893346Z 2024-12-18T00:39:21.6900504Z Running distributed tests for the test backend with file init_method 2024-12-18T00:39:21.6902399Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:39:21.6906893Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:39:21.690496] 2024-12-18T00:39:25.8955381Z 2024-12-18T00:39:25.8956860Z distributed/test_distributed_spawn 3/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.9_13492a65ea9e2555_.log 2024-12-18T00:39:25.8958063Z Running 0 items in this shard: 2024-12-18T00:39:25.8958287Z 2024-12-18T00:39:25.8961659Z Running distributed tests for the nccl backend with env init_method 2024-12-18T00:39:25.8963787Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:39:25.8968765Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:39:25.896679] 2024-12-18T00:43:59.7382369Z 2024-12-18T00:43:59.7385929Z distributed/test_distributed_spawn 3/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.9_e8ba8c21e1046a92_.log 2024-12-18T00:43:59.7403547Z Running 31 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-12-18T00:43:59.7421250Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2024-12-18T00:43:59.7422510Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2024-12-18T00:43:59.7423725Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2024-12-18T00:43:59.7425013Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2024-12-18T00:43:59.7426199Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2024-12-18T00:43:59.7427368Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2024-12-18T00:43:59.7428608Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2024-12-18T00:43:59.7429947Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2024-12-18T00:43:59.7431197Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2024-12-18T00:43:59.7432331Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2024-12-18T00:43:59.7433544Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2024-12-18T00:43:59.7434790Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2024-12-18T00:43:59.7436012Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2024-12-18T00:43:59.7437365Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2024-12-18T00:43:59.7438779Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2024-12-18T00:43:59.7440019Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2024-12-18T00:43:59.7441280Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2024-12-18T00:43:59.7442561Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2024-12-18T00:43:59.7444022Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2024-12-18T00:43:59.7445178Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2024-12-18T00:43:59.7446424Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2024-12-18T00:43:59.7447724Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2024-12-18T00:43:59.7448889Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2024-12-18T00:43:59.7450038Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2024-12-18T00:43:59.7451346Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2024-12-18T00:43:59.7452652Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2024-12-18T00:43:59.7454047Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2024-12-18T00:43:59.7455690Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2024-12-18T00:43:59.7457092Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2024-12-18T00:43:59.7458232Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2024-12-18T00:43:59.7459465Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-12-18T00:43:59.7460150Z 2024-12-18T00:43:59.7460392Z Running distributed tests for the nccl backend with file init_method 2024-12-18T00:43:59.7460862Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:43:59.7462083Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:43:59.739735] 2024-12-18T00:48:34.0046171Z 2024-12-18T00:48:34.0047384Z distributed/test_distributed_spawn 3/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.9_d9fa5c1202276875_.log 2024-12-18T00:48:34.0064987Z Running 31 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-12-18T00:48:34.0083043Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2024-12-18T00:48:34.0084563Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2024-12-18T00:48:34.0085828Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2024-12-18T00:48:34.0087334Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2024-12-18T00:48:34.0088565Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2024-12-18T00:48:34.0089770Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2024-12-18T00:48:34.0091048Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2024-12-18T00:48:34.0092428Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2024-12-18T00:48:34.0093723Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2024-12-18T00:48:34.0094902Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2024-12-18T00:48:34.0096379Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2024-12-18T00:48:34.0097589Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2024-12-18T00:48:34.0098773Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2024-12-18T00:48:34.0100084Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2024-12-18T00:48:34.0101452Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2024-12-18T00:48:34.0102652Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2024-12-18T00:48:34.0103871Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2024-12-18T00:48:34.0105218Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2024-12-18T00:48:34.0106386Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2024-12-18T00:48:34.0107594Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2024-12-18T00:48:34.0108702Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2024-12-18T00:48:34.0110084Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2024-12-18T00:48:34.0111166Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2024-12-18T00:48:34.0112250Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2024-12-18T00:48:34.0113426Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2024-12-18T00:48:34.0114664Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2024-12-18T00:48:34.0115987Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2024-12-18T00:48:34.0117454Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2024-12-18T00:48:34.0118863Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2024-12-18T00:48:34.0120151Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2024-12-18T00:48:34.0121452Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-12-18T00:48:34.0122181Z 2024-12-18T00:48:34.0122425Z Running distributed tests for the gloo backend with env init_method 2024-12-18T00:48:34.0122925Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:48:34.0124507Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:48:34.006493] 2024-12-18T00:53:32.8266968Z 2024-12-18T00:53:32.8268349Z distributed/test_distributed_spawn 3/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.9_f654a7e552e5fac5_.log 2024-12-18T00:53:32.8297164Z Running 31 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-12-18T00:53:32.8314219Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2024-12-18T00:53:32.8315431Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2024-12-18T00:53:32.8316619Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2024-12-18T00:53:32.8317876Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2024-12-18T00:53:32.8319050Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2024-12-18T00:53:32.8320180Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2024-12-18T00:53:32.8321392Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2024-12-18T00:53:32.8322696Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2024-12-18T00:53:32.8324195Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2024-12-18T00:53:32.8325367Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2024-12-18T00:53:32.8326564Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2024-12-18T00:53:32.8327965Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2024-12-18T00:53:32.8329245Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2024-12-18T00:53:32.8330648Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2024-12-18T00:53:32.8332092Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2024-12-18T00:53:32.8333375Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2024-12-18T00:53:32.8334675Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2024-12-18T00:53:32.8336104Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2024-12-18T00:53:32.8337270Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2024-12-18T00:53:32.8338365Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2024-12-18T00:53:32.8339519Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2024-12-18T00:53:32.8340739Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2024-12-18T00:53:32.8341889Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2024-12-18T00:53:32.8342969Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2024-12-18T00:53:32.8344144Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2024-12-18T00:53:32.8345374Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2024-12-18T00:53:32.8346689Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2024-12-18T00:53:32.8348145Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2024-12-18T00:53:32.8349469Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2024-12-18T00:53:32.8350680Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2024-12-18T00:53:32.8351991Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-12-18T00:53:32.8352719Z 2024-12-18T00:53:32.8352980Z Running distributed tests for the gloo backend with file init_method 2024-12-18T00:53:32.8353480Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:53:32.8354768Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:53:32.828420] 2024-12-18T00:58:31.5644314Z 2024-12-18T00:58:31.5645483Z distributed/test_distributed_spawn 3/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.9_483c9d7ef80d0881_.log 2024-12-18T00:58:31.5663121Z Running 31 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-12-18T00:58:31.5681128Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2024-12-18T00:58:31.5682386Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2024-12-18T00:58:31.5683877Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2024-12-18T00:58:31.5685217Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2024-12-18T00:58:31.5686583Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2024-12-18T00:58:31.5687767Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2024-12-18T00:58:31.5689046Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2024-12-18T00:58:31.5690435Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2024-12-18T00:58:31.5691722Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2024-12-18T00:58:31.5692905Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2024-12-18T00:58:31.5694097Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2024-12-18T00:58:31.5695478Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2024-12-18T00:58:31.5696803Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2024-12-18T00:58:31.5698101Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2024-12-18T00:58:31.5699475Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2024-12-18T00:58:31.5700755Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2024-12-18T00:58:31.5701973Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2024-12-18T00:58:31.5703222Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2024-12-18T00:58:31.5704392Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2024-12-18T00:58:31.5705489Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2024-12-18T00:58:31.5706663Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2024-12-18T00:58:31.5707874Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2024-12-18T00:58:31.5708964Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2024-12-18T00:58:31.5710054Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2024-12-18T00:58:31.5711218Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2024-12-18T00:58:31.5712439Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2024-12-18T00:58:31.5713756Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2024-12-18T00:58:31.5715199Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2024-12-18T00:58:31.5716531Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2024-12-18T00:58:31.5717968Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2024-12-18T00:58:31.5719324Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-12-18T00:58:31.5720079Z 2024-12-18T00:58:31.5720367Z Running distributed/test_distributed_spawn 6/9 ... [2024-12-18 00:58:31.565818] 2024-12-18T00:58:31.5720896Z /usr/bin/mpiexec 2024-12-18T00:58:31.5721254Z MPI not available -- MPI backend tests will be skipped 2024-12-18T00:58:31.5721819Z Running distributed tests for the test backend with env init_method 2024-12-18T00:58:31.5722313Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:58:31.5723906Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:58:31.567799] 2024-12-18T00:58:35.7714518Z 2024-12-18T00:58:35.7715733Z distributed/test_distributed_spawn 6/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.9_765f398be333954e_.log 2024-12-18T00:58:35.7716784Z Running 0 items in this shard: 2024-12-18T00:58:35.7717005Z 2024-12-18T00:58:35.7721405Z Running distributed tests for the test backend with file init_method 2024-12-18T00:58:35.7723498Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:58:35.7728132Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:58:35.772608] 2024-12-18T00:58:39.9726453Z 2024-12-18T00:58:39.9727667Z distributed/test_distributed_spawn 6/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.9_f2ea893aec4eb2b1_.log 2024-12-18T00:58:39.9728760Z Running 0 items in this shard: 2024-12-18T00:58:39.9728987Z 2024-12-18T00:58:39.9734834Z Running distributed tests for the nccl backend with env init_method 2024-12-18T00:58:39.9736798Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:58:39.9741084Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:58:39.973916] 2024-12-18T01:03:35.0216172Z 2024-12-18T01:03:35.0217367Z distributed/test_distributed_spawn 6/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.9_5522ec59a8ed4c64_.log 2024-12-18T01:03:35.0234378Z Running 30 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-12-18T01:03:35.0251812Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2024-12-18T01:03:35.0253393Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2024-12-18T01:03:35.0254880Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2024-12-18T01:03:35.0256266Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2024-12-18T01:03:35.0257450Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2024-12-18T01:03:35.0258706Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2024-12-18T01:03:35.0259959Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2024-12-18T01:03:35.0261218Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2024-12-18T01:03:35.0262496Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2024-12-18T01:03:35.0263787Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2024-12-18T01:03:35.0265003Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2024-12-18T01:03:35.0266547Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2024-12-18T01:03:35.0268041Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2024-12-18T01:03:35.0269533Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2024-12-18T01:03:35.0271159Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2024-12-18T01:03:35.0272621Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2024-12-18T01:03:35.0273839Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2024-12-18T01:03:35.0275070Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2024-12-18T01:03:35.0276371Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2024-12-18T01:03:35.0277915Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2024-12-18T01:03:35.0279270Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2024-12-18T01:03:35.0280553Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2024-12-18T01:03:35.0281725Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2024-12-18T01:03:35.0282803Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2024-12-18T01:03:35.0284149Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2024-12-18T01:03:35.0285258Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2024-12-18T01:03:35.0286419Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2024-12-18T01:03:35.0287593Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2024-12-18T01:03:35.0288790Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-12-18T01:03:35.0290053Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-12-18T01:03:35.0290761Z 2024-12-18T01:03:35.0291021Z Running distributed tests for the nccl backend with file init_method 2024-12-18T01:03:35.0291554Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:03:35.0292930Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:03:35.023599] 2024-12-18T01:08:30.9341170Z 2024-12-18T01:08:30.9342330Z distributed/test_distributed_spawn 6/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.9_b27eb46f6162468f_.log 2024-12-18T01:08:30.9359149Z Running 30 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-12-18T01:08:30.9376966Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2024-12-18T01:08:30.9378583Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2024-12-18T01:08:30.9379985Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2024-12-18T01:08:30.9381206Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2024-12-18T01:08:30.9382332Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2024-12-18T01:08:30.9383556Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2024-12-18T01:08:30.9384769Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2024-12-18T01:08:30.9386086Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2024-12-18T01:08:30.9387336Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2024-12-18T01:08:30.9388588Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2024-12-18T01:08:30.9389762Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2024-12-18T01:08:30.9390916Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2024-12-18T01:08:30.9392216Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2024-12-18T01:08:30.9393630Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2024-12-18T01:08:30.9395169Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2024-12-18T01:08:30.9396544Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2024-12-18T01:08:30.9397684Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2024-12-18T01:08:30.9398935Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2024-12-18T01:08:30.9400161Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2024-12-18T01:08:30.9401441Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2024-12-18T01:08:30.9402732Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2024-12-18T01:08:30.9404168Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2024-12-18T01:08:30.9405326Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2024-12-18T01:08:30.9406486Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2024-12-18T01:08:30.9407623Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2024-12-18T01:08:30.9408741Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2024-12-18T01:08:30.9409905Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2024-12-18T01:08:30.9411074Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2024-12-18T01:08:30.9412292Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-12-18T01:08:30.9413537Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-12-18T01:08:30.9414245Z 2024-12-18T01:08:30.9414502Z Running distributed tests for the gloo backend with env init_method 2024-12-18T01:08:30.9415032Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:08:30.9416544Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:08:30.935874] 2024-12-18T01:13:37.9149770Z 2024-12-18T01:13:37.9150921Z distributed/test_distributed_spawn 6/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.9_03572a2149d9e5ed_.log 2024-12-18T01:13:37.9168209Z Running 30 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-12-18T01:13:37.9185267Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2024-12-18T01:13:37.9186731Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2024-12-18T01:13:37.9188255Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2024-12-18T01:13:37.9189483Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2024-12-18T01:13:37.9190591Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2024-12-18T01:13:37.9191829Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2024-12-18T01:13:37.9193056Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2024-12-18T01:13:37.9194284Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2024-12-18T01:13:37.9195529Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2024-12-18T01:13:37.9196810Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2024-12-18T01:13:37.9197986Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2024-12-18T01:13:37.9199141Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2024-12-18T01:13:37.9200431Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2024-12-18T01:13:37.9201912Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2024-12-18T01:13:37.9203534Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2024-12-18T01:13:37.9205166Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2024-12-18T01:13:37.9206363Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2024-12-18T01:13:37.9207600Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2024-12-18T01:13:37.9208910Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2024-12-18T01:13:37.9210454Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2024-12-18T01:13:37.9211763Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2024-12-18T01:13:37.9212959Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2024-12-18T01:13:37.9214043Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2024-12-18T01:13:37.9215118Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2024-12-18T01:13:37.9216302Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2024-12-18T01:13:37.9217292Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2024-12-18T01:13:37.9218325Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2024-12-18T01:13:37.9219430Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2024-12-18T01:13:37.9220508Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-12-18T01:13:37.9221817Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-12-18T01:13:37.9222471Z 2024-12-18T01:13:37.9222746Z Running distributed tests for the gloo backend with file init_method 2024-12-18T01:13:37.9223236Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:13:37.9224532Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:13:37.916408] 2024-12-18T01:18:44.3318974Z 2024-12-18T01:18:44.3320187Z distributed/test_distributed_spawn 6/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.9_4572ef768c581112_.log 2024-12-18T01:18:44.3337885Z Running 30 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-12-18T01:18:44.3354554Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2024-12-18T01:18:44.3356080Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2024-12-18T01:18:44.3357518Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2024-12-18T01:18:44.3358785Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2024-12-18T01:18:44.3359961Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2024-12-18T01:18:44.3361223Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2024-12-18T01:18:44.3362470Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2024-12-18T01:18:44.3363823Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2024-12-18T01:18:44.3365590Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2024-12-18T01:18:44.3366946Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2024-12-18T01:18:44.3368204Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2024-12-18T01:18:44.3369432Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2024-12-18T01:18:44.3370804Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2024-12-18T01:18:44.3372298Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2024-12-18T01:18:44.3373937Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2024-12-18T01:18:44.3375411Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2024-12-18T01:18:44.3376615Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2024-12-18T01:18:44.3378009Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2024-12-18T01:18:44.3379349Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2024-12-18T01:18:44.3380552Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2024-12-18T01:18:44.3381975Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2024-12-18T01:18:44.3383241Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2024-12-18T01:18:44.3384325Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2024-12-18T01:18:44.3385409Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2024-12-18T01:18:44.3386481Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2024-12-18T01:18:44.3387619Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2024-12-18T01:18:44.3388848Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2024-12-18T01:18:44.3389955Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2024-12-18T01:18:44.3391094Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-12-18T01:18:44.3392263Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-12-18T01:18:44.3392922Z 2024-12-18T01:18:44.3393202Z Running distributed/test_distributed_spawn 9/9 ... [2024-12-18 01:18:44.333288] 2024-12-18T01:18:44.3393715Z /usr/bin/mpiexec 2024-12-18T01:18:44.3394063Z MPI not available -- MPI backend tests will be skipped 2024-12-18T01:18:44.3394605Z Running distributed tests for the test backend with env init_method 2024-12-18T01:18:44.3395175Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:18:44.3396454Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=9', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:18:44.335079] 2024-12-18T01:18:48.5293959Z 2024-12-18T01:18:48.5295150Z distributed/test_distributed_spawn 9/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_9.9_6bb790fe7c50bcb5_.log 2024-12-18T01:18:48.5296336Z Running 0 items in this shard: 2024-12-18T01:18:48.5296558Z 2024-12-18T01:18:48.5302825Z Running distributed tests for the test backend with file init_method 2024-12-18T01:18:48.5304539Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:18:48.5308546Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=9', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:18:48.530674] 2024-12-18T01:18:52.7310261Z 2024-12-18T01:18:52.7311482Z distributed/test_distributed_spawn 9/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_9.9_63555b7fa99fd534_.log 2024-12-18T01:18:52.7312547Z Running 0 items in this shard: 2024-12-18T01:18:52.7312791Z 2024-12-18T01:18:52.7315306Z Running distributed tests for the nccl backend with env init_method 2024-12-18T01:18:52.7317044Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:18:52.7321792Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=9', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:18:52.731985] 2024-12-18T01:22:51.9433835Z 2024-12-18T01:22:51.9436555Z distributed/test_distributed_spawn 9/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_9.9_9f5b9f6e0f791298_.log 2024-12-18T01:22:51.9452355Z Running 28 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-12-18T01:22:51.9467910Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2024-12-18T01:22:51.9469166Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2024-12-18T01:22:51.9470426Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2024-12-18T01:22:51.9471684Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2024-12-18T01:22:51.9472868Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2024-12-18T01:22:51.9474067Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2024-12-18T01:22:51.9475314Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2024-12-18T01:22:51.9476604Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2024-12-18T01:22:51.9478052Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2024-12-18T01:22:51.9479232Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2024-12-18T01:22:51.9480583Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2024-12-18T01:22:51.9481989Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2024-12-18T01:22:51.9483431Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2024-12-18T01:22:51.9484896Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2024-12-18T01:22:51.9486166Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2024-12-18T01:22:51.9487476Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2024-12-18T01:22:51.9488688Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2024-12-18T01:22:51.9489840Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2024-12-18T01:22:51.9491189Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2024-12-18T01:22:51.9492579Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2024-12-18T01:22:51.9493931Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2024-12-18T01:22:51.9495108Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2024-12-18T01:22:51.9496489Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2024-12-18T01:22:51.9497612Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2024-12-18T01:22:51.9498706Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2024-12-18T01:22:51.9499786Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-12-18T01:22:51.9500925Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-12-18T01:22:51.9502192Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-12-18T01:22:51.9502904Z 2024-12-18T01:22:51.9503155Z Running distributed tests for the nccl backend with file init_method 2024-12-18T01:22:51.9503654Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:22:51.9504944Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=9', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:22:51.944795] 2024-12-18T01:26:51.0068653Z 2024-12-18T01:26:51.0069838Z distributed/test_distributed_spawn 9/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_9.9_9c71e338cc606e79_.log 2024-12-18T01:26:51.0085848Z Running 28 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-12-18T01:26:51.0101335Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2024-12-18T01:26:51.0102575Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2024-12-18T01:26:51.0103797Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2024-12-18T01:26:51.0105030Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2024-12-18T01:26:51.0106167Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2024-12-18T01:26:51.0107338Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2024-12-18T01:26:51.0108547Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2024-12-18T01:26:51.0109862Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2024-12-18T01:26:51.0111098Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2024-12-18T01:26:51.0112289Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2024-12-18T01:26:51.0113632Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2024-12-18T01:26:51.0115061Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2024-12-18T01:26:51.0116403Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2024-12-18T01:26:51.0117664Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2024-12-18T01:26:51.0118886Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2024-12-18T01:26:51.0120149Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2024-12-18T01:26:51.0121324Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2024-12-18T01:26:51.0122442Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2024-12-18T01:26:51.0123987Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2024-12-18T01:26:51.0125390Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2024-12-18T01:26:51.0126736Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2024-12-18T01:26:51.0127917Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2024-12-18T01:26:51.0129127Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2024-12-18T01:26:51.0130323Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2024-12-18T01:26:51.0131492Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2024-12-18T01:26:51.0132629Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-12-18T01:26:51.0133852Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-12-18T01:26:51.0135345Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-12-18T01:26:51.0136085Z 2024-12-18T01:26:51.0136333Z Running distributed tests for the gloo backend with env init_method 2024-12-18T01:26:51.0136844Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:26:51.0138175Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=9', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:26:51.008957] 2024-12-18T01:31:31.2907266Z 2024-12-18T01:31:31.2909200Z distributed/test_distributed_spawn 9/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_9.9_b385968c42fb2a86_.log 2024-12-18T01:31:31.2924681Z Running 28 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-12-18T01:31:31.2940345Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2024-12-18T01:31:31.2941552Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2024-12-18T01:31:31.2942770Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2024-12-18T01:31:31.2943994Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2024-12-18T01:31:31.2945144Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2024-12-18T01:31:31.2946314Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2024-12-18T01:31:31.2947522Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2024-12-18T01:31:31.2948834Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2024-12-18T01:31:31.2950065Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2024-12-18T01:31:31.2951247Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2024-12-18T01:31:31.2952590Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2024-12-18T01:31:31.2954021Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2024-12-18T01:31:31.2955373Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2024-12-18T01:31:31.2956621Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2024-12-18T01:31:31.2957849Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2024-12-18T01:31:31.2959108Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2024-12-18T01:31:31.2960287Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2024-12-18T01:31:31.2961466Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2024-12-18T01:31:31.2962685Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2024-12-18T01:31:31.2964344Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2024-12-18T01:31:31.2965904Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2024-12-18T01:31:31.2967065Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2024-12-18T01:31:31.2968271Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2024-12-18T01:31:31.2969472Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2024-12-18T01:31:31.2970641Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2024-12-18T01:31:31.2971910Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-12-18T01:31:31.2973125Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-12-18T01:31:31.2974478Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-12-18T01:31:31.2975217Z 2024-12-18T01:31:31.2975500Z Running distributed tests for the gloo backend with file init_method 2024-12-18T01:31:31.2976019Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:31:31.2977465Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=9', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:31:31.292299] 2024-12-18T01:36:11.7315014Z 2024-12-18T01:36:11.7316211Z distributed/test_distributed_spawn 9/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_9.9_9f536d088f285976_.log 2024-12-18T01:36:11.7331691Z Running 28 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-12-18T01:36:11.7347260Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2024-12-18T01:36:11.7348465Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2024-12-18T01:36:11.7349670Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2024-12-18T01:36:11.7350902Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2024-12-18T01:36:11.7352053Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2024-12-18T01:36:11.7353215Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2024-12-18T01:36:11.7354482Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2024-12-18T01:36:11.7355748Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2024-12-18T01:36:11.7356974Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2024-12-18T01:36:11.7358155Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2024-12-18T01:36:11.7359487Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2024-12-18T01:36:11.7360921Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2024-12-18T01:36:11.7362284Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2024-12-18T01:36:11.7363806Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2024-12-18T01:36:11.7365076Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2024-12-18T01:36:11.7366602Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2024-12-18T01:36:11.7367821Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2024-12-18T01:36:11.7369153Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2024-12-18T01:36:11.7370405Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2024-12-18T01:36:11.7371819Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2024-12-18T01:36:11.7373163Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2024-12-18T01:36:11.7374346Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2024-12-18T01:36:11.7375544Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2024-12-18T01:36:11.7376748Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2024-12-18T01:36:11.7377969Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2024-12-18T01:36:11.7379039Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-12-18T01:36:11.7380168Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-12-18T01:36:11.7381448Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-12-18T01:36:11.7382165Z 2024-12-18T01:36:11.7382563Z Running distributed/algorithms/quantization/test_quantization 1/1 ... [2024-12-18 01:36:11.732777] 2024-12-18T01:36:11.7383195Z /usr/bin/mpiexec 2024-12-18T01:36:11.7383540Z MPI not available -- MPI backend tests will be skipped 2024-12-18T01:36:11.7384087Z Running distributed tests for the test backend with env init_method 2024-12-18T01:36:11.7384570Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:36:11.7386060Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:36:11.734541] 2024-12-18T01:36:15.7115781Z 2024-12-18T01:36:15.7117164Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_5018b3eab9aa6c28_.log 2024-12-18T01:36:15.7118478Z Running 0 items in this shard: 2024-12-18T01:36:15.7118720Z 2024-12-18T01:36:15.7121504Z Running distributed tests for the test backend with file init_method 2024-12-18T01:36:15.7123323Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:36:15.7128109Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:36:15.712594] 2024-12-18T01:36:19.6730085Z 2024-12-18T01:36:19.6731544Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_30f683676ae9324d_.log 2024-12-18T01:36:19.6732859Z Running 0 items in this shard: 2024-12-18T01:36:19.6733087Z 2024-12-18T01:36:19.6739359Z Running distributed tests for the nccl backend with env init_method 2024-12-18T01:36:19.6741092Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:36:19.6745322Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:36:19.674334] 2024-12-18T01:37:15.9590226Z 2024-12-18T01:37:15.9591723Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_7fd958a77a9e3abe_.log 2024-12-18T01:37:15.9598035Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-12-18T01:37:15.9602308Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16 2024-12-18T01:37:15.9603916Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16 2024-12-18T01:37:15.9605302Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16 2024-12-18T01:37:15.9606679Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16 2024-12-18T01:37:15.9608098Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16 2024-12-18T01:37:15.9609551Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-12-18T01:37:15.9610334Z 2024-12-18T01:37:15.9610964Z Running distributed tests for the nccl backend with file init_method 2024-12-18T01:37:15.9611488Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:37:15.9613006Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:37:15.960152] 2024-12-18T01:38:12.3961671Z 2024-12-18T01:38:12.3963250Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_573abeb7eb33ff11_.log 2024-12-18T01:38:12.3968626Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-12-18T01:38:12.3972974Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16 2024-12-18T01:38:12.3974822Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16 2024-12-18T01:38:12.3976192Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16 2024-12-18T01:38:12.3977684Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16 2024-12-18T01:38:12.3979057Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16 2024-12-18T01:38:12.3980453Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-12-18T01:38:12.3981224Z 2024-12-18T01:38:12.3981487Z Running distributed tests for the gloo backend with env init_method 2024-12-18T01:38:12.3981997Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:38:12.3983459Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:38:12.397285] 2024-12-18T01:38:50.6801254Z 2024-12-18T01:38:50.6802698Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_1f41face016ee93a_.log 2024-12-18T01:38:50.6807793Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-12-18T01:38:50.6812417Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16 2024-12-18T01:38:50.6813823Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16 2024-12-18T01:38:50.6815199Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16 2024-12-18T01:38:50.6816660Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16 2024-12-18T01:38:50.6818032Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16 2024-12-18T01:38:50.6819431Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-12-18T01:38:50.6820190Z 2024-12-18T01:38:50.6820475Z Running distributed tests for the gloo backend with file init_method 2024-12-18T01:38:50.6820989Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:38:50.6822436Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:38:50.681251] 2024-12-18T01:39:28.9614567Z 2024-12-18T01:39:28.9616026Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_6e87de6eb9862dbf_.log 2024-12-18T01:39:28.9621389Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-12-18T01:39:28.9625590Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16 2024-12-18T01:39:28.9626954Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16 2024-12-18T01:39:28.9628298Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16 2024-12-18T01:39:28.9629635Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16 2024-12-18T01:39:28.9631002Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16 2024-12-18T01:39:28.9632403Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-12-18T01:39:28.9633162Z 2024-12-18T01:39:28.9633490Z Running distributed/elastic/test_control_plane 1/1 ... [2024-12-18 01:39:28.962643] 2024-12-18T01:39:28.9634105Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:39:28.9635388Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/test_control_plane.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:39:28.962995] 2024-12-18T01:39:39.4086408Z 2024-12-18T01:39:39.4088020Z distributed/elastic/test_control_plane 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.test_control_plane_1.1_4a50fa4d621f9879_.log 2024-12-18T01:39:39.4093320Z Running 9 items in this shard: test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_dump_nccl_trace_pickle, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_dump_nccl_trace_pickle_with_json, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_dump_nccl_trace_pickle_with_params, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_dump_traceback, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_get_handler_names, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_get_handler_nonexistant, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_run_handler, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_tcp, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_worker_server 2024-12-18T01:39:39.4097660Z 2024-12-18T01:39:39.4098014Z Running distributed/checkpoint/test_tp_checkpoint 1/1 ... [2024-12-18 01:39:39.408872] 2024-12-18T01:39:39.4098599Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:39:39.4099905Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_tp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:39:39.409247] 2024-12-18T01:39:57.2201242Z 2024-12-18T01:39:57.2203183Z distributed/checkpoint/test_tp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_tp_checkpoint_1.1_aabbd809296862fa_.log 2024-12-18T01:39:57.2205705Z Running 2 items in this shard: test/distributed/checkpoint/test_tp_checkpoint.py::TestTpCheckpoint::test_tp_checkpoint, test/distributed/checkpoint/test_tp_checkpoint.py::TestTpCheckpoint::test_tp_checkpoint_load_on_meta_device 2024-12-18T01:39:57.2206929Z 2024-12-18T01:39:57.2207351Z Running distributed/_composable/fsdp/test_fully_shard_overlap 1/1 ... [2024-12-18 01:39:57.219909] 2024-12-18T01:39:57.2208015Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:39:57.2209403Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_overlap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:39:57.220280] 2024-12-18T01:40:07.6607220Z 2024-12-18T01:40:07.6608623Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_overlap_1.1_e0a826f963696653_.log 2024-12-18T01:40:07.6611168Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_post_optim_event_overlap, test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_training_overlap 2024-12-18T01:40:07.6612614Z 2024-12-18T01:40:07.6614815Z Running distributed/checkpoint/test_planner 1/1 ... [2024-12-18 01:40:07.661019] 2024-12-18T01:40:07.6615435Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:40:07.6616732Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_planner.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:40:07.661384] 2024-12-18T01:40:12.3379587Z 2024-12-18T01:40:12.3380903Z distributed/checkpoint/test_planner 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_planner_1.1_ffc376c01e92aab4_.log 2024-12-18T01:40:12.3386155Z Running 8 items in this shard: test/distributed/checkpoint/test_planner.py::TestSavePlan::test_global_plan, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_load_with_resharding, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_load_with_world_size_diff_by_one, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_local_load_plan, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_local_plan, test/distributed/checkpoint/test_planner.py::TestPlannerHelpers::test_create_read_item_from_chunks, test/distributed/checkpoint/test_planner.py::TestLoadPlanner::test_load_different_sizes_throws, test/distributed/checkpoint/test_planner.py::TestLoadPlanner::test_strict 2024-12-18T01:40:12.3389921Z 2024-12-18T01:40:12.3390333Z Running distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 ... [2024-12-18 01:40:12.337865] 2024-12-18T01:40:12.3390993Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:40:12.3392344Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_tp_checkpoint_conversion.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:40:12.338252] 2024-12-18T01:40:23.4319463Z 2024-12-18T01:40:23.4321044Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion_1.1_09e73bdf9e27a5e2_.log 2024-12-18T01:40:23.4322892Z Running 1 items in this shard: test/distributed/checkpoint/test_fsdp_tp_checkpoint_conversion.py::TestFsdpTpCheckpointConversion::test_fsdp_to_tp 2024-12-18T01:40:23.4324415Z 2024-12-18T01:40:23.4324799Z Running distributed/checkpoint/test_state_dict_utils 1/1 ... [2024-12-18 01:40:23.431980] 2024-12-18T01:40:23.4325427Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:40:23.4326799Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_state_dict_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:40:23.432366] 2024-12-18T01:41:07.8646628Z 2024-12-18T01:41:07.8647928Z distributed/checkpoint/test_state_dict_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_state_dict_utils_1.1_f57de718209b4e31_.log 2024-12-18T01:41:07.8652325Z Running 6 items in this shard: test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_complicated_dict, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_cpu_and_ranks_only, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_create_cpu_state_dict, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_gather_state_dict_dtensor, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_gather_with_cpu_and_ranks_only, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_state_dict_util_distribute_tensors 2024-12-18T01:41:07.8655697Z 2024-12-18T01:41:07.8656516Z Running distributed/fsdp/test_shard_utils 1/1 ... [2024-12-18 01:41:07.864829] 2024-12-18T01:41:07.8657126Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:41:07.8658411Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_shard_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:41:07.865220] 2024-12-18T01:41:23.8200055Z 2024-12-18T01:41:23.8201246Z distributed/fsdp/test_shard_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_shard_utils_1.1_972544faf9186e1c_.log 2024-12-18T01:41:23.8203909Z Running 2 items in this shard: test/distributed/fsdp/test_shard_utils.py::TestShardUtilsDistributed::test_create_chunk_sharded_tensor, test/distributed/fsdp/test_shard_utils.py::TestShardUtilsDistributedDTensor::test_create_chunk_dtensor 2024-12-18T01:41:23.8205344Z 2024-12-18T01:41:23.8205704Z Running distributed/checkpoint/test_compatibility 1/1 ... [2024-12-18 01:41:23.820075] 2024-12-18T01:41:23.8206325Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:41:23.8207694Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_compatibility.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:41:23.820491] 2024-12-18T01:41:28.4470134Z 2024-12-18T01:41:28.4471412Z distributed/checkpoint/test_compatibility 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_compatibility_1.1_0cf0457c18f43cdd_.log 2024-12-18T01:41:28.4474515Z Running 4 items in this shard: test/distributed/checkpoint/test_compatibility.py::TestDCPCompatbility::test_metadata, test/distributed/checkpoint/test_compatibility.py::TestDCPCompatbility::test_sharded_tensor_dependency, test/distributed/checkpoint/test_compatibility.py::TestDCPCompatbility::test_storage_meta, test/distributed/checkpoint/test_compatibility.py::TestDCPCompatbility::test_with_v_2_3 2024-12-18T01:41:28.4476701Z 2024-12-18T01:41:28.4477064Z Running distributed/checkpoint/test_hsdp_checkpoint 1/1 ... [2024-12-18 01:41:28.447081] 2024-12-18T01:41:28.4477676Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:41:28.4479016Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_hsdp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:41:28.447461] 2024-12-18T01:42:00.8446277Z 2024-12-18T01:42:00.8450636Z distributed/checkpoint/test_hsdp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_hsdp_checkpoint_1.1_b1ab383ffea376f3_.log 2024-12-18T01:42:00.8454286Z Running 4 items in this shard: test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_checkpoint_is_even_sharded_model_False, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_checkpoint_is_even_sharded_model_True, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_fsdp_checkpoint_conversion_is_even_sharded_model_False, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_fsdp_checkpoint_conversion_is_even_sharded_model_True 2024-12-18T01:42:00.8456996Z 2024-12-18T01:42:00.8457238Z Running distributed/test_nccl 1/1 ... [2024-12-18 01:42:00.845092] 2024-12-18T01:42:00.8457737Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:42:00.8461819Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:42:00.845595] 2024-12-18T01:42:08.0783450Z 2024-12-18T01:42:08.0784524Z distributed/test_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_nccl_1.1_91c90a830ac96836_.log 2024-12-18T01:42:08.0789928Z Running 12 items in this shard: test/distributed/test_nccl.py::TestNCCLCUDA::test_all_gather_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_gather_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_reduce_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_reduce_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_broadcast_cuda_float8_e4m3fn, test/distributed/test_nccl.py::TestNCCLCUDA::test_broadcast_cuda_float8_e5m2, test/distributed/test_nccl.py::TestNCCLCUDA::test_collective_errors_cuda, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_scatter_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_scatter_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_unique_id_cuda 2024-12-18T01:42:08.0794326Z 2024-12-18T01:42:08.0794623Z Running distributed/test_functional_api 1/1 ... [2024-12-18 01:42:08.078297] 2024-12-18T01:42:08.0795168Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:42:08.0796425Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_functional_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:42:08.078668] 2024-12-18T01:45:01.2925484Z 2024-12-18T01:45:01.2926695Z distributed/test_functional_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_functional_api_1.1_df0011489f1d29aa_.log 2024-12-18T01:45:01.2938801Z Running 18 items in this shard: test/distributed/test_functional_api.py::TestMetaCollectives::test_all_reduce, test/distributed/test_functional_api.py::TestMakeFx::test_all_reduce_tracing, test/distributed/test_functional_api.py::TestCollectivesWithDistributedBackendCUDA::test_all_gather_into_tensor_coalesced_cuda, test/distributed/test_functional_api.py::TestCollectivesWithDistributedBackendCUDA::test_all_to_all_single_1d_input_cuda, test/distributed/test_functional_api.py::TestCollectivesWithDistributedBackendCUDA::test_all_to_all_single_cuda, test/distributed/test_functional_api.py::TestCollectivesWithDistributedBackendCUDA::test_all_to_all_single_split_sizes_none_cuda, test/distributed/test_functional_api.py::TestCollectivesWithDistributedBackendCUDA::test_tracing_cuda, test/distributed/test_functional_api.py::TestCollectivesWithDistributedBackendCUDA::test_tracing_with_dce_code_cuda, test/distributed/test_functional_api.py::TestCollectivesWithDistributedBackendCUDA::test_tracing_with_fakepg_cuda, test/distributed/test_functional_api.py::TestDistributedBackendCollectivesWithWorldSize4CUDA::test_all_gather_into_tensor_coalesced, test/distributed/test_functional_api.py::TestDistributedBackendCollectivesWithWorldSize4CUDA::test_all_to_all_single, test/distributed/test_functional_api.py::TestDistributedBackendCollectivesWithWorldSize4CUDA::test_all_to_all_single_1d_input, test/distributed/test_functional_api.py::TestDistributedBackendCollectivesWithWorldSize4CUDA::test_all_to_all_single_split_sizes_none, test/distributed/test_functional_api.py::TestDistributedBackendCollectivesWithWorldSize4CUDA::test_permute_tensor_with_sub_group_cuda, test/distributed/test_functional_api.py::TestDistributedBackendCollectivesWithWorldSize4CUDA::test_tracing, test/distributed/test_functional_api.py::TestDistributedBackendCollectivesWithWorldSize4CUDA::test_tracing_with_dce_code, test/distributed/test_functional_api.py::TestDistributedBackendCollectivesWithWorldSize4CUDA::test_tracing_with_fakepg, test/distributed/test_functional_api.py::TestFunctionalAutogradWithDistributedBackendCUDA::test_all_to_all_single_cuda 2024-12-18T01:45:01.2950413Z 2024-12-18T01:45:01.2950723Z Running distributed/_tensor/test_optimizers 1/1 ... [2024-12-18 01:45:01.292505] 2024-12-18T01:45:01.2951288Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:45:01.2952555Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_optimizers.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:45:01.292878] 2024-12-18T01:46:23.9186039Z 2024-12-18T01:46:23.9187220Z distributed/_tensor/test_optimizers 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_optimizers_1.1_5dbf1fcfcbcd64c3_.log 2024-12-18T01:46:23.9193962Z Running 11 items in this shard: test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_RMSprop_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adadelta_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adagrad_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adam_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adamax_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adamw_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_asgd_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_nadam_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_optimizer_foreach_supported_types_include_DTensor, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_radam_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_sgd_1d_sharding 2024-12-18T01:46:23.9199427Z 2024-12-18T01:46:23.9199837Z Running distributed/_composable/fsdp/test_fully_shard_grad_scaler 1/1 ... [2024-12-18 01:46:23.918740] 2024-12-18T01:46:23.9200501Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:46:23.9201877Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_grad_scaler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:46:23.919125] 2024-12-18T01:46:36.4667461Z 2024-12-18T01:46:36.4668875Z distributed/_composable/fsdp/test_fully_shard_grad_scaler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_grad_scaler_1.1_6e9a6577b23aced9_.log 2024-12-18T01:46:36.4671308Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_grad_scaler.py::TestFullyShardGradientScaler::test_gradient_scaler 2024-12-18T01:46:36.4672210Z 2024-12-18T01:46:36.4672541Z Running distributed/checkpoint/test_utils 1/1 ... [2024-12-18 01:46:36.466825] 2024-12-18T01:46:36.4673109Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:46:36.4674518Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:46:36.467233] 2024-12-18T01:46:40.7927096Z 2024-12-18T01:46:40.7928328Z distributed/checkpoint/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_utils_1.1_0a5f786566a15efa_.log 2024-12-18T01:46:40.7932078Z Running 6 items in this shard: test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_dcp_logger, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_flat_data, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_index_hint_ignored_on_equals, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_index_hint_ignored_on_hash, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_init_convert_offset, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_sharded_tensor_lookup 2024-12-18T01:46:40.7934881Z 2024-12-18T01:46:40.7935176Z Running distributed/_tensor/test_utils 1/1 ... [2024-12-18 01:46:40.793039] 2024-12-18T01:46:40.7935705Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:46:40.7936950Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:46:40.793433] 2024-12-18T01:47:50.3399661Z 2024-12-18T01:47:50.3401008Z distributed/_tensor/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_utils_1.1_a3d950cb9d128b6f_.log 2024-12-18T01:47:50.3407363Z Running 10 items in this shard: test/distributed/_tensor/test_utils.py::UtilTest::test_compute_local_shape_and_global_offset_1D, test/distributed/_tensor/test_utils.py::UtilTest::test_compute_local_shape_and_global_offset_2D, test/distributed/_tensor/test_utils.py::UtilTest::test_fsdp_tp_meta_compute, test/distributed/_tensor/test_utils.py::UtilTest::test_hsdp_tp_meta_compute, test/distributed/_tensor/test_utils.py::UtilTest::test_strided_sharding_assumption_in_meta_compute, test/distributed/_tensor/test_utils.py::TestStridedSharding::test_1d_mesh_strided_sharding, test/distributed/_tensor/test_utils.py::TestStridedSharding::test_2d_mesh_2d_tensor_strided_sharding, test/distributed/_tensor/test_utils.py::TestStridedSharding::test_2d_mesh_strided_sharding, test/distributed/_tensor/test_utils.py::Test2DStridedLocalShard::test_fsdp1_tp_2d_dtensor_local_shards_and_offsets, test/distributed/_tensor/test_utils.py::Test2DStridedLocalShard::test_fsdp2_tp_2d_dtensor_local_shards_and_offsets 2024-12-18T01:47:50.3412241Z 2024-12-18T01:47:50.3412522Z Running distributed/test_c10d_nccl 1/3 ... [2024-12-18 01:47:50.339912] 2024-12-18T01:47:50.3413028Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:47:50.3414307Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=1', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:47:50.340306] 2024-12-18T02:02:26.7576871Z 2024-12-18T02:02:26.7578104Z distributed/test_c10d_nccl 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_1.3_37b8a1b552bdd372_.log 2024-12-18T02:02:26.7618202Z Running 82 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_True, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e5m2, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_True, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_get_future_result, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_non_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group, test/distributed/test_c10d_nccl.py::CommTest::test_wait_tensor, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_False, test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale 2024-12-18T02:02:26.7657610Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors 2024-12-18T02:02:26.7658551Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus 2024-12-18T02:02:26.7659629Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs 2024-12-18T02:02:26.7660653Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg 2024-12-18T02:02:26.7661667Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered 2024-12-18T02:02:26.7662759Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_False 2024-12-18T02:02:26.7663846Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_True 2024-12-18T02:02:26.7664881Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group 2024-12-18T02:02:26.7666583Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl 2024-12-18T02:02:26.7667767Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check 2024-12-18T02:02:26.7668922Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout 2024-12-18T02:02:26.7670080Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16 2024-12-18T02:02:26.7671183Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e5m2 2024-12-18T02:02:26.7672335Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_True 2024-12-18T02:02:26.7673461Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init 2024-12-18T02:02:26.7674592Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module 2024-12-18T02:02:26.7675939Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view 2024-12-18T02:02:26.7677108Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output 2024-12-18T02:02:26.7678670Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2024-12-18T02:02:26.7679893Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2024-12-18T02:02:26.7681158Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False 2024-12-18T02:02:26.7682413Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl 2024-12-18T02:02:26.7683576Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params 2024-12-18T02:02:26.7684883Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters 2024-12-18T02:02:26.7686123Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view 2024-12-18T02:02:26.7687435Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info 2024-12-18T02:02:26.7688853Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off 2024-12-18T02:02:26.7690257Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail 2024-12-18T02:02:26.7691711Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off 2024-12-18T02:02:26.7692922Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16 2024-12-18T02:02:26.7694002Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl 2024-12-18T02:02:26.7695143Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view 2024-12-18T02:02:26.7696465Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess 2024-12-18T02:02:26.7697678Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module 2024-12-18T02:02:26.7698784Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module 2024-12-18T02:02:26.7699998Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None 2024-12-18T02:02:26.7701338Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids 2024-12-18T02:02:26.7702577Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason 2024-12-18T02:02:26.7703606Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad 2024-12-18T02:02:26.7704733Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error 2024-12-18T02:02:26.7705803Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl 2024-12-18T02:02:26.7706900Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object 2024-12-18T02:02:26.7707854Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast 2024-12-18T02:02:26.7708752Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq 2024-12-18T02:02:26.7709648Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp 2024-12-18T02:02:26.7710552Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_get_future_result 2024-12-18T02:02:26.7711534Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env 2024-12-18T02:02:26.7712595Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit 2024-12-18T02:02:26.7713635Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm 2024-12-18T02:02:26.7714683Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_non_blocking_wait_with_barrier 2024-12-18T02:02:26.7715709Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors 2024-12-18T02:02:26.7716623Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config 2024-12-18T02:02:26.7717496Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced 2024-12-18T02:02:26.7718479Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default 2024-12-18T02:02:26.7719633Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup 2024-12-18T02:02:26.7720621Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group 2024-12-18T02:02:26.7721505Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_wait_tensor 2024-12-18T02:02:26.7722532Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base 2024-12-18T02:02:26.7724092Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2 2024-12-18T02:02:26.7725610Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-12-18T02:02:26.7726982Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_True 2024-12-18T02:02:26.7728254Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_False 2024-12-18T02:02:26.7729597Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_True 2024-12-18T02:02:26.7730817Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_True 2024-12-18T02:02:26.7731897Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_False 2024-12-18T02:02:26.7733053Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_False 2024-12-18T02:02:26.7734210Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_True 2024-12-18T02:02:26.7735463Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_False 2024-12-18T02:02:26.7736954Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_False 2024-12-18T02:02:26.7738175Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_False 2024-12-18T02:02:26.7739351Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_False 2024-12-18T02:02:26.7740424Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata 2024-12-18T02:02:26.7741331Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe 2024-12-18T02:02:26.7742342Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True 2024-12-18T02:02:26.7743328Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long 2024-12-18T02:02:26.7744415Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False 2024-12-18T02:02:26.7745602Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False 2024-12-18T02:02:26.7746779Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True 2024-12-18T02:02:26.7748000Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False 2024-12-18T02:02:26.7749162Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True 2024-12-18T02:02:26.7750212Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired 2024-12-18T02:02:26.7751258Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale 2024-12-18T02:02:26.7751887Z 2024-12-18T02:02:26.7752165Z Running distributed/fsdp/test_fsdp_overlap 1/1 ... [2024-12-18 02:02:26.760131] 2024-12-18T02:02:26.7752658Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:02:26.7753825Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:02:26.760542] 2024-12-18T02:03:07.1218886Z 2024-12-18T02:03:07.1220138Z distributed/fsdp/test_fsdp_overlap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_overlap_1.1_d18592d87c2b6534_.log 2024-12-18T02:03:07.1222247Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_overlap.py::TestForwardOverlapWorldSizeTwo::test_forward_overlap, test/distributed/fsdp/test_fsdp_overlap.py::TestForwardOverlapWorldSizeOneCUDA::test_forward_overlap_cuda 2024-12-18T02:03:07.1223997Z 2024-12-18T02:03:07.1226292Z Running distributed/fsdp/test_fsdp_optim_state 1/1 ... [2024-12-18 02:03:07.122163] 2024-12-18T02:03:07.1227173Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:03:07.1228849Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_optim_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:03:07.122662] 2024-12-18T02:10:30.7318606Z 2024-12-18T02:10:30.7319810Z distributed/fsdp/test_fsdp_optim_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_optim_state_1.1_bd4971d31bd9df2c_.log 2024-12-18T02:10:30.7368022Z Running 60 items in this shard: test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_compatible_with_trec, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_flatten_sharded_optim_state_dict_nested, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_flatten_sharded_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_full_optim_state_dict_keys, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_full_optim_state_dict_nested_invalid, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_interface_arguments, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_no_grad, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_input_warning, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_without_param_groups, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type0_use_multiple_param_groups_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type0_use_multiple_param_groups_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type1_use_multiple_param_groups_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type1_use_multiple_param_groups_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_names, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_save_load_without_0th_param_state_state_dict_type0, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_save_load_without_0th_param_state_state_dict_type1, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_halve_world_size, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_halve_world_size, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type0_add_to_fsdp_module_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type0_add_to_fsdp_module_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type1_add_to_fsdp_module_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type1_add_to_fsdp_module_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_state_dict_with_none_tensor_state, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_use_orig_params, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_with_empty_optimizer_state, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_with_no_shard 2024-12-18T02:10:30.7411263Z 2024-12-18T02:10:30.7411659Z Running distributed/checkpoint/test_file_system_checkpoint 1/1 ... [2024-12-18 02:10:30.734835] 2024-12-18T02:10:30.7412320Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:10:30.7413729Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:10:30.735196] 2024-12-18T02:11:03.5325273Z 2024-12-18T02:11:03.5326731Z distributed/checkpoint/test_file_system_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_1.1_cca3bc9fbdea75c5_.log 2024-12-18T02:11:03.5331939Z Running 6 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_save_load_bytes, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor 2024-12-18T02:11:03.5335958Z 2024-12-18T02:11:03.5336254Z Running distributed/fsdp/test_fsdp_apply 1/1 ... [2024-12-18 02:11:03.532609] 2024-12-18T02:11:03.5336809Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:11:03.5338137Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:11:03.532998] 2024-12-18T02:11:31.4677444Z 2024-12-18T02:11:31.4678591Z distributed/fsdp/test_fsdp_apply 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_apply_1.1_e1118e58e6b9e6c7_.log 2024-12-18T02:11:31.4680896Z Running 3 items in this shard: test/distributed/fsdp/test_fsdp_apply.py::TestApplyCUDA::test_apply_in_summon_raises_error_cuda, test/distributed/fsdp/test_fsdp_apply.py::TestApplyCUDA::test_nested_module_apply_cuda, test/distributed/fsdp/test_fsdp_apply.py::TestApplyCUDA::test_transformer_module_apply_cuda 2024-12-18T02:11:31.4682486Z 2024-12-18T02:11:31.4683098Z Running distributed/fsdp/test_fsdp_flatten_params 1/1 ... [2024-12-18 02:11:31.467768] 2024-12-18T02:11:31.4684019Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:11:31.4685357Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_flatten_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:11:31.468140] 2024-12-18T02:12:41.7597257Z 2024-12-18T02:12:41.7598247Z distributed/fsdp/test_fsdp_flatten_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_flatten_params_1.1_1ff9b083961fc955_.log 2024-12-18T02:12:41.7606773Z Running 13 items in this shard: test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_empty_module, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_aligned_full_precision, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_aligned_mixed_precision, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_unaligned, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_with_memory_format_memory_format0, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_with_memory_format_memory_format1, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flatten_nothing, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_numel_with_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_numel_without_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_output_with_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_output_without_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_partial_flattening, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_pnorm_after_step_with_shared_params 2024-12-18T02:12:41.7614461Z 2024-12-18T02:12:41.7614835Z Running distributed/optim/test_zero_redundancy_optimizer 1/1 ... [2024-12-18 02:12:41.759764] 2024-12-18T02:12:41.7615434Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:12:41.7616736Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_zero_redundancy_optimizer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:12:41.760157] 2024-12-18T02:17:22.6992377Z 2024-12-18T02:17:22.6993876Z distributed/optim/test_zero_redundancy_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_zero_redundancy_optimizer_1.1_fe9285a434702bb0_.log 2024-12-18T02:17:22.7031045Z Running 42 items in this shard: test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_constructor, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_lr_scheduler, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_same_dense_param_type, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_state_dict, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_with_extra_inner_key, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_with_kwargs, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_without_closure, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_zero_grad, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_add_param_group, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_collect_shards, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_lr_scheduler, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_multiple_param_groups, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_nondefault_process_group, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_sharding, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_step, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_step_with_closure, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_join_cpu, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_join_gpu, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_model_parallel_parameters_as_bucket_view_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_model_parallel_parameters_as_bucket_view_True 2024-12-18T02:17:22.7067645Z 2024-12-18T02:17:22.7068066Z Running distributed/_composable/fsdp/test_fully_shard_memory 1/1 ... [2024-12-18 02:17:22.699885] 2024-12-18T02:17:22.7068731Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:17:22.7070138Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:17:22.700263] 2024-12-18T02:17:40.0598619Z 2024-12-18T02:17:40.0600015Z distributed/_composable/fsdp/test_fully_shard_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_memory_1.1_922270446896e03e_.log 2024-12-18T02:17:40.0602321Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_memory.py::TestFullyShardMemory::test_fully_shard_del_memory, test/distributed/_composable/fsdp/test_fully_shard_memory.py::TestFullyShardMemory::test_fully_shard_training_memory 2024-12-18T02:17:40.0604085Z 2024-12-18T02:17:40.0604449Z Running distributed/fsdp/test_checkpoint_wrapper 1/1 ... [2024-12-18 02:17:40.059820] 2024-12-18T02:17:40.0605047Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:17:40.0606397Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:17:40.060263] 2024-12-18T02:17:44.9876223Z 2024-12-18T02:17:44.9877514Z distributed/fsdp/test_checkpoint_wrapper 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_checkpoint_wrapper_1.1_af9752128a8dbd8e_.log 2024-12-18T02:17:44.9883213Z Running 8 items in this shard: test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_apply_activation_checkpointing, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_args_kwargs, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_cpu_offload, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_kwarg_support, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_parity, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_forward_missing_attributes, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_fqn, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_load_activation_checkpointed_module 2024-12-18T02:17:44.9888195Z 2024-12-18T02:17:44.9888487Z Running distributed/fsdp/test_utils 1/1 ... [2024-12-18 02:17:44.987552] 2024-12-18T02:17:44.9889034Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:17:44.9890318Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:17:44.987927] 2024-12-18T02:17:50.5166913Z 2024-12-18T02:17:50.5168000Z distributed/fsdp/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_utils_1.1_0a2eb8c3934cec37_.log 2024-12-18T02:17:50.5171162Z Running 5 items in this shard: test/distributed/fsdp/test_utils.py::TestUtilsCUDA::test_apply_to_tensors_cpu_cuda_cuda, test/distributed/fsdp/test_utils.py::TestUtilsCUDA::test_apply_to_tensors_device_list0_cuda, test/distributed/fsdp/test_utils.py::TestUtilsCUDA::test_apply_to_tensors_device_list1_cuda, test/distributed/fsdp/test_utils.py::TestUtilsCUDA::test_packed_sequence_cuda, test/distributed/fsdp/test_utils.py::TestUtilsCUDA::test_replace_by_prefix_cuda 2024-12-18T02:17:50.5173768Z 2024-12-18T02:17:50.5174122Z Running distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 ... [2024-12-18 02:17:50.516782] 2024-12-18T02:17:50.5174771Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:17:50.5176080Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_hsdp_dtensor_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:17:50.517174] 2024-12-18T02:19:23.8688213Z 2024-12-18T02:19:23.8689532Z distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_hsdp_dtensor_state_dict_1.1_c36bcfd4e91ef5bf_.log 2024-12-18T02:19:23.8696708Z Running 8 items in this shard: test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_False_cuda, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_True_cuda, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_False_cuda, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_True_cuda, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_False_cuda, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_True_cuda, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensorCUDA::test_hsdp_init_with_device_mesh_cuda, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensorCUDA::test_root_module_is_not_FSDP_cuda 2024-12-18T02:19:23.8703139Z 2024-12-18T02:19:23.8703783Z Running distributed/fsdp/test_fsdp_tp_integration 1/1 ... [2024-12-18 02:19:23.869282] 2024-12-18T02:19:23.8704387Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:19:23.8705681Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:19:23.869651] 2024-12-18T02:19:54.8667251Z 2024-12-18T02:19:54.8668540Z distributed/fsdp/test_fsdp_tp_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_c6e3b3eb4f45cc2d_.log 2024-12-18T02:19:54.8673446Z Running 3 items in this shard: test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_extension_grad, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_sync_module_state 2024-12-18T02:19:54.8675744Z 2024-12-18T02:19:54.8676053Z Running distributed/fsdp/test_fsdp_checkpoint 1/1 ... [2024-12-18 02:19:54.866801] 2024-12-18T02:19:54.8676632Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:19:54.8677921Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:19:54.867246] 2024-12-18T02:22:11.4836738Z 2024-12-18T02:22:11.4841083Z distributed/fsdp/test_fsdp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_checkpoint_1.1_b376c86cc359d178_.log 2024-12-18T02:22:11.4855535Z Running 17 items in this shard: test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpointSubmoduleCUDA::test_checkpoint_submodule_use_reentrant_False_cuda 2024-12-18T02:22:11.4868657Z 2024-12-18T02:22:11.4868968Z Running distributed/algorithms/test_join 1/1 ... [2024-12-18 02:22:11.483670] 2024-12-18T02:22:11.4869537Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:22:11.4870811Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/test_join.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:22:11.484051] 2024-12-18T02:23:16.9681856Z 2024-12-18T02:23:16.9683176Z distributed/algorithms/test_join 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.test_join_1.1_0f741634d511e795_.log 2024-12-18T02:23:16.9687924Z Running 9 items in this shard: test/distributed/algorithms/test_join.py::TestJoin::test_join_kwargs, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinable_disable, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinables, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinables_throw, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_disable, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_main_hooks, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_post_hooks, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_throw 2024-12-18T02:23:16.9692116Z 2024-12-18T02:23:16.9692444Z Running distributed/fsdp/test_fsdp_hybrid_shard 1/1 ... [2024-12-18 02:23:16.968254] 2024-12-18T02:23:16.9693022Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:23:16.9694311Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_hybrid_shard.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:23:16.968662] 2024-12-18T02:24:20.8045891Z 2024-12-18T02:24:20.8047178Z distributed/fsdp/test_fsdp_hybrid_shard 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_hybrid_shard_1.1_3ce571e7b20dadee_.log 2024-12-18T02:24:20.8052674Z Running 6 items in this shard: test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_fsdp_hybrid_shard_basic_setup, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_fsdp_hybrid_shard_parity, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_hsdp_save_load_state_dict, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_hsdp_sync_module_state, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_invalid_pg_specification_raises, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_raises_manual_wrap_hybrid_shard_when_none_policy 2024-12-18T02:24:20.8055952Z 2024-12-18T02:24:20.8056366Z Running distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 ... [2024-12-18 02:24:20.804653] 2024-12-18T02:24:20.8057026Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:24:20.8058391Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint_cpu.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:24:20.805039] 2024-12-18T02:25:12.7524614Z 2024-12-18T02:25:12.7526438Z distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_cpu_1.1_1485b7711dd75d67_.log 2024-12-18T02:25:12.7536381Z Running 12 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_save_load_bytes_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_save_load_bytes_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor_thread_count_2 2024-12-18T02:25:12.7545406Z 2024-12-18T02:25:12.7545687Z Running distributed/test_data_parallel 1/1 ... [2024-12-18 02:25:12.752748] 2024-12-18T02:25:12.7546204Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:25:12.7547415Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_data_parallel.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:25:12.753132] 2024-12-18T02:25:21.7905747Z 2024-12-18T02:25:21.7906687Z distributed/test_data_parallel 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_data_parallel_1.1_8857beeb3b02e639_.log 2024-12-18T02:25:21.7930293Z Running 46 items in this shard: test/distributed/test_data_parallel.py::TestDataParallel::test_autocast, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_buffers_requiring_grad, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_complex, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_device_args, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_function_deletion, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_lazy_linear, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_model_device, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_model_no_refcycles, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_module_zero_inputs, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_multiple_input, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_nested_input, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_nested_output, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_no_grad, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_rnn, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_small_back, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_sparse, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_cpu, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_different_len_dicts, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_gpu, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply_autocast, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply_passes_exception, test/distributed/test_data_parallel.py::TestDataParallel::test_parameter_list_dict_replica, test/distributed/test_data_parallel.py::TestDataParallel::test_replicate, test/distributed/test_data_parallel.py::TestDataParallel::test_replicate_buffers, test/distributed/test_data_parallel.py::TestDataParallel::test_save_replica_module, test/distributed/test_data_parallel.py::TestDataParallel::test_scatter_cpu, test/distributed/test_data_parallel.py::TestDataParallel::test_scatter_gpu, test/distributed/test_data_parallel.py::TestDataParallel::test_strided_grad_layout, test/distributed/test_data_parallel.py::TestDataParallel::test_zero_grad, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float64 2024-12-18T02:25:21.7951889Z 2024-12-18T02:25:21.7952165Z Running distributed/fsdp/test_fsdp_core 1/2 ... [2024-12-18 02:25:21.790678] 2024-12-18T02:25:21.7952695Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:25:21.7953909Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '--shard-id=1', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:25:21.791047] 2024-12-18T02:37:43.0867032Z 2024-12-18T02:37:43.0868578Z distributed/fsdp/test_fsdp_core 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_core_1.2_c7d701e3e9880e09_.log 2024-12-18T02:37:43.0890255Z Running 33 items in this shard: test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_cuda_first_True_cuda, test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_False_mixed_precision_False_cuda, test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_False_mixed_precision_True_cuda, test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_True_mixed_precision_False_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestNoGradCUDA::test_transformer_no_grad_mixed_precision_True_cuda 2024-12-18T02:37:43.0909949Z 2024-12-18T02:37:43.0910241Z Running distributed/rpc/test_faulty_agent 1/1 ... [2024-12-18 02:37:43.087735] 2024-12-18T02:37:43.0910794Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:37:43.0912126Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/rpc/test_faulty_agent.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:37:43.088104] 2024-12-18T02:37:47.3963043Z 2024-12-18T02:37:47.3964524Z distributed/rpc/test_faulty_agent 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.test_faulty_agent_1.1_07ed557f81a14759_.log 2024-12-18T02:37:47.3965938Z Running 0 items in this shard: 2024-12-18T02:37:47.3966182Z 2024-12-18T02:37:47.3966544Z Running distributed/fsdp/test_fsdp_fine_tune 1/1 ... [2024-12-18 02:37:47.396365] 2024-12-18T02:37:47.3967125Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:37:47.3969857Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fine_tune.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:37:47.396787] 2024-12-18T02:38:31.6713260Z 2024-12-18T02:38:31.6719745Z distributed/fsdp/test_fsdp_fine_tune 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_fine_tune_1.1_8acaab5825bd30d4_.log 2024-12-18T02:38:31.6723095Z Running 4 items in this shard: test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTuneCUDA::test_backward_reshard_hooks_cuda, test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTuneCUDA::test_hooks_multi_traversal_cuda, test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTuneCUDA::test_parity_with_ddp_cuda, test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTuneCUDA::test_parity_with_non_frozen_fsdp_cuda 2024-12-18T02:38:31.6725468Z 2024-12-18T02:38:31.6725848Z Running distributed/_shard/sharded_optim/test_sharded_optim 1/1 ... [2024-12-18 02:38:31.671227] 2024-12-18T02:38:31.6726500Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:38:31.6727890Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:38:31.671649] 2024-12-18T02:38:49.6841866Z 2024-12-18T02:38:49.6843318Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_optim.test_sharded_optim_1.1_970bc5344113e429_.log 2024-12-18T02:38:49.6845930Z Running 2 items in this shard: test/distributed/_shard/sharded_optim/test_sharded_optim.py::TestShardedOptimizer::test_named_params_with_sharded_tensor, test/distributed/_shard/sharded_optim/test_sharded_optim.py::TestShardedOptimizer::test_sharded_optim 2024-12-18T02:38:49.6847269Z 2024-12-18T02:38:49.6847611Z Running distributed/fsdp/test_fsdp_unshard_params 1/1 ... [2024-12-18 02:38:49.684188] 2024-12-18T02:38:49.6848211Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:38:49.6849669Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_unshard_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:38:49.684616] 2024-12-18T02:40:20.2337238Z 2024-12-18T02:40:20.2338567Z distributed/fsdp/test_fsdp_unshard_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_unshard_params_1.1_f5b3e12ffc11afdc_.log 2024-12-18T02:40:20.2347762Z Running 15 items in this shard: test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_named_parameters_and_buffers, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_param_data, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_recurse, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_respects_reshard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_writeback, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_singleton_param_writeback, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_submodule, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_with_grads_core, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_with_grads_none_grads, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsNoShard::test_unshard_params_param_data_no_shard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsNoShard::test_unshard_params_writeback_no_shard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_offload_to_cpu_no_shard_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_rank0_only_with_writeback_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_unshard_params_from_backward_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_unshard_params_from_forward_raises 2024-12-18T02:40:20.2356181Z 2024-12-18T02:40:20.2356509Z Running distributed/fsdp/test_fsdp_mixed_precision 1/1 ... [2024-12-18 02:40:20.233783] 2024-12-18T02:40:20.2357087Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:40:20.2358362Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:40:20.234182] 2024-12-18T02:46:43.9726614Z 2024-12-18T02:46:43.9728113Z distributed/fsdp/test_fsdp_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_mixed_precision_1.1_970b5c2f2308582e_.log 2024-12-18T02:46:43.9779396Z Running 65 items in this shard: test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_eval_root_cast_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval_buffers, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval_comm, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_grads_reduced_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_input_grads_with_param_mixed_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_no_reshard_after_forward, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_resnet, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_batchnorm_convert_sync_bn_False, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_batchnorm_convert_sync_bn_True, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_default, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_only_params_and_bufs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_params_and_reduce_diff, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_reduce, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_grads_reduced_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_mixed_precision_e2e_full_shard, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_mixed_precision_no_reshard_after_forward, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionIgnoredModules::test_mixed_precision_with_ignored_module, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule_skip_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule_skip_inputs_error, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_different_precisions, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_different_precisions_error, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_external_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPTrainEval::test_train_ema_eval_flow 2024-12-18T02:46:43.9825588Z 2024-12-18T02:46:43.9825855Z Running distributed/test_c10d_spawn_gloo 1/1 ... [2024-12-18 02:46:43.973120] 2024-12-18T02:46:43.9826356Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:46:43.9827561Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:46:43.973489] 2024-12-18T02:48:19.3022437Z 2024-12-18T02:48:19.3024187Z distributed/test_c10d_spawn_gloo 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_gloo_1.1_3dd7b9dcc81c070e_.log 2024-12-18T02:48:19.3030461Z Running 11 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cpu, test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cuda, test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_rnn, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all_single, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_allreduce, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_broadcast, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_gather, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_reduce, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_scatter 2024-12-18T02:48:19.3036673Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cpu 2024-12-18T02:48:19.3037831Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cuda 2024-12-18T02:48:19.3038976Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_rnn 2024-12-18T02:48:19.3040086Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather 2024-12-18T02:48:19.3041157Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all 2024-12-18T02:48:19.3042279Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all_single 2024-12-18T02:48:19.3043525Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_allreduce 2024-12-18T02:48:19.3044794Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_broadcast 2024-12-18T02:48:19.3045892Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_gather 2024-12-18T02:48:19.3046964Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_reduce 2024-12-18T02:48:19.3048037Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_scatter 2024-12-18T02:48:19.3048644Z 2024-12-18T02:48:19.3048935Z Running distributed/test_c10d_spawn_ucc 1/1 ... [2024-12-18 02:48:19.302327] 2024-12-18T02:48:19.3049467Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:48:19.3050815Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_spawn_ucc.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:48:19.302702] 2024-12-18T02:49:18.7546478Z 2024-12-18T02:49:18.7547947Z distributed/test_c10d_spawn_ucc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_ucc_1.1_24352c650089b7ec_.log 2024-12-18T02:49:18.7551601Z Running 6 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_gather, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all_single, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_allreduce, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_broadcast, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_reduce 2024-12-18T02:49:18.7554884Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_gather 2024-12-18T02:49:18.7555955Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all 2024-12-18T02:49:18.7557053Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all_single 2024-12-18T02:49:18.7558135Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_allreduce 2024-12-18T02:49:18.7559187Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_broadcast 2024-12-18T02:49:18.7560226Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_reduce 2024-12-18T02:49:18.7560946Z 2024-12-18T02:49:18.7561232Z Running distributed/test_c10d_spawn_nccl 1/1 ... [2024-12-18 02:49:18.754681] 2024-12-18T02:49:18.7561771Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:49:18.7563073Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_spawn_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:49:18.755050] 2024-12-18T02:50:54.6294978Z 2024-12-18T02:50:54.6296247Z distributed/test_c10d_spawn_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_nccl_1.1_6e28777d7d990ce1_.log 2024-12-18T02:50:54.6301311Z Running 9 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather_base, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all_single, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_allreduce, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_broadcast, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter_non_contiguous 2024-12-18T02:50:54.6306035Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather 2024-12-18T02:50:54.6307154Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather_base 2024-12-18T02:50:54.6308272Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all 2024-12-18T02:50:54.6309394Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all_single 2024-12-18T02:50:54.6310832Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_allreduce 2024-12-18T02:50:54.6311945Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_broadcast 2024-12-18T02:50:54.6313012Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce 2024-12-18T02:50:54.6314093Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter 2024-12-18T02:50:54.6315289Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter_non_contiguous 2024-12-18T02:50:54.6316020Z 2024-12-18T02:50:54.6316254Z Running distributed/test_store 1/1 ... [2024-12-18 02:50:54.629954] 2024-12-18T02:50:54.6316750Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:50:54.6318028Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_store.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:50:54.630315] 2024-12-18T02:57:45.8189358Z 2024-12-18T02:57:45.8190383Z distributed/test_store 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_store_1.1_20ade82a32f16def_.log 2024-12-18T02:57:45.8222617Z Running 88 items in this shard: test/distributed/test_store.py::FileStoreTest::test_append, test/distributed/test_store.py::FileStoreTest::test_compare_set, test/distributed/test_store.py::FileStoreTest::test_init_pg_and_rpc_with_same_file, test/distributed/test_store.py::FileStoreTest::test_multi_get, test/distributed/test_store.py::FileStoreTest::test_multi_set, test/distributed/test_store.py::FileStoreTest::test_refcount, test/distributed/test_store.py::FileStoreTest::test_set_get_check, test/distributed/test_store.py::FileStoreTest::test_simple_wait, test/distributed/test_store.py::HashStoreTest::test_append, test/distributed/test_store.py::HashStoreTest::test_compare_set, test/distributed/test_store.py::HashStoreTest::test_multi_get, test/distributed/test_store.py::HashStoreTest::test_multi_set, test/distributed/test_store.py::HashStoreTest::test_set_get_check, test/distributed/test_store.py::HashStoreTest::test_simple_wait, test/distributed/test_store.py::PrefixStoreTest::test_get_underlying_store, test/distributed/test_store.py::PrefixFileStoreTest::test_append, test/distributed/test_store.py::PrefixFileStoreTest::test_compare_set, test/distributed/test_store.py::PrefixFileStoreTest::test_multi_get, test/distributed/test_store.py::PrefixFileStoreTest::test_multi_set, test/distributed/test_store.py::PrefixFileStoreTest::test_set_get_check, test/distributed/test_store.py::PrefixFileStoreTest::test_simple_wait, test/distributed/test_store.py::TCPStoreTest::test_address_already_in_use, test/distributed/test_store.py::TCPStoreTest::test_append, test/distributed/test_store.py::TCPStoreTest::test_compare_set, test/distributed/test_store.py::TCPStoreTest::test_init_pg_and_rpc_with_same_socket, test/distributed/test_store.py::TCPStoreTest::test_multi_get, test/distributed/test_store.py::TCPStoreTest::test_multi_set, test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_fixed_world_size, test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_nonfixed_world_size, test/distributed/test_store.py::TCPStoreTest::test_multitenancy, test/distributed/test_store.py::TCPStoreTest::test_numkeys_delkeys, test/distributed/test_store.py::TCPStoreTest::test_repr, test/distributed/test_store.py::TCPStoreTest::test_set_get_check, test/distributed/test_store.py::TCPStoreTest::test_simple_wait, test/distributed/test_store.py::TCPStoreTest::test_store_timeout_on_missing_clients, test/distributed/test_store.py::TCPStoreTest::test_take_over_listen_socket, test/distributed/test_store.py::TCPStoreTest::test_world_size_0_raises, test/distributed/test_store.py::LibUvTCPStoreTest::test_address_already_in_use, test/distributed/test_store.py::LibUvTCPStoreTest::test_append, test/distributed/test_store.py::LibUvTCPStoreTest::test_compare_set, test/distributed/test_store.py::LibUvTCPStoreTest::test_init_pg_and_rpc_with_same_socket, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_get, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_set, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_fixed_world_size, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_nonfixed_world_size, test/distributed/test_store.py::LibUvTCPStoreTest::test_multitenancy, test/distributed/test_store.py::LibUvTCPStoreTest::test_numkeys_delkeys, test/distributed/test_store.py::LibUvTCPStoreTest::test_repr, test/distributed/test_store.py::LibUvTCPStoreTest::test_set_get_check, test/distributed/test_store.py::LibUvTCPStoreTest::test_simple_wait, test/distributed/test_store.py::LibUvTCPStoreTest::test_store_timeout_on_missing_clients, test/distributed/test_store.py::LibUvTCPStoreTest::test_take_over_listen_socket, test/distributed/test_store.py::LibUvTCPStoreTest::test_world_size_0_raises, test/distributed/test_store.py::PrefixTCPStoreTest::test_append, test/distributed/test_store.py::PrefixTCPStoreTest::test_compare_set, test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_get, test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_set, test/distributed/test_store.py::PrefixTCPStoreTest::test_set_get_check, test/distributed/test_store.py::PrefixTCPStoreTest::test_simple_wait, test/distributed/test_store.py::PrefixTCPStoreTest::test_underlying_non_prefix_store, test/distributed/test_store.py::PythonStoreTest::test_set_get, test/distributed/test_store.py::RendezvousTest::test_unknown_handler, test/distributed/test_store.py::RendezvousTest::test_url_with_node_params, test/distributed/test_store.py::RendezvousEnvTest::test_nominal, test/distributed/test_store.py::RendezvousFileTest::test_common_errors, test/distributed/test_store.py::RendezvousFileTest::test_nominal, test/distributed/test_store.py::RendezvousTCPTest::test_common_errors, test/distributed/test_store.py::RendezvousTCPTest::test_dns_timeout, test/distributed/test_store.py::RendezvousTCPTest::test_nominal, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_doest_break_client, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_set, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_url_with_libuv, test/distributed/test_store.py::TestPythonStore::test_append_roundtrip, test/distributed/test_store.py::TestPythonStore::test_extended_methods_fallbacks, test/distributed/test_store.py::TestPythonStore::test_has_extended_api_passthrough, test/distributed/test_store.py::TestPythonStore::test_has_extended_api_roundtrip, test/distributed/test_store.py::TestPythonStore::test_multi_get_roundtrip, test/distributed/test_store.py::TestPythonStore::test_multi_set_roundtrip, test/distributed/test_store.py::TestPythonStore::test_optional_methods_fail, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_file_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_hash_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_prefix_file_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store_uv, test/distributed/test_store.py::TimeoutTest::test_interrupt_doesnt_break_wait, test/distributed/test_store.py::InitPgWithNonUvStore::test_with_env_var, test/distributed/test_store.py::InitPgWithNonUvStore::test_with_url_param, test/distributed/test_store.py::TestClientProtocol::test_client_connect 2024-12-18T02:57:45.8253889Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_append 2024-12-18T02:57:45.8254740Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_compare_set 2024-12-18T02:57:45.8255903Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_init_pg_and_rpc_with_same_file 2024-12-18T02:57:45.8256865Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_multi_get 2024-12-18T02:57:45.8257670Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_multi_set 2024-12-18T02:57:45.8258461Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_refcount 2024-12-18T02:57:45.8259262Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_set_get_check 2024-12-18T02:57:45.8260093Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_simple_wait 2024-12-18T02:57:45.8260893Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_append 2024-12-18T02:57:45.8261688Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_compare_set 2024-12-18T02:57:45.8262491Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_multi_get 2024-12-18T02:57:45.8263288Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_multi_set 2024-12-18T02:57:45.8264199Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_set_get_check 2024-12-18T02:57:45.8264963Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_simple_wait 2024-12-18T02:57:45.8266340Z Running 1 items in this shard: test/distributed/test_store.py::PrefixStoreTest::test_get_underlying_store 2024-12-18T02:57:45.8267274Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_append 2024-12-18T02:57:45.8268188Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_compare_set 2024-12-18T02:57:45.8269243Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_multi_get 2024-12-18T02:57:45.8270157Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_multi_set 2024-12-18T02:57:45.8271078Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_set_get_check 2024-12-18T02:57:45.8272023Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_simple_wait 2024-12-18T02:57:45.8272969Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_address_already_in_use 2024-12-18T02:57:45.8273854Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_append 2024-12-18T02:57:45.8274691Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_compare_set 2024-12-18T02:57:45.8275642Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_init_pg_and_rpc_with_same_socket 2024-12-18T02:57:45.8276568Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_get 2024-12-18T02:57:45.8277399Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_set 2024-12-18T02:57:45.8278561Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_fixed_world_size 2024-12-18T02:57:45.8279579Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_nonfixed_world_size 2024-12-18T02:57:45.8280501Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multitenancy 2024-12-18T02:57:45.8281330Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_numkeys_delkeys 2024-12-18T02:57:45.8282113Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_repr 2024-12-18T02:57:45.8282880Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_set_get_check 2024-12-18T02:57:45.8283951Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_simple_wait 2024-12-18T02:57:45.8284904Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_store_timeout_on_missing_clients 2024-12-18T02:57:45.8287502Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_take_over_listen_socket 2024-12-18T02:57:45.8288517Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_world_size_0_raises 2024-12-18T02:57:45.8289490Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_address_already_in_use 2024-12-18T02:57:45.8290413Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_append 2024-12-18T02:57:45.8291307Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_compare_set 2024-12-18T02:57:45.8292324Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_init_pg_and_rpc_with_same_socket 2024-12-18T02:57:45.8293324Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_get 2024-12-18T02:57:45.8294208Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_set 2024-12-18T02:57:45.8295218Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_fixed_world_size 2024-12-18T02:57:45.8296441Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_nonfixed_world_size 2024-12-18T02:57:45.8297407Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multitenancy 2024-12-18T02:57:45.8298289Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_numkeys_delkeys 2024-12-18T02:57:45.8299128Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_repr 2024-12-18T02:57:45.8300040Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_set_get_check 2024-12-18T02:57:45.8300907Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_simple_wait 2024-12-18T02:57:45.8301984Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_store_timeout_on_missing_clients 2024-12-18T02:57:45.8302930Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_take_over_listen_socket 2024-12-18T02:57:45.8303800Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_world_size_0_raises 2024-12-18T02:57:45.8304621Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_append 2024-12-18T02:57:45.8305422Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_compare_set 2024-12-18T02:57:45.8306234Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_get 2024-12-18T02:57:45.8307037Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_set 2024-12-18T02:57:45.8307857Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_set_get_check 2024-12-18T02:57:45.8308677Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_simple_wait 2024-12-18T02:57:45.8309565Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_underlying_non_prefix_store 2024-12-18T02:57:45.8310419Z Running 1 items in this shard: test/distributed/test_store.py::PythonStoreTest::test_set_get 2024-12-18T02:57:45.8311202Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTest::test_unknown_handler 2024-12-18T02:57:45.8312030Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTest::test_url_with_node_params 2024-12-18T02:57:45.8312842Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousEnvTest::test_nominal 2024-12-18T02:57:45.8313650Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousFileTest::test_common_errors 2024-12-18T02:57:45.8314444Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousFileTest::test_nominal 2024-12-18T02:57:45.8315334Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_common_errors 2024-12-18T02:57:45.8316161Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_dns_timeout 2024-12-18T02:57:45.8316954Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_nominal 2024-12-18T02:57:45.8317849Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_doest_break_client 2024-12-18T02:57:45.8318808Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_set 2024-12-18T02:57:45.8319699Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_url_with_libuv 2024-12-18T02:57:45.8320574Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_append_roundtrip 2024-12-18T02:57:45.8321455Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_extended_methods_fallbacks 2024-12-18T02:57:45.8322373Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_has_extended_api_passthrough 2024-12-18T02:57:45.8323378Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_has_extended_api_roundtrip 2024-12-18T02:57:45.8324514Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_multi_get_roundtrip 2024-12-18T02:57:45.8325472Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_multi_set_roundtrip 2024-12-18T02:57:45.8326428Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_optional_methods_fail 2024-12-18T02:57:45.8327474Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_file_store 2024-12-18T02:57:45.8328452Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_hash_store 2024-12-18T02:57:45.8329461Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_prefix_file_store 2024-12-18T02:57:45.8330468Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store 2024-12-18T02:57:45.8331462Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store_uv 2024-12-18T02:57:45.8332461Z Running 1 items in this shard: test/distributed/test_store.py::TimeoutTest::test_interrupt_doesnt_break_wait 2024-12-18T02:57:45.8333421Z Running 1 items in this shard: test/distributed/test_store.py::InitPgWithNonUvStore::test_with_env_var 2024-12-18T02:57:45.8334378Z Running 1 items in this shard: test/distributed/test_store.py::InitPgWithNonUvStore::test_with_url_param 2024-12-18T02:57:45.8335329Z Running 1 items in this shard: test/distributed/test_store.py::TestClientProtocol::test_client_connect 2024-12-18T02:57:45.8336093Z 2024-12-18T02:57:45.8336437Z Running distributed/elastic/timer/local_timer_example 1/1 ... [2024-12-18 02:57:45.820465] 2024-12-18T02:57:45.8337038Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:57:45.8338336Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:57:45.820837] 2024-12-18T02:58:07.0401597Z 2024-12-18T02:58:07.0403004Z distributed/elastic/timer/local_timer_example 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.local_timer_example_1.1_f53082b28c197712_.log 2024-12-18T02:58:07.0406250Z Running 2 items in this shard: test/distributed/elastic/timer/local_timer_example.py::LocalTimerExample::test_example_start_method_spawn, test/distributed/elastic/timer/local_timer_example.py::LocalTimerExample::test_torch_mp_example 2024-12-18T02:58:07.0407626Z 2024-12-18T02:58:07.0409881Z Running distributed/elastic/utils/distributed_test 1/1 ... [2024-12-18 02:58:07.040030] 2024-12-18T02:58:07.0410535Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:58:07.0411878Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/utils/distributed_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:58:07.040440] 2024-12-18T02:58:17.8388691Z 2024-12-18T02:58:17.8390059Z distributed/elastic/utils/distributed_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.distributed_test_1.1_2d0b1e3224b2546b_.log 2024-12-18T02:58:17.8395527Z Running 8 items in this shard: test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_multi, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_no_port_multi, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_single_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_timeout_on_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_timeout_on_worker, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_with_libuv_support, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_port_already_in_use_on_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_port_already_in_use_on_worker 2024-12-18T02:58:17.8400388Z 2024-12-18T02:58:18.5982657Z Running test batch 'tests to run' cost 10184.34 seconds 2024-12-18T02:58:19.4166013Z 2024-12-18T02:58:19.4166491Z real 169m50.503s 2024-12-18T02:58:19.4166828Z user 373m41.589s 2024-12-18T02:58:19.4167105Z sys 176m0.171s 2024-12-18T02:58:19.4167364Z + assert_git_not_dirty 2024-12-18T02:58:19.4167769Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-12-18T02:58:19.4168246Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-12-18T02:58:19.4170129Z ++ git status --porcelain 2024-12-18T02:58:19.4171057Z ++ grep -v '?? third_party' 2024-12-18T02:58:22.0263128Z ++ true 2024-12-18T02:58:22.0263442Z + git_status= 2024-12-18T02:58:22.0263721Z + [[ -n '' ]] 2024-12-18T02:58:22.0264052Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-12-18T02:58:22.0264462Z + [[ 3 == 1 ]] 2024-12-18T02:58:22.0264718Z + [[ 3 == 1 ]] 2024-12-18T02:58:22.0264981Z + cleanup_workspace 2024-12-18T02:58:22.0265746Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-12-18T02:58:22.0266707Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-12-18T02:58:22.0267433Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-12-18T02:58:22.0267977Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-12-18T02:58:22.0268610Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-12-18T02:58:22.0269285Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-12-18T02:58:22.0269836Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-12-18T02:58:22.5249586Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@release/2.6 2024-12-18T02:58:22.5250203Z with: 2024-12-18T02:58:22.5250500Z benchmark-results-dir: test/test-reports 2024-12-18T02:58:22.5250889Z dry-run: false 2024-12-18T02:58:22.5251158Z schema-version: v3 2024-12-18T02:58:22.5251683Z github-token: *** 2024-12-18T02:58:22.5251984Z env: 2024-12-18T02:58:22.5252240Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:22.5252643Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:22.5253307Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:22.5253914Z ##[endgroup] 2024-12-18T02:58:22.5286995Z ##[group]Run set -eux 2024-12-18T02:58:22.5287311Z set -eux 2024-12-18T02:58:22.5287623Z python3 -mpip install boto3==1.35.33 2024-12-18T02:58:22.5311251Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:58:22.5311691Z env: 2024-12-18T02:58:22.5311982Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:22.5312367Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:22.5313002Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:22.5313578Z ##[endgroup] 2024-12-18T02:58:22.5344242Z + python3 -mpip install boto3==1.35.33 2024-12-18T02:58:22.7906632Z Defaulting to user installation because normal site-packages is not writeable 2024-12-18T02:58:23.9511907Z Collecting boto3==1.35.33 2024-12-18T02:58:23.9915473Z Downloading boto3-1.35.33-py3-none-any.whl (139 kB) 2024-12-18T02:58:24.0199178Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0) 2024-12-18T02:58:24.0589278Z Collecting s3transfer<0.11.0,>=0.10.0 2024-12-18T02:58:24.0635349Z Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB) 2024-12-18T02:58:25.3163144Z Collecting botocore<1.36.0,>=1.35.33 2024-12-18T02:58:25.3206671Z Downloading botocore-1.35.83-py3-none-any.whl (13.3 MB) 2024-12-18T02:58:25.5448546Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1) 2024-12-18T02:58:25.5457842Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10) 2024-12-18T02:58:25.7003839Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0) 2024-12-18T02:58:25.7889996Z Installing collected packages: botocore, s3transfer, boto3 2024-12-18T02:58:26.3787014Z Successfully installed boto3-1.35.33 botocore-1.35.83 s3transfer-0.10.4 2024-12-18T02:58:26.4719985Z ##[group]Run set -eux 2024-12-18T02:58:26.4720302Z set -eux 2024-12-18T02:58:26.4720569Z  2024-12-18T02:58:26.4720847Z if [[ -z "${GITHUB_TOKEN}" ]]; then 2024-12-18T02:58:26.4721269Z  echo "Missing github-token input" 2024-12-18T02:58:26.4721636Z  exit 1 2024-12-18T02:58:26.4721876Z fi 2024-12-18T02:58:26.4728839Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:58:26.4729298Z env: 2024-12-18T02:58:26.4729563Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:26.4729964Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:26.4730646Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:26.4731563Z GITHUB_TOKEN: *** 2024-12-18T02:58:26.4731850Z ##[endgroup] 2024-12-18T02:58:26.4761577Z + [[ -z *** ]] 2024-12-18T02:58:26.4818382Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main 2024-12-18T02:58:26.4818853Z with: 2024-12-18T02:58:26.4819222Z github-token: *** 2024-12-18T02:58:26.4819480Z env: 2024-12-18T02:58:26.4819716Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:26.4820066Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:26.4820835Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:26.4821383Z ##[endgroup] 2024-12-18T02:58:26.4846577Z ##[group]Run set -eux 2024-12-18T02:58:26.4846900Z set -eux 2024-12-18T02:58:26.4847180Z  2024-12-18T02:58:26.4847741Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-12-18T02:58:26.4853945Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:58:26.4854400Z env: 2024-12-18T02:58:26.4854658Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:26.4855061Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:26.4855842Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:26.4856722Z GITHUB_TOKEN: *** 2024-12-18T02:58:26.4856964Z ##[endgroup] 2024-12-18T02:58:26.4880413Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 12383255652 i-0c6125b7fa38a3d09 2024-12-18T02:58:29.4409807Z setting job-id=34566463052 2024-12-18T02:58:29.4410532Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T02:58:29.4519035Z ##[group]Run set -eux 2024-12-18T02:58:29.4519390Z set -eux 2024-12-18T02:58:29.4519668Z  2024-12-18T02:58:29.4520216Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \ 2024-12-18T02:58:29.4520759Z  --schema-version "${SCHEMA_VERSION}" \ 2024-12-18T02:58:29.4521134Z  --repo "${REPO}" \ 2024-12-18T02:58:29.4521463Z  --head-branch "${HEAD_BRANCH}" \ 2024-12-18T02:58:29.4521820Z  --head-sha "${HEAD_SHA}" \ 2024-12-18T02:58:29.4522180Z  --workflow-id "${WORKFLOW_RUN_ID}" \ 2024-12-18T02:58:29.4522545Z  --run-attempt "${RUN_ATTEMPT}" \ 2024-12-18T02:58:29.4522904Z  --job-id "${JOB_ID}" \ 2024-12-18T02:58:29.4523225Z  --job-name "${JOB_NAME}" 2024-12-18T02:58:29.4529486Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:58:29.4529942Z env: 2024-12-18T02:58:29.4530202Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:29.4530586Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:29.4531421Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:29.4532038Z SCHEMA_VERSION: v3 2024-12-18T02:58:29.4532341Z REPO: pytorch/pytorch 2024-12-18T02:58:29.4532666Z HEAD_BRANCH: refs/heads/release/2.6 2024-12-18T02:58:29.4533070Z HEAD_SHA: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T02:58:29.4533484Z WORKFLOW_RUN_ID: 12383255652 2024-12-18T02:58:29.4533813Z RUN_ATTEMPT: 1 2024-12-18T02:58:29.4534083Z JOB_ID: 34566463052 2024-12-18T02:58:29.4534663Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T02:58:29.4535307Z ##[endgroup] 2024-12-18T02:58:29.4561466Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/release/2.6/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/release/2.6 --head-sha 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 --workflow-id 12383255652 --run-attempt 1 --job-id 34566463052 --job-name 'linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu)' 2024-12-18T02:58:29.4908598Z ##[group]Run set -eux 2024-12-18T02:58:29.4908890Z set -eux 2024-12-18T02:58:29.4909123Z  2024-12-18T02:58:29.4909394Z # TODO (huydhn): Implement this part 2024-12-18T02:58:29.4909792Z echo "runners=[]" >> "${GITHUB_OUTPUT}" 2024-12-18T02:58:29.4915052Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:58:29.4915458Z env: 2024-12-18T02:58:29.4915688Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:29.4916192Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:29.4916795Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:29.4917340Z ##[endgroup] 2024-12-18T02:58:29.4940130Z + echo 'runners=[]' 2024-12-18T02:58:29.4964810Z ##[group]Run set -eux 2024-12-18T02:58:29.4965128Z set -eux 2024-12-18T02:58:29.4965561Z  2024-12-18T02:58:29.4965862Z # TODO (huydhn): Implement this part 2024-12-18T02:58:29.4966331Z echo "dependencies={}" >> "${GITHUB_OUTPUT}" 2024-12-18T02:58:29.4972212Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:58:29.4972644Z env: 2024-12-18T02:58:29.4972890Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:29.4973272Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:29.4973921Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:29.4974498Z ##[endgroup] 2024-12-18T02:58:29.4999707Z + echo 'dependencies={}' 2024-12-18T02:58:29.5030276Z ##[group]Run set -eux 2024-12-18T02:58:29.5030572Z set -eux 2024-12-18T02:58:29.5030821Z  2024-12-18T02:58:29.5031109Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then 2024-12-18T02:58:29.5031580Z  echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping" 2024-12-18T02:58:29.5032110Z  # We don't want the job to fail if the directory doesn't exist 2024-12-18T02:58:29.5032532Z  exit 0 2024-12-18T02:58:29.5032775Z fi 2024-12-18T02:58:29.5033001Z  2024-12-18T02:58:29.5033257Z if [[ "${DRY_RUN}" == "true" ]]; then 2024-12-18T02:58:29.5033759Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2024-12-18T02:58:29.5034351Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2024-12-18T02:58:29.5034810Z  --metadata "${BENCHMARK_METADATA}" \ 2024-12-18T02:58:29.5035194Z  --runners "${RUNNER_INFO}" \ 2024-12-18T02:58:29.5035565Z  --dependencies "${DEPENDENCIES}" \ 2024-12-18T02:58:29.5035918Z  --dry-run 2024-12-18T02:58:29.5036173Z else 2024-12-18T02:58:29.5036577Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2024-12-18T02:58:29.5037162Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2024-12-18T02:58:29.5037743Z  --metadata "${BENCHMARK_METADATA}" \ 2024-12-18T02:58:29.5038124Z  --runners "${RUNNER_INFO}" \ 2024-12-18T02:58:29.5038494Z  --dependencies "${DEPENDENCIES}" 2024-12-18T02:58:29.5038829Z fi 2024-12-18T02:58:29.5044004Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:58:29.5044457Z env: 2024-12-18T02:58:29.5044790Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:29.5045190Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:29.5045863Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:29.5046508Z BENCHMARK_RESULTS_DIR: test/test-reports 2024-12-18T02:58:29.5046887Z DRY_RUN: false 2024-12-18T02:58:29.5048586Z BENCHMARK_METADATA: {"timestamp": 1734490709, "schema_version": "v3", "name": "linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/release/2.6", "head_sha": "0cdf8b1d09254cfda66191d1bd01e3041c3c76f7", "workflow_id": 12383255652, "run_attempt": 1, "job_id": 34566463052} 2024-12-18T02:58:29.5050318Z RUNNER_INFO: [] 2024-12-18T02:58:29.5050598Z DEPENDENCIES: {} 2024-12-18T02:58:29.5050881Z ##[endgroup] 2024-12-18T02:58:29.5073604Z + [[ ! -d test/test-reports ]] 2024-12-18T02:58:29.5073981Z + [[ false == \t\r\u\e ]] 2024-12-18T02:58:29.5077008Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/release/2.6/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1734490709, "schema_version": "v3", "name": "linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/release/2.6", "head_sha": "0cdf8b1d09254cfda66191d1bd01e3041c3c76f7", "workflow_id": 12383255652, "run_attempt": 1, "job_id": 34566463052}' --runners '[]' --dependencies '{}' 2024-12-18T02:58:29.6833172Z ##[group]Run cat test/**/*_toprint.log || true 2024-12-18T02:58:29.6833581Z cat test/**/*_toprint.log || true 2024-12-18T02:58:29.6839032Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:58:29.6839441Z env: 2024-12-18T02:58:29.6839677Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:29.6840036Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:29.6840648Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:29.6841177Z ##[endgroup] 2024-12-18T02:58:29.6914452Z cat: 'test/**/*_toprint.log': No such file or directory 2024-12-18T02:58:29.6948107Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-12-18T02:58:29.6948492Z kill "$MONITOR_SCRIPT_PID" 2024-12-18T02:58:29.6953805Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:58:29.6954236Z env: 2024-12-18T02:58:29.6954481Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:29.6954870Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:29.6955510Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:29.6956082Z MONITOR_SCRIPT_PID: 57227 2024-12-18T02:58:29.6956384Z ##[endgroup] 2024-12-18T02:58:29.7134539Z Prepare all required actions 2024-12-18T02:58:29.7135016Z Getting action download info 2024-12-18T02:58:29.8609412Z Download action repository 'actions/upload-artifact@v4' (SHA:6f51ac03b9356f520e9adb1b1b7802705f340c2b) 2024-12-18T02:58:30.1501328Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-12-18T02:58:30.1501735Z with: 2024-12-18T02:58:30.1502171Z file-suffix: test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_34566463052 2024-12-18T02:58:30.1502708Z s3-bucket: gha-artifacts 2024-12-18T02:58:30.1503000Z env: 2024-12-18T02:58:30.1503239Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:30.1503616Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:30.1504407Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:30.1504982Z ##[endgroup] 2024-12-18T02:58:30.1534622Z ##[group]Run # Remove any previous test jsons if they exist 2024-12-18T02:58:30.1535226Z # Remove any previous test jsons if they exist 2024-12-18T02:58:30.1535772Z rm -f test-jsons-*.zip 2024-12-18T02:58:30.1536548Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2024-12-18T02:58:30.1543560Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:58:30.1544009Z env: 2024-12-18T02:58:30.1544265Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:30.1544657Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:30.1545312Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:30.1546106Z FILE_SUFFIX: test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_34566463052 2024-12-18T02:58:30.1546657Z ##[endgroup] 2024-12-18T02:58:30.1712017Z adding: test/test-reports/td_exclusions-bee2856e37bf34d125da.json (deflated 86%) 2024-12-18T02:58:30.1740899Z ##[group]Run # Remove any previous test reports if they exist 2024-12-18T02:58:30.1741431Z # Remove any previous test reports if they exist 2024-12-18T02:58:30.1741880Z rm -f test-reports-*.zip 2024-12-18T02:58:30.1742409Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2024-12-18T02:58:30.1747889Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:58:30.1748289Z env: 2024-12-18T02:58:30.1748523Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:30.1748883Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:30.1749489Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:30.1750198Z FILE_SUFFIX: test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_34566463052 2024-12-18T02:58:30.1750695Z ##[endgroup] 2024-12-18T02:58:30.1856316Z adding: test/test-reports/python-pytest/distributed.test_inductor_collectives/distributed.test_inductor_collectives-fb05a8a9bdabe00c.xml (deflated 87%) 2024-12-18T02:58:30.1858026Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_binary_cmp/distributed._shard.sharded_tensor.ops.test_binary_cmp-f75aa1ac10a4f29a.xml (deflated 67%) 2024-12-18T02:58:30.1859649Z adding: test/test-reports/python-pytest/distributed._tensor.test_common_rules/distributed._tensor.test_common_rules-84cc3395caa18d7b.xml (deflated 78%) 2024-12-18T02:58:30.1861047Z adding: test/test-reports/python-pytest/distributed._tensor.test_init/distributed._tensor.test_init-2ae0d7f4556e3650.xml (deflated 71%) 2024-12-18T02:58:30.1862470Z adding: test/test-reports/python-pytest/distributed._tensor.test_xla_integration/distributed._tensor.test_xla_integration-f28cdfc04411f21d.xml (deflated 74%) 2024-12-18T02:58:30.1863996Z adding: test/test-reports/python-pytest/distributed._tensor.test_convolution_ops/distributed._tensor.test_convolution_ops-339beb24b23898e0.xml (deflated 51%) 2024-12-18T02:58:30.1865641Z adding: test/test-reports/python-pytest/distributed._tensor.test_math_ops/distributed._tensor.test_math_ops-cd9a1d95db77481b.xml (deflated 79%) 2024-12-18T02:58:30.1867293Z adding: test/test-reports/python-pytest/distributed._tensor.test_pointwise_ops/distributed._tensor.test_pointwise_ops-f29ab7def8bd8ad6.xml (deflated 70%) 2024-12-18T02:58:30.1868904Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_init/distributed._composable.fsdp.test_fully_shard_init-65690163a3764c2a.xml (deflated 86%) 2024-12-18T02:58:30.1870500Z adding: test/test-reports/python-pytest/distributed._tensor.test_embedding_ops/distributed._tensor.test_embedding_ops-12b0bf46d877c82a.xml (deflated 64%) 2024-12-18T02:58:30.1871958Z adding: test/test-reports/python-pytest/distributed._tensor.test_matrix_ops/distributed._tensor.test_matrix_ops-940d7deed02220cf.xml (deflated 78%) 2024-12-18T02:58:30.1873573Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-76a577cca3c3a2e1.xml (deflated 37%) 2024-12-18T02:58:30.1875152Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5047073007c2484c.xml (deflated 37%) 2024-12-18T02:58:30.1876719Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-159e52994529bb83.xml (deflated 39%) 2024-12-18T02:58:30.1878286Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d4d15a5e685a62b2.xml (deflated 38%) 2024-12-18T02:58:30.1879828Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-76ff8174629f328e.xml (deflated 39%) 2024-12-18T02:58:30.1881389Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-758f53607c3a8d4d.xml (deflated 39%) 2024-12-18T02:58:30.1882965Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0a6bbf3c32c1e690.xml (deflated 39%) 2024-12-18T02:58:30.1884844Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ff6e77f484951902.xml (deflated 39%) 2024-12-18T02:58:30.1886465Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-7ce046ff97854675.xml (deflated 39%) 2024-12-18T02:58:30.1888076Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-6f2c922e1c4d848b.xml (deflated 39%) 2024-12-18T02:58:30.1889703Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a22c86f23dbe4a74.xml (deflated 39%) 2024-12-18T02:58:30.1891329Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2227146ac9881d16.xml (deflated 39%) 2024-12-18T02:58:30.1892952Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ea84f61b1aa15ff3.xml (deflated 38%) 2024-12-18T02:58:30.1894569Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-517e12004d3f7d30.xml (deflated 39%) 2024-12-18T02:58:30.1896261Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-030e060bf20e2918.xml (deflated 39%) 2024-12-18T02:58:30.1897834Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-77b4cad24b57a645.xml (deflated 38%) 2024-12-18T02:58:30.1899403Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-18f38a9e135de797.xml (deflated 38%) 2024-12-18T02:58:30.1901065Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-771362824c18d1e3.xml (deflated 38%) 2024-12-18T02:58:30.1902635Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-f3fede4835f84b01.xml (deflated 38%) 2024-12-18T02:58:30.1904196Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-75959d0701113803.xml (deflated 38%) 2024-12-18T02:58:30.1905763Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-3c55e4906ec92ef9.xml (deflated 38%) 2024-12-18T02:58:30.1907334Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-19bff1ea49c18c34.xml (deflated 38%) 2024-12-18T02:58:30.1908971Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-61dda573b2de483b.xml (deflated 38%) 2024-12-18T02:58:30.1910548Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-543325a1b3a056c4.xml (deflated 38%) 2024-12-18T02:58:30.1912102Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-dcc96b999a705e66.xml (deflated 38%) 2024-12-18T02:58:30.1913671Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-3a16026c4fc15da0.xml (deflated 38%) 2024-12-18T02:58:30.1915248Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1db6a37dc9237ea3.xml (deflated 38%) 2024-12-18T02:58:30.1916827Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d588b0e126499346.xml (deflated 38%) 2024-12-18T02:58:30.1918406Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5cf55b201cacffb2.xml (deflated 38%) 2024-12-18T02:58:30.1919984Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ce6ebdee5e31d115.xml (deflated 38%) 2024-12-18T02:58:30.1921559Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-7bf4d61bf750eafe.xml (deflated 38%) 2024-12-18T02:58:30.1923135Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-8e043d2fe1ee8d95.xml (deflated 38%) 2024-12-18T02:58:30.1925004Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e642dccbcdfa5fac.xml (deflated 38%) 2024-12-18T02:58:30.1926624Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-f25a963f184e6fd4.xml (deflated 38%) 2024-12-18T02:58:30.1928244Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9969549c13fb88e3.xml (deflated 38%) 2024-12-18T02:58:30.1929864Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-42d1231d843b51d1.xml (deflated 39%) 2024-12-18T02:58:30.1931485Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d26849c47595d46a.xml (deflated 39%) 2024-12-18T02:58:30.1933112Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-899e4b31d7afa201.xml (deflated 37%) 2024-12-18T02:58:30.1934834Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b9efa3418ce165d7.xml (deflated 37%) 2024-12-18T02:58:30.1936546Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-448e95c07ac3a098.xml (deflated 38%) 2024-12-18T02:58:30.1938113Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-75ad83338f98e56d.xml (deflated 38%) 2024-12-18T02:58:30.1939689Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b2e26140da305d16.xml (deflated 38%) 2024-12-18T02:58:30.1941258Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-fc5879a66678b644.xml (deflated 38%) 2024-12-18T02:58:30.1942887Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-72c6d6b26507a909.xml (deflated 38%) 2024-12-18T02:58:30.1944447Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-cab4a0c68a3ccd89.xml (deflated 38%) 2024-12-18T02:58:30.1946023Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-f0ac697962f4990d.xml (deflated 38%) 2024-12-18T02:58:30.1947599Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-21b25c3b982fef24.xml (deflated 38%) 2024-12-18T02:58:30.1949177Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-49fcd9f51ed520df.xml (deflated 38%) 2024-12-18T02:58:30.1950763Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0a9fe9f0853cd5dd.xml (deflated 38%) 2024-12-18T02:58:30.1952351Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2370004efcb6e0ad.xml (deflated 38%) 2024-12-18T02:58:30.1953933Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ea7da3663e8668ef.xml (deflated 39%) 2024-12-18T02:58:30.1955505Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-05871c6ede9d77e4.xml (deflated 38%) 2024-12-18T02:58:30.1957175Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_embedding/distributed._shard.sharded_tensor.ops.test_embedding-528fcacc322fd211.xml (deflated 53%) 2024-12-18T02:58:30.1958862Z adding: test/test-reports/python-pytest/distributed.flight_recorder.test_fr_analysis/distributed.flight_recorder.test_fr_analysis-86c0f620be32e43d.xml (deflated 36%) 2024-12-18T02:58:30.1960432Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_compile/distributed._tensor.test_dtensor_compile-a871a72482a85634.xml (deflated 86%) 2024-12-18T02:58:30.1962081Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_compile/distributed._composable.fsdp.test_fully_shard_compile-494c54fbf81e6e60.xml (deflated 84%) 2024-12-18T02:58:30.1963911Z adding: test/test-reports/python-pytest/distributed._tools.test_mem_tracker/distributed._tools.test_mem_tracker-aabb7c70c0774467.xml (deflated 58%) 2024-12-18T02:58:30.1965559Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_fx/distributed.fsdp.test_fsdp_fx-393527c86b263205.xml (deflated 35%) 2024-12-18T02:58:30.1966977Z adding: test/test-reports/python-pytest/distributed._tensor.test_api/distributed._tensor.test_api-76bc49b6f2e8af44.xml (deflated 75%) 2024-12-18T02:58:30.1968296Z adding: test/test-reports/python-pytest/distributed.test_backends/distributed.test_backends-1a7c9ea674001704.xml (deflated 49%) 2024-12-18T02:58:30.1969965Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_logging/distributed._composable.fsdp.test_fully_shard_logging-1224ba0542db437e.xml (deflated 28%) 2024-12-18T02:58:30.1971611Z adding: test/test-reports/python-pytest/distributed.test_symmetric_memory/distributed.test_symmetric_memory-c60a731f858df211.xml (deflated 97%) 2024-12-18T02:58:30.1973075Z adding: test/test-reports/python-pytest/distributed.pipelining.test_stage/distributed.pipelining.test_stage-f9924a1457155ce1.xml (deflated 61%) 2024-12-18T02:58:30.1974554Z adding: test/test-reports/python-pytest/distributed.pipelining.test_stage/distributed.pipelining.test_stage-5338341d48f50446.xml (deflated 69%) 2024-12-18T02:58:30.1976163Z adding: test/test-reports/python-pytest/distributed._tools.test_memory_tracker/distributed._tools.test_memory_tracker-e5b27386d21b5c38.xml (deflated 63%) 2024-12-18T02:58:30.1978000Z adding: test/test-reports/python-pytest/distributed._composable.test_composability.test_pp_composability/distributed._composable.test_composability.test_pp_composability-d9704f656f864f68.xml (deflated 88%) 2024-12-18T02:58:30.1979783Z adding: test/test-reports/python-pytest/distributed.elastic.test_control_plane/distributed.elastic.test_control_plane-2ea8196cfba9fc39.xml (deflated 73%) 2024-12-18T02:58:30.1981326Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_tp_checkpoint/distributed.checkpoint.test_tp_checkpoint-649c56c590bd6bc3.xml (deflated 52%) 2024-12-18T02:58:30.1983005Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_overlap/distributed._composable.fsdp.test_fully_shard_overlap-7e763a4a4a5c6610.xml (deflated 63%) 2024-12-18T02:58:30.1984635Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_planner/distributed.checkpoint.test_planner-789442e608b7dd33.xml (deflated 71%) 2024-12-18T02:58:30.1986332Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion-cfc4c2e5219450e3.xml (deflated 40%) 2024-12-18T02:58:30.1988105Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_state_dict_utils/distributed.checkpoint.test_state_dict_utils-8d5329dbb4c1a879.xml (deflated 71%) 2024-12-18T02:58:30.1989599Z adding: test/test-reports/python-pytest/distributed.fsdp.test_shard_utils/distributed.fsdp.test_shard_utils-01766e1d724e35e8.xml (deflated 50%) 2024-12-18T02:58:30.1991104Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_compatibility/distributed.checkpoint.test_compatibility-80b0aafe7082c55f.xml (deflated 70%) 2024-12-18T02:58:30.1992727Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_hsdp_checkpoint/distributed.checkpoint.test_hsdp_checkpoint-3d8178d258eca033.xml (deflated 68%) 2024-12-18T02:58:30.1994126Z adding: test/test-reports/python-pytest/distributed.test_nccl/distributed.test_nccl-c8d20a81e47eebe9.xml (deflated 80%) 2024-12-18T02:58:30.1995375Z adding: test/test-reports/python-pytest/distributed.test_functional_api/distributed.test_functional_api-5847822871ee8fda.xml (deflated 84%) 2024-12-18T02:58:30.1996771Z adding: test/test-reports/python-pytest/distributed._tensor.test_optimizers/distributed._tensor.test_optimizers-0786b26c920dcef8.xml (deflated 80%) 2024-12-18T02:58:30.1998416Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_grad_scaler/distributed._composable.fsdp.test_fully_shard_grad_scaler-4c87169c969b453f.xml (deflated 39%) 2024-12-18T02:58:30.2000064Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_utils/distributed.checkpoint.test_utils-659fcc578fc08172.xml (deflated 71%) 2024-12-18T02:58:30.2001454Z adding: test/test-reports/python-pytest/distributed._tensor.test_utils/distributed._tensor.test_utils-bf621b255adae8cb.xml (deflated 75%) 2024-12-18T02:58:30.2002747Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-21717c4482974d2a.xml (deflated 35%) 2024-12-18T02:58:30.2004433Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f41155767fb0b6d5.xml (deflated 44%) 2024-12-18T02:58:30.2005730Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b87b8f3646e2abd0.xml (deflated 34%) 2024-12-18T02:58:30.2006999Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6fc1b47cd35e4a41.xml (deflated 34%) 2024-12-18T02:58:30.2008288Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c5ef340b5378d234.xml (deflated 34%) 2024-12-18T02:58:30.2009569Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-769fbf8d9fc977b5.xml (deflated 34%) 2024-12-18T02:58:30.2011119Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-109f68ecd039e12a.xml (deflated 34%) 2024-12-18T02:58:30.2012414Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-01e8b832a233433c.xml (deflated 35%) 2024-12-18T02:58:30.2013696Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2798b875f9a40df2.xml (deflated 36%) 2024-12-18T02:58:30.2014981Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a5b1b367741a7fd8.xml (deflated 35%) 2024-12-18T02:58:30.2016358Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-58d35989c1b933a5.xml (deflated 36%) 2024-12-18T02:58:30.2017590Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d7579408c3809e1c.xml (deflated 49%) 2024-12-18T02:58:30.2018838Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c3ef6683d214b581.xml (deflated 49%) 2024-12-18T02:58:30.2020091Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-77d14c388471ad27.xml (deflated 34%) 2024-12-18T02:58:30.2021337Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d9bdf136fa50502b.xml (deflated 34%) 2024-12-18T02:58:30.2022581Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f435e170a202300f.xml (deflated 36%) 2024-12-18T02:58:30.2023822Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5acd600ae342bc01.xml (deflated 36%) 2024-12-18T02:58:30.2025066Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-dc59dbe0720d735b.xml (deflated 37%) 2024-12-18T02:58:30.2026307Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0f4b6a129fb23532.xml (deflated 36%) 2024-12-18T02:58:30.2027539Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5ba19fea280d128c.xml (deflated 36%) 2024-12-18T02:58:30.2028788Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0f654000e7b6b3c4.xml (deflated 36%) 2024-12-18T02:58:30.2030024Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-56385199302fd5c4.xml (deflated 37%) 2024-12-18T02:58:30.2031263Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3be3407dfa2576a6.xml (deflated 36%) 2024-12-18T02:58:30.2032510Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ddede7cb7ff52229.xml (deflated 36%) 2024-12-18T02:58:30.2033761Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c93a8ce6aad8cb67.xml (deflated 36%) 2024-12-18T02:58:30.2035013Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8f084c10a6f845c5.xml (deflated 36%) 2024-12-18T02:58:30.2036304Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7f72edf039c25988.xml (deflated 36%) 2024-12-18T02:58:30.2037554Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4a6384a014f8bc28.xml (deflated 36%) 2024-12-18T02:58:30.2038804Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ea94c1579d3dbcf9.xml (deflated 36%) 2024-12-18T02:58:30.2040052Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-549aeb28624fa0b4.xml (deflated 37%) 2024-12-18T02:58:30.2041296Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a99519c9d424c35d.xml (deflated 37%) 2024-12-18T02:58:30.2042614Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-251bcd1a087c3b90.xml (deflated 36%) 2024-12-18T02:58:30.2044133Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9282a46570ca1c06.xml (deflated 36%) 2024-12-18T02:58:30.2045422Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f4d8429cef7903aa.xml (deflated 37%) 2024-12-18T02:58:30.2046695Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-16cb7028cab4f580.xml (deflated 44%) 2024-12-18T02:58:30.2047986Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5deaedaa28d65bc4.xml (deflated 38%) 2024-12-18T02:58:30.2049269Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8bf3c897138d6493.xml (deflated 37%) 2024-12-18T02:58:30.2050555Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-192176d8aa3dd95c.xml (deflated 38%) 2024-12-18T02:58:30.2051845Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a64bd98b5b3abe88.xml (deflated 37%) 2024-12-18T02:58:30.2053150Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3ebed7ee932aef56.xml (deflated 37%) 2024-12-18T02:58:30.2054438Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fe47a90c982481ea.xml (deflated 36%) 2024-12-18T02:58:30.2055814Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-af04b9c00d3810af.xml (deflated 35%) 2024-12-18T02:58:30.2057068Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a6d3c7c62a54cd20.xml (deflated 35%) 2024-12-18T02:58:30.2058321Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7d359248aea266d6.xml (deflated 35%) 2024-12-18T02:58:30.2059577Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f8926dc89fdc420b.xml (deflated 35%) 2024-12-18T02:58:30.2060837Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8f144494b16954d3.xml (deflated 35%) 2024-12-18T02:58:30.2062085Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4905ad5405caa618.xml (deflated 35%) 2024-12-18T02:58:30.2063326Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3939d6733db353bc.xml (deflated 36%) 2024-12-18T02:58:30.2064581Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f7ce5ce95db3ab92.xml (deflated 36%) 2024-12-18T02:58:30.2066135Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-08cb117cfde16c8b.xml (deflated 35%) 2024-12-18T02:58:30.2067435Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9a0123e2adda8d81.xml (deflated 37%) 2024-12-18T02:58:30.2068835Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4a19258ec31ca119.xml (deflated 36%) 2024-12-18T02:58:30.2070119Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ce17b2753c41102f.xml (deflated 36%) 2024-12-18T02:58:30.2071405Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-1677de70bb39fe91.xml (deflated 37%) 2024-12-18T02:58:30.2072692Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7cc03688b569f123.xml (deflated 36%) 2024-12-18T02:58:30.2073977Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c100a3f00ce10f7f.xml (deflated 35%) 2024-12-18T02:58:30.2075261Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4350bcd2842935f8.xml (deflated 36%) 2024-12-18T02:58:30.2076615Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b7011e792d3c1ae6.xml (deflated 33%) 2024-12-18T02:58:30.2077993Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-388e6dd1665de982.xml (deflated 34%) 2024-12-18T02:58:30.2079240Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2bcbc1293f9c603a.xml (deflated 34%) 2024-12-18T02:58:30.2080491Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7b1662af9435894e.xml (deflated 35%) 2024-12-18T02:58:30.2081735Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6f8a3b4f4e2d73b4.xml (deflated 35%) 2024-12-18T02:58:30.2082984Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-35d99c84c5ab558c.xml (deflated 35%) 2024-12-18T02:58:30.2084483Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-42a5bcc821586677.xml (deflated 35%) 2024-12-18T02:58:30.2085811Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e17c4cab9a660d6d.xml (deflated 35%) 2024-12-18T02:58:30.2087093Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-09d255b2c8a363e0.xml (deflated 35%) 2024-12-18T02:58:30.2088377Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-83a394e9fa113d95.xml (deflated 36%) 2024-12-18T02:58:30.2109783Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c3e661b227cc2697.xml (deflated 35%) 2024-12-18T02:58:30.2111054Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a74fba82afd36b82.xml (deflated 35%) 2024-12-18T02:58:30.2112319Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-31e1cd10d248fc65.xml (deflated 36%) 2024-12-18T02:58:30.2113549Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-afad4e31b6a53295.xml (deflated 35%) 2024-12-18T02:58:30.2114802Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2e4f2dfe0bc9d89a.xml (deflated 35%) 2024-12-18T02:58:30.2116025Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-90f9cb4936e4866c.xml (deflated 35%) 2024-12-18T02:58:30.2117258Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-60b891f0a0004eaf.xml (deflated 34%) 2024-12-18T02:58:30.2118477Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d3a7f941e5e9ed56.xml (deflated 35%) 2024-12-18T02:58:30.2119713Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-845d82e2c8b55c6b.xml (deflated 35%) 2024-12-18T02:58:30.2120949Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f8643f90b48ed72e.xml (deflated 34%) 2024-12-18T02:58:30.2122319Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-65a164b4d4ee80eb.xml (deflated 35%) 2024-12-18T02:58:30.2123656Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-645371691ef41f23.xml (deflated 35%) 2024-12-18T02:58:30.2125130Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-19269e5672392c60.xml (deflated 36%) 2024-12-18T02:58:30.2126394Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f83c081f290b7c30.xml (deflated 35%) 2024-12-18T02:58:30.2127672Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6432aeb9f9b32e1e.xml (deflated 43%) 2024-12-18T02:58:30.2129129Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_overlap/distributed.fsdp.test_fsdp_overlap-358476f6c65eff76.xml (deflated 51%) 2024-12-18T02:58:30.2130624Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_optim_state/distributed.fsdp.test_fsdp_optim_state-fe687ce610bae078.xml (deflated 93%) 2024-12-18T02:58:30.2132304Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_file_system_checkpoint/distributed.checkpoint.test_file_system_checkpoint-90fcbcc236ea5e91.xml (deflated 75%) 2024-12-18T02:58:30.2133931Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_apply/distributed.fsdp.test_fsdp_apply-cc095a175abcbbf7.xml (deflated 59%) 2024-12-18T02:58:30.2135438Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_flatten_params/distributed.fsdp.test_fsdp_flatten_params-dbe4419b94844503.xml (deflated 82%) 2024-12-18T02:58:30.2137158Z adding: test/test-reports/python-pytest/distributed.optim.test_zero_redundancy_optimizer/distributed.optim.test_zero_redundancy_optimizer-29ac51eb873f7a38.xml (deflated 91%) 2024-12-18T02:58:30.2139025Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_memory/distributed._composable.fsdp.test_fully_shard_memory-837f3ff9eed984d2.xml (deflated 52%) 2024-12-18T02:58:30.2140852Z adding: test/test-reports/python-pytest/distributed.fsdp.test_checkpoint_wrapper/distributed.fsdp.test_checkpoint_wrapper-a4d383e1330408b3.xml (deflated 73%) 2024-12-18T02:58:30.2142259Z adding: test/test-reports/python-pytest/distributed.fsdp.test_utils/distributed.fsdp.test_utils-02bcb299c5ccea0d.xml (deflated 68%) 2024-12-18T02:58:30.2143689Z adding: test/test-reports/python-pytest/distributed.fsdp.test_hsdp_dtensor_state_dict/distributed.fsdp.test_hsdp_dtensor_state_dict-563999a29228d179.xml (deflated 80%) 2024-12-18T02:58:30.2145244Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_tp_integration/distributed.fsdp.test_fsdp_tp_integration-c9e3588eb8a0d42f.xml (deflated 58%) 2024-12-18T02:58:30.2146741Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-6551ef46affc58d7.xml (deflated 89%) 2024-12-18T02:58:30.2148174Z adding: test/test-reports/python-pytest/distributed.algorithms.test_join/distributed.algorithms.test_join-7d0fe81571f08e88.xml (deflated 76%) 2024-12-18T02:58:30.2149621Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_hybrid_shard/distributed.fsdp.test_fsdp_hybrid_shard-eb6c24377024c488.xml (deflated 68%) 2024-12-18T02:58:30.2151275Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_file_system_checkpoint_cpu/distributed.checkpoint.test_file_system_checkpoint_cpu-b023517fd4733c2a.xml (deflated 77%) 2024-12-18T02:58:30.2152859Z adding: test/test-reports/python-pytest/distributed.test_data_parallel/distributed.test_data_parallel-1be877fe3f3fa86c.xml (deflated 85%) 2024-12-18T02:58:30.2154281Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-7c25744fa5f2bc4e.xml (deflated 88%) 2024-12-18T02:58:30.2155683Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_fine_tune/distributed.fsdp.test_fsdp_fine_tune-24e25e081a71dd0f.xml (deflated 63%) 2024-12-18T02:58:30.2157212Z adding: test/test-reports/python-pytest/distributed._shard.sharded_optim.test_sharded_optim/distributed._shard.sharded_optim.test_sharded_optim-4d86c0210cfbd29a.xml (deflated 52%) 2024-12-18T02:58:30.2158794Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_unshard_params/distributed.fsdp.test_fsdp_unshard_params-0ef770be16db4c1f.xml (deflated 81%) 2024-12-18T02:58:30.2160471Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_mixed_precision/distributed.fsdp.test_fsdp_mixed_precision-8a3225cd2b0d9df0.xml (deflated 92%) 2024-12-18T02:58:30.2161927Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-c2e37ee18d6dea1f.xml (deflated 36%) 2024-12-18T02:58:30.2163409Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-38c80ba26a991b3a.xml (deflated 36%) 2024-12-18T02:58:30.2165027Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-e4a1db23b3ab32d9.xml (deflated 36%) 2024-12-18T02:58:30.2166567Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-031d7c54771b3ded.xml (deflated 35%) 2024-12-18T02:58:30.2167943Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-85e7e12b62ff5907.xml (deflated 36%) 2024-12-18T02:58:30.2169331Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-5fd0af84dcdbdf26.xml (deflated 35%) 2024-12-18T02:58:30.2170724Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-efaee8e841776172.xml (deflated 35%) 2024-12-18T02:58:30.2172124Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-bd20c38b1673c6bd.xml (deflated 35%) 2024-12-18T02:58:30.2173518Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-6df060510b1a7154.xml (deflated 35%) 2024-12-18T02:58:30.2174907Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-721ce10dca9e0531.xml (deflated 35%) 2024-12-18T02:58:30.2176294Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-710d48e58fbbeb60.xml (deflated 35%) 2024-12-18T02:58:30.2177827Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-cf7e1306f8fa83ad.xml (deflated 48%) 2024-12-18T02:58:30.2179029Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-22e1877a4e80f9c5.xml (deflated 35%) 2024-12-18T02:58:30.2180240Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-69b33176fb20603d.xml (deflated 36%) 2024-12-18T02:58:30.2181442Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-68fab184c391b79e.xml (deflated 36%) 2024-12-18T02:58:30.2182644Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-15860e98a907e167.xml (deflated 35%) 2024-12-18T02:58:30.2183841Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-3f7a1f5054341af8.xml (deflated 35%) 2024-12-18T02:58:30.2185055Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-951a9ef38916f06e.xml (deflated 35%) 2024-12-18T02:58:30.2186273Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-2cbe370827c2334d.xml (deflated 35%) 2024-12-18T02:58:30.2187503Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-c990ddab721735cc.xml (deflated 36%) 2024-12-18T02:58:30.2188837Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-e7513f8f7de5913a.xml (deflated 36%) 2024-12-18T02:58:30.2190058Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-ad2568bd508e17c8.xml (deflated 36%) 2024-12-18T02:58:30.2191288Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-dcdf6dc7ad6b81d7.xml (deflated 35%) 2024-12-18T02:58:30.2192518Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-8c1e3fea9f784520.xml (deflated 35%) 2024-12-18T02:58:30.2193749Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-f389220d1d4e621f.xml (deflated 36%) 2024-12-18T02:58:30.2195063Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-2830ebf3e8ddb61b.xml (deflated 36%) 2024-12-18T02:58:30.2196225Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-debd30e30184ef99.xml (deflated 38%) 2024-12-18T02:58:30.2197298Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-23e3026a6d9655e4.xml (deflated 38%) 2024-12-18T02:58:30.2198361Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-86353b29dd2ed10e.xml (deflated 41%) 2024-12-18T02:58:30.2199438Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-dc639bacd7e709f0.xml (deflated 38%) 2024-12-18T02:58:30.2200515Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-fc372cf33c029460.xml (deflated 38%) 2024-12-18T02:58:30.2201589Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-c750d54ff91a0048.xml (deflated 39%) 2024-12-18T02:58:30.2202662Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-25f6269b555de839.xml (deflated 38%) 2024-12-18T02:58:30.2203983Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-6fa69e709745efe4.xml (deflated 38%) 2024-12-18T02:58:30.2205188Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-b08eb1f73fa13d1e.xml (deflated 38%) 2024-12-18T02:58:30.2206404Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-3c01400f4b90ba1d.xml (deflated 38%) 2024-12-18T02:58:30.2207599Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d312866759b7d64e.xml (deflated 38%) 2024-12-18T02:58:30.2208800Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-6954902d8a8af26d.xml (deflated 38%) 2024-12-18T02:58:30.2210006Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-7934d7b46c7df701.xml (deflated 38%) 2024-12-18T02:58:30.2211212Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-ad33a694c87eb185.xml (deflated 38%) 2024-12-18T02:58:30.2212414Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-88fbe616564ecb9f.xml (deflated 36%) 2024-12-18T02:58:30.2213608Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5f04aa061c95281e.xml (deflated 38%) 2024-12-18T02:58:30.2214806Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-49efaa68f5a02677.xml (deflated 37%) 2024-12-18T02:58:30.2216208Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-32805a122cfcbeff.xml (deflated 38%) 2024-12-18T02:58:30.2217286Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-70852dc17e350419.xml (deflated 38%) 2024-12-18T02:58:30.2218341Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-18adb9a79cf62883.xml (deflated 38%) 2024-12-18T02:58:30.2219470Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-6ff3a9542cb99eee.xml (deflated 37%) 2024-12-18T02:58:30.2220543Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-0247cca1ad88d901.xml (deflated 38%) 2024-12-18T02:58:30.2221610Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-45c4351e0c4a1cb5.xml (deflated 38%) 2024-12-18T02:58:30.2222676Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-caa215b45d0e919c.xml (deflated 38%) 2024-12-18T02:58:30.2223734Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-f35489a9383dcb63.xml (deflated 41%) 2024-12-18T02:58:30.2224858Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-b333f7b83f1d2058.xml (deflated 38%) 2024-12-18T02:58:30.2225929Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e9bca08037619045.xml (deflated 38%) 2024-12-18T02:58:30.2226998Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-38b5bc20b1e67c10.xml (deflated 37%) 2024-12-18T02:58:30.2228069Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-4b3bdc998d76e038.xml (deflated 37%) 2024-12-18T02:58:30.2229136Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-8061341ff3a7d285.xml (deflated 37%) 2024-12-18T02:58:30.2230195Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-88f92056f2f28d97.xml (deflated 38%) 2024-12-18T02:58:30.2231255Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-143dd3b38c83567b.xml (deflated 38%) 2024-12-18T02:58:30.2232331Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-ab6fd8bafc980d84.xml (deflated 38%) 2024-12-18T02:58:30.2233405Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d6e415501fc68a96.xml (deflated 38%) 2024-12-18T02:58:30.2234472Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-b26938e56480cd6e.xml (deflated 39%) 2024-12-18T02:58:30.2235547Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-7295f0ac1c4adf38.xml (deflated 38%) 2024-12-18T02:58:30.2236619Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-64fbb2318cd7c187.xml (deflated 38%) 2024-12-18T02:58:30.2237690Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-475469e9a971db0d.xml (deflated 37%) 2024-12-18T02:58:30.2238756Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-129f5a14f39c8323.xml (deflated 38%) 2024-12-18T02:58:30.2239823Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-14768a52bab840da.xml (deflated 37%) 2024-12-18T02:58:30.2240896Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d134acb35844158a.xml (deflated 40%) 2024-12-18T02:58:30.2241965Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-357462fe5a9c47bf.xml (deflated 37%) 2024-12-18T02:58:30.2243034Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-dc45dc1df9bb4969.xml (deflated 38%) 2024-12-18T02:58:30.2244519Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-1c61a233152bc629.xml (deflated 37%) 2024-12-18T02:58:30.2245730Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-4be56c7cba2b8b3e.xml (deflated 36%) 2024-12-18T02:58:30.2246946Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-41df8c1ad372b003.xml (deflated 38%) 2024-12-18T02:58:30.2248210Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e000f620cda3fc19.xml (deflated 38%) 2024-12-18T02:58:30.2249424Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-06e8e7b8daaff2c3.xml (deflated 37%) 2024-12-18T02:58:30.2250633Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-423b02f78b9d2f0f.xml (deflated 37%) 2024-12-18T02:58:30.2251833Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-fd7241ddd772a4b5.xml (deflated 37%) 2024-12-18T02:58:30.2253042Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-f9acb105e8a0cf69.xml (deflated 38%) 2024-12-18T02:58:30.2254251Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-ecd72f4ea87a1870.xml (deflated 37%) 2024-12-18T02:58:30.2255515Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-da92a5756590aee3.xml (deflated 37%) 2024-12-18T02:58:30.2256741Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-c6c7d8e6ed97317d.xml (deflated 38%) 2024-12-18T02:58:30.2257802Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-93b822bc52ad6229.xml (deflated 37%) 2024-12-18T02:58:30.2258866Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-659c949c4416008e.xml (deflated 37%) 2024-12-18T02:58:30.2259935Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-a3fc11a7be1aaeb6.xml (deflated 37%) 2024-12-18T02:58:30.2261003Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-747202364f5042dd.xml (deflated 37%) 2024-12-18T02:58:30.2262076Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-268389ad71eb35b6.xml (deflated 38%) 2024-12-18T02:58:30.2263140Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-3ef0814c694df2b9.xml (deflated 38%) 2024-12-18T02:58:30.2264212Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e6fcbfc86cea8bae.xml (deflated 38%) 2024-12-18T02:58:30.2265411Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-1c34f5d5f23f2ede.xml (deflated 38%) 2024-12-18T02:58:30.2266739Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-947c07448bf8cc00.xml (deflated 38%) 2024-12-18T02:58:30.2267915Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e41cdd5dca369a8b.xml (deflated 38%) 2024-12-18T02:58:30.2269087Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-07ef651c8ca63aa7.xml (deflated 38%) 2024-12-18T02:58:30.2270264Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-023280b5c2c86b4d.xml (deflated 37%) 2024-12-18T02:58:30.2271429Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-1b81ff78b68856e1.xml (deflated 38%) 2024-12-18T02:58:30.2272592Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-4009a83a357e75bf.xml (deflated 37%) 2024-12-18T02:58:30.2273753Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-df6bb090800210d7.xml (deflated 37%) 2024-12-18T02:58:30.2274917Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-b2d5c5232025b3e9.xml (deflated 38%) 2024-12-18T02:58:30.2276083Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-786b5c4fadd82a8b.xml (deflated 38%) 2024-12-18T02:58:30.2277249Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-c91b671bc95de8bb.xml (deflated 37%) 2024-12-18T02:58:30.2278471Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-737428a40730c635.xml (deflated 38%) 2024-12-18T02:58:30.2279622Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-3b5a573bb137d2f3.xml (deflated 37%) 2024-12-18T02:58:30.2280692Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-f16f3fcd81880b3b.xml (deflated 37%) 2024-12-18T02:58:30.2281758Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d71a1954d55785ec.xml (deflated 38%) 2024-12-18T02:58:30.2282828Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5e5f69276bf6be1e.xml (deflated 37%) 2024-12-18T02:58:30.2284147Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-68349bf4916d421c.xml (deflated 38%) 2024-12-18T02:58:30.2285360Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-27dc752b02e14cf8.xml (deflated 38%) 2024-12-18T02:58:30.2286646Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-4a5d875d4fa1b53f.xml (deflated 37%) 2024-12-18T02:58:30.2287855Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-29de91a2e86bf83e.xml (deflated 38%) 2024-12-18T02:58:30.2289057Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-b4481108e4ad12ac.xml (deflated 37%) 2024-12-18T02:58:30.2290252Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-2ba16604c227376a.xml (deflated 38%) 2024-12-18T02:58:30.2291448Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-9933850017805d96.xml (deflated 38%) 2024-12-18T02:58:30.2292655Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-3af4cebefc648e06.xml (deflated 38%) 2024-12-18T02:58:30.2293858Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-19062820bcc1d102.xml (deflated 36%) 2024-12-18T02:58:30.2295069Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-fdce66d151a0cb62.xml (deflated 37%) 2024-12-18T02:58:30.2296376Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-9dd8ab8e91eb33dd.xml (deflated 37%) 2024-12-18T02:58:30.2297663Z adding: test/test-reports/python-pytest/distributed.elastic.timer.local_timer_example/distributed.elastic.timer.local_timer_example-600084467ec586da.xml (deflated 49%) 2024-12-18T02:58:30.2299158Z adding: test/test-reports/python-pytest/distributed.elastic.utils.distributed_test/distributed.elastic.utils.distributed_test-41d202593a1ff3ce.xml (deflated 78%) 2024-12-18T02:58:30.2300521Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-16598bcccc1a8d4d.xml (deflated 44%) 2024-12-18T02:58:30.2301770Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cbb6ff8c9fb45576.xml (deflated 43%) 2024-12-18T02:58:30.2303010Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cbebe951a1668689.xml (deflated 44%) 2024-12-18T02:58:30.2304248Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3444d9e1f6e67ef3.xml (deflated 43%) 2024-12-18T02:58:30.2305479Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-293f170129ae27b5.xml (deflated 43%) 2024-12-18T02:58:30.2306715Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-740ea57dafa53983.xml (deflated 49%) 2024-12-18T02:58:30.2307956Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b88fe10fa5f7feae.xml (deflated 44%) 2024-12-18T02:58:30.2309205Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4b7b5e48c806138.xml (deflated 48%) 2024-12-18T02:58:30.2310506Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3b93b380cf10f727.xml (deflated 44%) 2024-12-18T02:58:30.2311745Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bc0d2870726e0aa4.xml (deflated 49%) 2024-12-18T02:58:30.2312975Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dcf841d8fa0efc20.xml (deflated 41%) 2024-12-18T02:58:30.2314228Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0e332dd2ced64b41.xml (deflated 48%) 2024-12-18T02:58:30.2315477Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-346a96f2e92878e6.xml (deflated 43%) 2024-12-18T02:58:30.2316770Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-12d6e96d8d45ebd9.xml (deflated 37%) 2024-12-18T02:58:30.2318023Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d16fba38e15febe6.xml (deflated 38%) 2024-12-18T02:58:30.2319263Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-77f46ff15f079d24.xml (deflated 38%) 2024-12-18T02:58:30.2320493Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8aef38f46097b985.xml (deflated 37%) 2024-12-18T02:58:30.2321732Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-158e41a69ba41ab6.xml (deflated 37%) 2024-12-18T02:58:30.2322964Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-113f5e963bf8c4e4.xml (deflated 37%) 2024-12-18T02:58:30.2324517Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c04fd0e77491412.xml (deflated 39%) 2024-12-18T02:58:30.2325917Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-441726646e89c2b2.xml (deflated 37%) 2024-12-18T02:58:30.2327319Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6c1a324ef2c9d402.xml (deflated 37%) 2024-12-18T02:58:30.2328715Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3025d086dc5453bd.xml (deflated 43%) 2024-12-18T02:58:30.2330106Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-31c5b7081beb2b8e.xml (deflated 38%) 2024-12-18T02:58:30.2331514Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ae5df72b5f3437c.xml (deflated 38%) 2024-12-18T02:58:30.2332900Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-14a1c08fb81adc03.xml (deflated 38%) 2024-12-18T02:58:30.2334307Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d584cc645cc5f7cd.xml (deflated 37%) 2024-12-18T02:58:30.2335703Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c6f5a5d1793a45d.xml (deflated 37%) 2024-12-18T02:58:30.2337194Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-20266132e3fa8e5c.xml (deflated 44%) 2024-12-18T02:58:30.2338430Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-80f52ee56ac60c39.xml (deflated 45%) 2024-12-18T02:58:30.2339670Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b5bed5bcd78430b1.xml (deflated 38%) 2024-12-18T02:58:30.2340913Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d22e1d4950a0c1f4.xml (deflated 44%) 2024-12-18T02:58:30.2342225Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c2a2c4db9d382589.xml (deflated 43%) 2024-12-18T02:58:30.2343461Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aadc5c74bf5a6f8e.xml (deflated 44%) 2024-12-18T02:58:30.2344706Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-750f91aab6c21841.xml (deflated 43%) 2024-12-18T02:58:30.2345938Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-469e3c02a2b42149.xml (deflated 43%) 2024-12-18T02:58:30.2347178Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3b8936f5f1db0762.xml (deflated 49%) 2024-12-18T02:58:30.2348476Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b8a002ebce66c0d8.xml (deflated 44%) 2024-12-18T02:58:30.2349723Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8031b35b8de9e406.xml (deflated 48%) 2024-12-18T02:58:30.2350961Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-262607dc86f6b4cf.xml (deflated 44%) 2024-12-18T02:58:30.2352197Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fd60db6df5d42e75.xml (deflated 49%) 2024-12-18T02:58:30.2353425Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-656193313719534a.xml (deflated 41%) 2024-12-18T02:58:30.2354653Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e5d8b4ea538a393.xml (deflated 48%) 2024-12-18T02:58:30.2355890Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f3f20e2ed558a515.xml (deflated 43%) 2024-12-18T02:58:30.2357134Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bf2cbafab654a9b3.xml (deflated 37%) 2024-12-18T02:58:30.2358538Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d7a4cc23adf73a5d.xml (deflated 37%) 2024-12-18T02:58:30.2359852Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17b0e1d613fe6545.xml (deflated 37%) 2024-12-18T02:58:30.2361163Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aff58aeaca36b77e.xml (deflated 37%) 2024-12-18T02:58:30.2362478Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ccfbc38f62f17f41.xml (deflated 37%) 2024-12-18T02:58:30.2364017Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-45d920995e90f9d8.xml (deflated 38%) 2024-12-18T02:58:30.2365588Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f7caf4d697ec0813.xml (deflated 38%) 2024-12-18T02:58:30.2366987Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c041f60bfa1217bc.xml (deflated 37%) 2024-12-18T02:58:30.2368375Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-87dc503d80d1f070.xml (deflated 37%) 2024-12-18T02:58:30.2369758Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-61f804c18b222702.xml (deflated 43%) 2024-12-18T02:58:30.2371160Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-49cce790c10fb12d.xml (deflated 38%) 2024-12-18T02:58:30.2372558Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0727e9ea30491427.xml (deflated 37%) 2024-12-18T02:58:30.2374059Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-df589ee6c9e7dd1a.xml (deflated 37%) 2024-12-18T02:58:30.2375463Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51a2b5fa3091c6b3.xml (deflated 37%) 2024-12-18T02:58:30.2376919Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3f84917d7e10be23.xml (deflated 37%) 2024-12-18T02:58:30.2378286Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f705475e6a8dc74b.xml (deflated 44%) 2024-12-18T02:58:30.2379529Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-302d9e50bf8a120a.xml (deflated 45%) 2024-12-18T02:58:30.2380839Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c9b3b068df355e9.xml (deflated 38%) 2024-12-18T02:58:30.2382080Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8ae17f6323d5c7ed.xml (deflated 37%) 2024-12-18T02:58:30.2383308Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1f374c0b3d2294d9.xml (deflated 39%) 2024-12-18T02:58:30.2384756Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8874a6f007dba804.xml (deflated 38%) 2024-12-18T02:58:30.2386075Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3a35bc925a3b0483.xml (deflated 43%) 2024-12-18T02:58:30.2387381Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-62e5decad85586c6.xml (deflated 46%) 2024-12-18T02:58:30.2388702Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-142b083fedb91272.xml (deflated 37%) 2024-12-18T02:58:30.2390024Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c36dfb64a894518f.xml (deflated 43%) 2024-12-18T02:58:30.2391337Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-60a6af06d098f3f9.xml (deflated 37%) 2024-12-18T02:58:30.2392637Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c0fa4cf2cf9940ef.xml (deflated 38%) 2024-12-18T02:58:30.2393955Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d772c87c927ea1c.xml (deflated 37%) 2024-12-18T02:58:30.2395379Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5816d090aefe09de.xml (deflated 38%) 2024-12-18T02:58:30.2396622Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8d9f23d38f5e85c2.xml (deflated 38%) 2024-12-18T02:58:30.2397848Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d91d63fd1193992b.xml (deflated 38%) 2024-12-18T02:58:30.2399079Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d008415616ff1df3.xml (deflated 38%) 2024-12-18T02:58:30.2400312Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c3b4ad1137dde02.xml (deflated 38%) 2024-12-18T02:58:30.2401558Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a083570cfee46507.xml (deflated 38%) 2024-12-18T02:58:30.2402793Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0d3b3ce681270664.xml (deflated 38%) 2024-12-18T02:58:30.2404401Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4faab26813f8e5e5.xml (deflated 38%) 2024-12-18T02:58:30.2405804Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-be332c1f4491affb.xml (deflated 38%) 2024-12-18T02:58:30.2407205Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a46ae381a7bc00f1.xml (deflated 38%) 2024-12-18T02:58:30.2408591Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-64b08f45e563943f.xml (deflated 37%) 2024-12-18T02:58:30.2409988Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2d217560a2debec9.xml (deflated 43%) 2024-12-18T02:58:30.2411444Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-91581796bff37e43.xml (deflated 43%) 2024-12-18T02:58:30.2412854Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fa02efd1a1666d52.xml (deflated 38%) 2024-12-18T02:58:30.2414260Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-765ecc20458df7f6.xml (deflated 44%) 2024-12-18T02:58:30.2415760Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fc8c599614fb8595.xml (deflated 43%) 2024-12-18T02:58:30.2417126Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9d711218f8829a78.xml (deflated 43%) 2024-12-18T02:58:30.2418359Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-091effc96828cd23.xml (deflated 45%) 2024-12-18T02:58:30.2419592Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8000a2553aaee5f0.xml (deflated 45%) 2024-12-18T02:58:30.2420825Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0154f8882725db67.xml (deflated 37%) 2024-12-18T02:58:30.2422070Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1d8b5b262de71c11.xml (deflated 37%) 2024-12-18T02:58:30.2423307Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1cc3306dc6f6de45.xml (deflated 39%) 2024-12-18T02:58:30.2424552Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4114c4f26ab1b30c.xml (deflated 39%) 2024-12-18T02:58:30.2425794Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fd75fee7d888d78e.xml (deflated 43%) 2024-12-18T02:58:30.2427032Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15eb7dc1a7090064.xml (deflated 46%) 2024-12-18T02:58:30.2428275Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-088be32d6ed1325b.xml (deflated 38%) 2024-12-18T02:58:30.2429502Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-20eaf361c3d1d650.xml (deflated 43%) 2024-12-18T02:58:30.2430743Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-40e30f97cdc5c920.xml (deflated 37%) 2024-12-18T02:58:30.2431975Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-084a00075763e4c0.xml (deflated 38%) 2024-12-18T02:58:30.2433208Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a80e476b08500c62.xml (deflated 37%) 2024-12-18T02:58:30.2434448Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5a0f0b92a9272195.xml (deflated 38%) 2024-12-18T02:58:30.2435748Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b4230ab0dff76bfd.xml (deflated 37%) 2024-12-18T02:58:30.2436977Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7e4074980f4e22a9.xml (deflated 39%) 2024-12-18T02:58:30.2438204Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-18eb8c204a83917d.xml (deflated 38%) 2024-12-18T02:58:30.2439431Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-85b84037d1dda7d3.xml (deflated 38%) 2024-12-18T02:58:30.2440667Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7d05b4bf9c25ec72.xml (deflated 37%) 2024-12-18T02:58:30.2441965Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f5ae4b536c7bcab0.xml (deflated 38%) 2024-12-18T02:58:30.2443208Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bb7b98d8a5ab2169.xml (deflated 38%) 2024-12-18T02:58:30.2444795Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bbd6896040be3999.xml (deflated 38%) 2024-12-18T02:58:30.2446195Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c5d3ce692f057c3.xml (deflated 38%) 2024-12-18T02:58:30.2447591Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-566f72826d1d94b2.xml (deflated 37%) 2024-12-18T02:58:30.2448985Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ee0848171b4a32e9.xml (deflated 43%) 2024-12-18T02:58:30.2450392Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bb08f40adb379c59.xml (deflated 43%) 2024-12-18T02:58:30.2451789Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-59b14f65afeac4d3.xml (deflated 38%) 2024-12-18T02:58:30.2453198Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-90bde8ded5f4eef3.xml (deflated 43%) 2024-12-18T02:58:30.2454603Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-369b9109b34b8a61.xml (deflated 43%) 2024-12-18T02:58:30.2456099Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-52c8073af8578cae.xml (deflated 43%) 2024-12-18T02:58:30.2457450Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6dcff6ad0815752a.xml (deflated 45%) 2024-12-18T02:58:30.2458689Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-29ce123c6952f6e8.xml (deflated 45%) 2024-12-18T02:58:30.2459924Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8c5a111e479b1d98.xml (deflated 37%) 2024-12-18T02:58:30.2461163Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fe7cce39fe417074.xml (deflated 46%) 2024-12-18T02:58:30.2462398Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cc32e4aaa05fddfa.xml (deflated 43%) 2024-12-18T02:58:30.2463642Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d3a63ad3285e85cb.xml (deflated 44%) 2024-12-18T02:58:30.2464880Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c38d73d61a5dd4f2.xml (deflated 44%) 2024-12-18T02:58:30.2466478Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17613ad901438b50.xml (deflated 43%) 2024-12-18T02:58:30.2467924Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ec1103f471be63a5.xml (deflated 38%) 2024-12-18T02:58:30.2469278Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-36c62c93eeec4969.xml (deflated 38%) 2024-12-18T02:58:30.2470624Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4029fa605bd29d81.xml (deflated 38%) 2024-12-18T02:58:30.2471985Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-23de8caeb378b4d4.xml (deflated 38%) 2024-12-18T02:58:30.2473329Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0ab9d79d9668d793.xml (deflated 44%) 2024-12-18T02:58:30.2474746Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-acfe06726c4d87f2.xml (deflated 37%) 2024-12-18T02:58:30.2476102Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d946f6582fa300d7.xml (deflated 37%) 2024-12-18T02:58:30.2477451Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-558920e8b83729fe.xml (deflated 38%) 2024-12-18T02:58:30.2478855Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fac2d624526a790f.xml (deflated 38%) 2024-12-18T02:58:30.2480090Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-27b92151a0c3c9ea.xml (deflated 37%) 2024-12-18T02:58:30.2481323Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a8c93305adc32645.xml (deflated 38%) 2024-12-18T02:58:30.2482564Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-996b6abe503f3565.xml (deflated 43%) 2024-12-18T02:58:30.2484033Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7e15ce751e6204c6.xml (deflated 38%) 2024-12-18T02:58:30.2485442Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3c6256fafcaadc14.xml (deflated 44%) 2024-12-18T02:58:30.2486854Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1977e837f7174fce.xml (deflated 38%) 2024-12-18T02:58:30.2488255Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-795ab60f23d42a93.xml (deflated 43%) 2024-12-18T02:58:30.2489676Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-25c53dfe7f991f64.xml (deflated 43%) 2024-12-18T02:58:30.2491094Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e3e4838417cd015.xml (deflated 38%) 2024-12-18T02:58:30.2492499Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-03a2378a19ceceff.xml (deflated 43%) 2024-12-18T02:58:30.2493913Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fc0daaf236d5be12.xml (deflated 38%) 2024-12-18T02:58:30.2495319Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7d08fa9a93c92d0f.xml (deflated 46%) 2024-12-18T02:58:30.2496865Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c88f17da182e82eb.xml (deflated 45%) 2024-12-18T02:58:30.2498117Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c69ec2fb194a3477.xml (deflated 37%) 2024-12-18T02:58:30.2499421Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b647cd90638cfb4.xml (deflated 46%) 2024-12-18T02:58:30.2500675Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f66f0e71712c047.xml (deflated 43%) 2024-12-18T02:58:30.2501929Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-113a5620de4c5da1.xml (deflated 44%) 2024-12-18T02:58:30.2503180Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-75a2d82ecc149c66.xml (deflated 44%) 2024-12-18T02:58:30.2504436Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-00ed72cb0bd42e3d.xml (deflated 43%) 2024-12-18T02:58:30.2505736Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e705f9cf29f473cb.xml (deflated 38%) 2024-12-18T02:58:30.2506989Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0745bb3eb3562192.xml (deflated 38%) 2024-12-18T02:58:30.2508242Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ed5471addb586b5f.xml (deflated 38%) 2024-12-18T02:58:30.2509496Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-555cb325f010ed1c.xml (deflated 38%) 2024-12-18T02:58:30.2510745Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca0618808a9485ef.xml (deflated 44%) 2024-12-18T02:58:30.2511996Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0bfd3377af85c130.xml (deflated 37%) 2024-12-18T02:58:30.2513250Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-93ed20a6bf1fc4b1.xml (deflated 37%) 2024-12-18T02:58:30.2514504Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9737686fe6538025.xml (deflated 38%) 2024-12-18T02:58:30.2515745Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c11eafb6b98420b8.xml (deflated 38%) 2024-12-18T02:58:30.2516999Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e77479cc53ff215.xml (deflated 37%) 2024-12-18T02:58:30.2518247Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eec54688828b5a9f.xml (deflated 38%) 2024-12-18T02:58:30.2519498Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-db999730da317f2e.xml (deflated 43%) 2024-12-18T02:58:30.2520752Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-84b4383e739cf9ee.xml (deflated 37%) 2024-12-18T02:58:30.2522003Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-580144e3d4320139.xml (deflated 44%) 2024-12-18T02:58:30.2523257Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-df0596decff451b6.xml (deflated 37%) 2024-12-18T02:58:30.2524904Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cd68302b2a0c5d72.xml (deflated 43%) 2024-12-18T02:58:30.2526326Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9bbf4dc4810cc69.xml (deflated 43%) 2024-12-18T02:58:30.2527728Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0b757f3b0f51166b.xml (deflated 38%) 2024-12-18T02:58:30.2529144Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-456ec7b164755021.xml (deflated 43%) 2024-12-18T02:58:30.2530621Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a58e34352ecc1eca.xml (deflated 37%) 2024-12-18T02:58:30.2532036Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b4f263fca14ef6f8.xml (deflated 46%) 2024-12-18T02:58:30.2533447Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b5c2a788ca487abc.xml (deflated 45%) 2024-12-18T02:58:30.2534858Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f72d20ca725e2389.xml (deflated 37%) 2024-12-18T02:58:30.2536546Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-08e887212e7a44d2.xml (deflated 50%) 2024-12-18T02:58:30.2538309Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-63f1003adbf63509.xml (deflated 49%) 2024-12-18T02:58:30.2539989Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-a2d7b4c3858a8c11.xml (deflated 40%) 2024-12-18T02:58:30.2541687Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-b0152dccca2ef24a.xml (deflated 40%) 2024-12-18T02:58:30.2543388Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-fbbdec38f44967ca.xml (deflated 40%) 2024-12-18T02:58:30.2545078Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-d188ec1f2b3633ed.xml (deflated 39%) 2024-12-18T02:58:30.2546773Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-851e4558cb95306d.xml (deflated 49%) 2024-12-18T02:58:30.2548461Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-ea16bdd880a4b3b1.xml (deflated 50%) 2024-12-18T02:58:30.2550148Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-b261973e5e754667.xml (deflated 40%) 2024-12-18T02:58:30.2551838Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-9b42907a4dec6a48.xml (deflated 40%) 2024-12-18T02:58:30.2553540Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-56a9ba3b21ac3fe6.xml (deflated 39%) 2024-12-18T02:58:30.2555236Z adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-61b98c282723e085.xml (deflated 39%) 2024-12-18T02:58:30.2556708Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-826db105f2e54ce5.xml (deflated 38%) 2024-12-18T02:58:30.2557955Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0708208eafe4dbad.xml (deflated 49%) 2024-12-18T02:58:30.2559211Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e728ea5f0d0817c2.xml (deflated 48%) 2024-12-18T02:58:30.2560465Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-574dfe0872c153c7.xml (deflated 38%) 2024-12-18T02:58:30.2561724Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-874407f6e75bdf0b.xml (deflated 49%) 2024-12-18T02:58:30.2563033Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-179454bf0aac04b7.xml (deflated 44%) 2024-12-18T02:58:30.2564591Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0d0138926d5b7abb.xml (deflated 44%) 2024-12-18T02:58:30.2566365Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5f1ec9adae5549cc.xml (deflated 44%) 2024-12-18T02:58:30.2567782Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-233e6a6b0fad4e11.xml (deflated 38%) 2024-12-18T02:58:30.2569183Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eadd73aa3d3523cd.xml (deflated 49%) 2024-12-18T02:58:30.2570717Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-772f047ca496c3fc.xml (deflated 38%) 2024-12-18T02:58:30.2572131Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2e63b828dd7833c8.xml (deflated 42%) 2024-12-18T02:58:30.2573539Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7733685b9fbea12b.xml (deflated 38%) 2024-12-18T02:58:30.2574947Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-917a58cca22cc6fa.xml (deflated 37%) 2024-12-18T02:58:30.2576445Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e0c11c39e3b41ff4.xml (deflated 38%) 2024-12-18T02:58:30.2577870Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ed14e483a5e9f7f0.xml (deflated 38%) 2024-12-18T02:58:30.2579130Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-387ce7fb89624f02.xml (deflated 37%) 2024-12-18T02:58:30.2580376Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-50f6661ac161c0fc.xml (deflated 38%) 2024-12-18T02:58:30.2581633Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fbd92ab494abf6b4.xml (deflated 37%) 2024-12-18T02:58:30.2582887Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fe60b16503699c27.xml (deflated 38%) 2024-12-18T02:58:30.2584139Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d7029989f9d34f14.xml (deflated 37%) 2024-12-18T02:58:30.2585389Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-793408dfa7ce52f9.xml (deflated 37%) 2024-12-18T02:58:30.2586649Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a700ec62b0b1d535.xml (deflated 38%) 2024-12-18T02:58:30.2587905Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e9a7264b4b29b12.xml (deflated 45%) 2024-12-18T02:58:30.2589153Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0803198e89f79785.xml (deflated 45%) 2024-12-18T02:58:30.2590394Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1f87717085c680a5.xml (deflated 37%) 2024-12-18T02:58:30.2591642Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2bf0786950229d73.xml (deflated 37%) 2024-12-18T02:58:30.2592883Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2cb3e174a173aa7d.xml (deflated 37%) 2024-12-18T02:58:30.2594140Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-71462f137d03168c.xml (deflated 38%) 2024-12-18T02:58:30.2595459Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8fcc768de6b7eb23.xml (deflated 38%) 2024-12-18T02:58:30.2596697Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-66ac0b549ab0c649.xml (deflated 38%) 2024-12-18T02:58:30.2597943Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b043f0682901f541.xml (deflated 38%) 2024-12-18T02:58:30.2599185Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15af7ef841e9552b.xml (deflated 49%) 2024-12-18T02:58:30.2600421Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2cd662a6b0f9179b.xml (deflated 48%) 2024-12-18T02:58:30.2601714Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-150c95f06acd9ef9.xml (deflated 38%) 2024-12-18T02:58:30.2602973Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9de7a681bb53f79f.xml (deflated 49%) 2024-12-18T02:58:30.2604582Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0cb9419c65b6e9f5.xml (deflated 44%) 2024-12-18T02:58:30.2605997Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bc6acda7ab2edd36.xml (deflated 44%) 2024-12-18T02:58:30.2607400Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4fb62b3923e67e24.xml (deflated 45%) 2024-12-18T02:58:30.2608807Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bbf8d70cac20437c.xml (deflated 37%) 2024-12-18T02:58:30.2610220Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-967b382da4473afd.xml (deflated 49%) 2024-12-18T02:58:30.2611625Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-982cf8591364bbc9.xml (deflated 38%) 2024-12-18T02:58:30.2613025Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5bd0673554a2e694.xml (deflated 42%) 2024-12-18T02:58:30.2614419Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-408ccbc3a833f07f.xml (deflated 38%) 2024-12-18T02:58:30.2615920Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-25377a4ef02876b0.xml (deflated 37%) 2024-12-18T02:58:30.2617281Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a155acdd177c6e8.xml (deflated 38%) 2024-12-18T02:58:30.2618535Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-da6a1281bdc339df.xml (deflated 38%) 2024-12-18T02:58:30.2619778Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bee70a6920d63e90.xml (deflated 37%) 2024-12-18T02:58:30.2621018Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4bcc2bd1285a78e8.xml (deflated 37%) 2024-12-18T02:58:30.2622267Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dd5fe214072134d6.xml (deflated 37%) 2024-12-18T02:58:30.2623501Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-778d335e4a3b8a86.xml (deflated 38%) 2024-12-18T02:58:30.2624739Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-824f590e180804ae.xml (deflated 37%) 2024-12-18T02:58:30.2626038Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c32e3406c849e4f.xml (deflated 37%) 2024-12-18T02:58:30.2627281Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8b8632e10f7d3942.xml (deflated 38%) 2024-12-18T02:58:30.2628525Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9dcea972e72134f2.xml (deflated 45%) 2024-12-18T02:58:30.2629774Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-47d7756d66be1184.xml (deflated 45%) 2024-12-18T02:58:30.2631013Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-82a91710045e49dd.xml (deflated 37%) 2024-12-18T02:58:30.2632307Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c41e26ee7e9453dd.xml (deflated 37%) 2024-12-18T02:58:30.2633550Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b264174e41de0af6.xml (deflated 37%) 2024-12-18T02:58:30.2634798Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d988ad5b5c05d293.xml (deflated 38%) 2024-12-18T02:58:30.2636039Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fdd947c94d21b851.xml (deflated 37%) 2024-12-18T02:58:30.2637294Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ffaabf0fbe100959.xml (deflated 38%) 2024-12-18T02:58:30.2638549Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8ac0d694d410bc65.xml (deflated 37%) 2024-12-18T02:58:30.2639803Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cc6b47678febbaa6.xml (deflated 38%) 2024-12-18T02:58:30.2641045Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-11849caafc903453.xml (deflated 38%) 2024-12-18T02:58:30.2642290Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-09af9b0363c81659.xml (deflated 38%) 2024-12-18T02:58:30.2643748Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5efb8d7d556b7b85.xml (deflated 38%) 2024-12-18T02:58:30.2645325Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b6fba4b1ae81e037.xml (deflated 45%) 2024-12-18T02:58:30.2646793Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b3daca3e863809cb.xml (deflated 49%) 2024-12-18T02:58:30.2648270Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8b35346265748a0d.xml (deflated 42%) 2024-12-18T02:58:30.2649724Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b501c63a82c08f07.xml (deflated 42%) 2024-12-18T02:58:30.2651133Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-513f1a903e819b72.xml (deflated 37%) 2024-12-18T02:58:30.2652523Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-78fc0b67298af050.xml (deflated 38%) 2024-12-18T02:58:30.2653926Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ee4c53905d427c6.xml (deflated 37%) 2024-12-18T02:58:30.2655322Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3c05cfff9e6c2678.xml (deflated 38%) 2024-12-18T02:58:30.2656917Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-236d904845c7d316.xml (deflated 38%) 2024-12-18T02:58:30.2658293Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-40dd64dc86e89cc9.xml (deflated 38%) 2024-12-18T02:58:30.2659604Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f89b410852fff7e9.xml (deflated 38%) 2024-12-18T02:58:30.2660920Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d2452f797094c1e0.xml (deflated 37%) 2024-12-18T02:58:30.2662230Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a5ba2c89b0135b3c.xml (deflated 38%) 2024-12-18T02:58:30.2663607Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c8ce4e2e09d26eb.xml (deflated 45%) 2024-12-18T02:58:30.2664916Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3ececda6bb89059.xml (deflated 38%) 2024-12-18T02:58:30.2666570Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a8034163446f6f1d.xml (deflated 37%) 2024-12-18T02:58:30.2667921Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-10e9315f2115c59e.xml (deflated 38%) 2024-12-18T02:58:30.2669291Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0bfa5188ae4d8b6b.xml (deflated 38%) 2024-12-18T02:58:30.2670658Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-df8e8398eacf4ab7.xml (deflated 43%) 2024-12-18T02:58:30.2672023Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-de89a46227c8f45c.xml (deflated 38%) 2024-12-18T02:58:30.2673387Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3bc4d90bc276c424.xml (deflated 38%) 2024-12-18T02:58:30.2674755Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9a6616b90eb5f95.xml (deflated 38%) 2024-12-18T02:58:30.2676106Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fbee9751646f2cab.xml (deflated 38%) 2024-12-18T02:58:30.2677450Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e9262bcc063a4d34.xml (deflated 38%) 2024-12-18T02:58:30.2678846Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-081900d77d53c249.xml (deflated 37%) 2024-12-18T02:58:30.2680072Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-87b0726c385e65e1.xml (deflated 38%) 2024-12-18T02:58:30.2681312Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a035aae9ceeab4d.xml (deflated 38%) 2024-12-18T02:58:30.2682567Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cbf10b44bfd5cc59.xml (deflated 38%) 2024-12-18T02:58:30.2684108Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca08080769a2e473.xml (deflated 38%) 2024-12-18T02:58:30.2685508Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-59691a1d1bc74256.xml (deflated 38%) 2024-12-18T02:58:30.2686908Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-14057f9d6ef108e5.xml (deflated 45%) 2024-12-18T02:58:30.2688320Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c8b9bc7ce644b35b.xml (deflated 49%) 2024-12-18T02:58:30.2689721Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d01cdbfb83aad3a0.xml (deflated 42%) 2024-12-18T02:58:30.2691239Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3fef74a2dd8de0bf.xml (deflated 42%) 2024-12-18T02:58:30.2692658Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-98d6b0a9aa26f5c8.xml (deflated 37%) 2024-12-18T02:58:30.2694059Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c8a705da04a2a753.xml (deflated 38%) 2024-12-18T02:58:30.2695467Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a576757b41cf63e5.xml (deflated 37%) 2024-12-18T02:58:30.2697015Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb344f1e70b58567.xml (deflated 38%) 2024-12-18T02:58:30.2698319Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f467e6861990de3f.xml (deflated 38%) 2024-12-18T02:58:30.2699560Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-db09d4ecd43c7452.xml (deflated 38%) 2024-12-18T02:58:30.2700795Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c71b4ed15705bf9.xml (deflated 38%) 2024-12-18T02:58:30.2702042Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b49a52499da52b28.xml (deflated 38%) 2024-12-18T02:58:30.2703287Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-647b1c5b829c3067.xml (deflated 38%) 2024-12-18T02:58:30.2704533Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b8079ede70d2ca5.xml (deflated 45%) 2024-12-18T02:58:30.2705781Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-40137992a32bac6d.xml (deflated 38%) 2024-12-18T02:58:30.2707031Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-90271a5a436a0e5e.xml (deflated 37%) 2024-12-18T02:58:30.2708272Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a99a7213610e695f.xml (deflated 38%) 2024-12-18T02:58:30.2709518Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-be5027489f629dda.xml (deflated 38%) 2024-12-18T02:58:30.2710754Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b42bd3d960e132f0.xml (deflated 43%) 2024-12-18T02:58:30.2711994Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-211ea4e12fc0dc84.xml (deflated 38%) 2024-12-18T02:58:30.2713241Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf7224f7604ce7f4.xml (deflated 37%) 2024-12-18T02:58:30.2714487Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-225fc3f55a5562d3.xml (deflated 38%) 2024-12-18T02:58:30.2715730Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9e008657db9df97.xml (deflated 37%) 2024-12-18T02:58:30.2716973Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bf57bd36e49cf469.xml (deflated 38%) 2024-12-18T02:58:30.2718219Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-90f6e118f6d6af02.xml (deflated 37%) 2024-12-18T02:58:30.2719468Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d2d2d926a4f8df81.xml (deflated 48%) 2024-12-18T02:58:30.2720767Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ff26b7fa2e4554e2.xml (deflated 38%) 2024-12-18T02:58:30.2722012Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c8b1b063a080e1ee.xml (deflated 49%) 2024-12-18T02:58:30.2723258Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4dde311b6f38168e.xml (deflated 38%) 2024-12-18T02:58:30.2724835Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7634eb17323bcf3c.xml (deflated 38%) 2024-12-18T02:58:30.2726240Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ffc57795d72a9c5.xml (deflated 37%) 2024-12-18T02:58:30.2727700Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-56f67bfa16e99b3e.xml (deflated 43%) 2024-12-18T02:58:30.2729102Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c0881c7458fadba4.xml (deflated 45%) 2024-12-18T02:58:30.2730506Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c501c2dcddc5a632.xml (deflated 38%) 2024-12-18T02:58:30.2731896Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1d7d1a5f42beea92.xml (deflated 38%) 2024-12-18T02:58:30.2733343Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-267758db9663355b.xml (deflated 37%) 2024-12-18T02:58:30.2734748Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-84fa9497df13ff1f.xml (deflated 37%) 2024-12-18T02:58:30.2736238Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-20b4f11604978cc2.xml (deflated 38%) 2024-12-18T02:58:30.2737604Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ee01b75330300af7.xml (deflated 38%) 2024-12-18T02:58:30.2738976Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3e61e400edaaa28.xml (deflated 37%) 2024-12-18T02:58:30.2740344Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d67f5c361b41181b.xml (deflated 38%) 2024-12-18T02:58:30.2741719Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f471ae16ee04dddc.xml (deflated 38%) 2024-12-18T02:58:30.2743079Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b1bb84f85d25028.xml (deflated 37%) 2024-12-18T02:58:30.2744443Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f4f2192345f7e2ec.xml (deflated 48%) 2024-12-18T02:58:30.2745795Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-823d83978fcfd305.xml (deflated 37%) 2024-12-18T02:58:30.2747148Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1dbd6137510fe4b9.xml (deflated 38%) 2024-12-18T02:58:30.2748514Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-be8e1639d750f8ec.xml (deflated 49%) 2024-12-18T02:58:30.2749868Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e0509feb6797ef21.xml (deflated 45%) 2024-12-18T02:58:30.2751230Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b55004efb3ed1d4e.xml (deflated 38%) 2024-12-18T02:58:30.2752597Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7c5bc6a05f9cd727.xml (deflated 42%) 2024-12-18T02:58:30.2754122Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1db5c5765133a026.xml (deflated 38%) 2024-12-18T02:58:30.2755488Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d3df66306c807f8a.xml (deflated 37%) 2024-12-18T02:58:30.2756727Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3b3d451dfcb48a34.xml (deflated 37%) 2024-12-18T02:58:30.2757980Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8fe8e917a4b16ea8.xml (deflated 49%) 2024-12-18T02:58:30.2759425Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-374c28271060f1cb.xml (deflated 38%) 2024-12-18T02:58:30.2760797Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7c1128aaa086d62f.xml (deflated 48%) 2024-12-18T02:58:30.2762121Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-729ec9dcbe7f1ce7.xml (deflated 38%) 2024-12-18T02:58:30.2763496Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1fe0e1b0ad3052e9.xml (deflated 38%) 2024-12-18T02:58:30.2765053Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-715fa5dc9e4f2721.xml (deflated 38%) 2024-12-18T02:58:30.2766603Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b3e4bf312ddfbb2d.xml (deflated 43%) 2024-12-18T02:58:30.2768000Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c3762124dcfdfbf.xml (deflated 45%) 2024-12-18T02:58:30.2769402Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-03c928484e49846b.xml (deflated 38%) 2024-12-18T02:58:30.2770801Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a07e370e2bf1e12.xml (deflated 38%) 2024-12-18T02:58:30.2772211Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dfcefcd85801d73c.xml (deflated 37%) 2024-12-18T02:58:30.2773611Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a222776a556e743c.xml (deflated 37%) 2024-12-18T02:58:30.2775024Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-16db439fc9d6e8bd.xml (deflated 38%) 2024-12-18T02:58:30.2776505Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-657f56260f351300.xml (deflated 38%) 2024-12-18T02:58:30.2778005Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ce6127af2be8b42e.xml (deflated 38%) 2024-12-18T02:58:30.2779243Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ed30ca856b943797.xml (deflated 38%) 2024-12-18T02:58:30.2780489Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e744babdbe139454.xml (deflated 38%) 2024-12-18T02:58:30.2781732Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4cb4ff464cf92f2f.xml (deflated 38%) 2024-12-18T02:58:30.2782978Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-874c9ede34b14bed.xml (deflated 48%) 2024-12-18T02:58:30.2784224Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b87883c265de8276.xml (deflated 37%) 2024-12-18T02:58:30.2785468Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-29705caeab318c23.xml (deflated 38%) 2024-12-18T02:58:30.2786816Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e9c4acc54211b8ea.xml (deflated 49%) 2024-12-18T02:58:30.2788073Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a91ef4bf5889947.xml (deflated 45%) 2024-12-18T02:58:30.2789305Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5658546ea19d6136.xml (deflated 39%) 2024-12-18T02:58:30.2790558Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-df412e2e7215b69b.xml (deflated 42%) 2024-12-18T02:58:30.2791814Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-74ebe65ab788cc95.xml (deflated 38%) 2024-12-18T02:58:30.2793138Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-91f24c65d1b0b724.xml (deflated 37%) 2024-12-18T02:58:30.2794399Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fc280b79a16cbf6b.xml (deflated 37%) 2024-12-18T02:58:30.2795870Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-9be27bb63204c96a.xml (deflated 39%) 2024-12-18T02:58:30.2797559Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-4c84279dc34448c3.xml (deflated 39%) 2024-12-18T02:58:30.2799247Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-6517cd6c4c28bb15.xml (deflated 50%) 2024-12-18T02:58:30.2800936Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-f2329279f7ac5a1c.xml (deflated 50%) 2024-12-18T02:58:30.2802627Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-5543af6f8c599b0f.xml (deflated 51%) 2024-12-18T02:58:30.2804609Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-a6ee2497ce03815a.xml (deflated 51%) 2024-12-18T02:58:30.2806501Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-25074914211301ee.xml (deflated 39%) 2024-12-18T02:58:30.2808406Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-55fd8945f8f91b94.xml (deflated 39%) 2024-12-18T02:58:30.2810319Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-0a84a1b2d7ef2e12.xml (deflated 50%) 2024-12-18T02:58:30.2812228Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-67330ed6e3a0465a.xml (deflated 50%) 2024-12-18T02:58:30.2814123Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-e0d51627d918e1bb.xml (deflated 51%) 2024-12-18T02:58:30.2816101Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-5b77addf7938f5dd.xml (deflated 51%) 2024-12-18T02:58:30.2859935Z ##[group]Run # Remove any previous usage logs if they exist 2024-12-18T02:58:30.2860477Z # Remove any previous usage logs if they exist 2024-12-18T02:58:30.2860904Z rm -f logs-*.zip 2024-12-18T02:58:30.2861442Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-12-18T02:58:30.2862148Z # so check to see if the file exists first 2024-12-18T02:58:30.2862550Z if [ -f 'usage_log.txt' ]; then 2024-12-18T02:58:30.2862978Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-12-18T02:58:30.2863379Z fi 2024-12-18T02:58:30.2863808Z if find "test/test-reports" -name "*.log" 2>/dev/null | grep -q .; then 2024-12-18T02:58:30.2864441Z  zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' 2024-12-18T02:58:30.2864893Z fi 2024-12-18T02:58:30.2870616Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:58:30.2871049Z env: 2024-12-18T02:58:30.2871299Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:30.2871680Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:30.2872443Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:30.2873225Z FILE_SUFFIX: test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_34566463052 2024-12-18T02:58:30.2873760Z ##[endgroup] 2024-12-18T02:58:30.3397666Z adding: usage_log.txt (deflated 97%) 2024-12-18T02:58:30.3478591Z adding: test/test-reports/distributed.test_inductor_collectives_1.1_63740d089a9e43f8_.log (deflated 87%) 2024-12-18T02:58:30.3480126Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_binary_cmp_1.1_ee9735d507b5c4ac_.log (deflated 96%) 2024-12-18T02:58:30.3481948Z adding: test/test-reports/distributed._tensor.test_common_rules_1.1_4a28379d84814dad_.log (deflated 91%) 2024-12-18T02:58:30.3483416Z adding: test/test-reports/distributed._tensor.test_init_1.1_1383db0fc1b52b4c_.log (deflated 89%) 2024-12-18T02:58:30.3484585Z adding: test/test-reports/distributed._tensor.test_xla_integration_1.1_135b28974c321750_.log (deflated 66%) 2024-12-18T02:58:30.3485610Z adding: test/test-reports/distributed._tensor.test_convolution_ops_1.1_2449f6e121be4711_.log (deflated 59%) 2024-12-18T02:58:30.3486613Z adding: test/test-reports/distributed._tensor.test_math_ops_1.1_4f12d41d9d4b04b7_.log (deflated 77%) 2024-12-18T02:58:30.3487592Z adding: test/test-reports/distributed._tensor.test_pointwise_ops_1.1_48c6bf496cc72b65_.log (deflated 69%) 2024-12-18T02:58:30.3488658Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_init_1.1_2650a7d3e24133c7_.log (deflated 86%) 2024-12-18T02:58:30.3489828Z adding: test/test-reports/distributed._tensor.test_embedding_ops_1.1_28c25ffea555a0c4_.log (deflated 65%) 2024-12-18T02:58:30.3490783Z adding: test/test-reports/distributed._tensor.test_matrix_ops_1.1_8f913be26be6f2d6_.log (deflated 73%) 2024-12-18T02:58:30.3501075Z adding: test/test-reports/distributed.rpc.cuda.test_tensorpipe_agent_2.2_ac43784569642ac7_.log (deflated 97%) 2024-12-18T02:58:30.3502201Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_embedding_1.1_3e1aaec7d8b82ad0_.log (deflated 92%) 2024-12-18T02:58:30.3503283Z adding: test/test-reports/distributed.flight_recorder.test_fr_analysis_1.1_6c9cda0b572ecd26_.log (deflated 55%) 2024-12-18T02:58:30.3504306Z adding: test/test-reports/distributed._tensor.test_dtensor_compile_1.1_cc99e17d88e4c2d6_.log (deflated 84%) 2024-12-18T02:58:30.3506245Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_compile_1.1_e0e054511287f3aa_.log (deflated 89%) 2024-12-18T02:58:30.3507286Z adding: test/test-reports/distributed._tools.test_mem_tracker_1.1_7c59a2b4b6fdcba4_.log (deflated 61%) 2024-12-18T02:58:30.3508200Z adding: test/test-reports/distributed.fsdp.test_fsdp_fx_1.1_e2bee0f2efd5a070_.log (deflated 53%) 2024-12-18T02:58:30.3509092Z adding: test/test-reports/distributed._tensor.test_api_1.1_69d830f6f3e7705a_.log (deflated 71%) 2024-12-18T02:58:30.3509959Z adding: test/test-reports/distributed.test_backends_1.1_913e5a0b16a41327_.log (deflated 57%) 2024-12-18T02:58:30.3510961Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_logging_1.1_d5e95c2e90d518d6_.log (deflated 53%) 2024-12-18T02:58:30.3512689Z adding: test/test-reports/distributed.test_symmetric_memory_1.1_835a17eca150a31e_.log (deflated 93%) 2024-12-18T02:58:30.3513636Z adding: test/test-reports/distributed.pipelining.test_stage_1.1_0d605cb6674317dc_.log (deflated 81%) 2024-12-18T02:58:30.3514602Z adding: test/test-reports/distributed._tools.test_memory_tracker_1.1_be743280618cb84b_.log (deflated 54%) 2024-12-18T02:58:30.3515730Z adding: test/test-reports/distributed._composable.test_composability.test_pp_composability_1.1_2cd743d308019d00_.log (deflated 90%) 2024-12-18T02:58:30.3516835Z adding: test/test-reports/distributed.test_distributed_spawn_3.9_eaa7f13919c45ec7_.log (deflated 12%) 2024-12-18T02:58:30.3517783Z adding: test/test-reports/distributed.test_distributed_spawn_3.9_13492a65ea9e2555_.log (deflated 12%) 2024-12-18T02:58:30.3521114Z adding: test/test-reports/distributed.test_distributed_spawn_3.9_e8ba8c21e1046a92_.log (deflated 94%) 2024-12-18T02:58:30.3526475Z adding: test/test-reports/distributed.test_distributed_spawn_3.9_d9fa5c1202276875_.log (deflated 94%) 2024-12-18T02:58:30.3531548Z adding: test/test-reports/distributed.test_distributed_spawn_3.9_f654a7e552e5fac5_.log (deflated 94%) 2024-12-18T02:58:30.3536591Z adding: test/test-reports/distributed.test_distributed_spawn_3.9_483c9d7ef80d0881_.log (deflated 94%) 2024-12-18T02:58:30.3537705Z adding: test/test-reports/distributed.test_distributed_spawn_6.9_765f398be333954e_.log (deflated 12%) 2024-12-18T02:58:30.3538652Z adding: test/test-reports/distributed.test_distributed_spawn_6.9_f2ea893aec4eb2b1_.log (deflated 12%) 2024-12-18T02:58:30.3543015Z adding: test/test-reports/distributed.test_distributed_spawn_6.9_5522ec59a8ed4c64_.log (deflated 92%) 2024-12-18T02:58:30.3548864Z adding: test/test-reports/distributed.test_distributed_spawn_6.9_b27eb46f6162468f_.log (deflated 92%) 2024-12-18T02:58:30.3554281Z adding: test/test-reports/distributed.test_distributed_spawn_6.9_03572a2149d9e5ed_.log (deflated 93%) 2024-12-18T02:58:30.3559529Z adding: test/test-reports/distributed.test_distributed_spawn_6.9_4572ef768c581112_.log (deflated 93%) 2024-12-18T02:58:30.3560646Z adding: test/test-reports/distributed.test_distributed_spawn_9.9_6bb790fe7c50bcb5_.log (deflated 12%) 2024-12-18T02:58:30.3561580Z adding: test/test-reports/distributed.test_distributed_spawn_9.9_63555b7fa99fd534_.log (deflated 12%) 2024-12-18T02:58:30.3565462Z adding: test/test-reports/distributed.test_distributed_spawn_9.9_9f5b9f6e0f791298_.log (deflated 93%) 2024-12-18T02:58:30.3570496Z adding: test/test-reports/distributed.test_distributed_spawn_9.9_9c71e338cc606e79_.log (deflated 93%) 2024-12-18T02:58:30.3575384Z adding: test/test-reports/distributed.test_distributed_spawn_9.9_b385968c42fb2a86_.log (deflated 94%) 2024-12-18T02:58:30.3580124Z adding: test/test-reports/distributed.test_distributed_spawn_9.9_9f536d088f285976_.log (deflated 94%) 2024-12-18T02:58:30.3581339Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_5018b3eab9aa6c28_.log (stored 0%) 2024-12-18T02:58:30.3582490Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_30f683676ae9324d_.log (stored 0%) 2024-12-18T02:58:30.3583656Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_7fd958a77a9e3abe_.log (deflated 90%) 2024-12-18T02:58:30.3584841Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_573abeb7eb33ff11_.log (deflated 90%) 2024-12-18T02:58:30.3586018Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_1f41face016ee93a_.log (deflated 90%) 2024-12-18T02:58:30.3587194Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_6e87de6eb9862dbf_.log (deflated 90%) 2024-12-18T02:58:30.3588287Z adding: test/test-reports/distributed.elastic.test_control_plane_1.1_4a50fa4d621f9879_.log (deflated 75%) 2024-12-18T02:58:30.3589294Z adding: test/test-reports/distributed.checkpoint.test_tp_checkpoint_1.1_aabbd809296862fa_.log (deflated 73%) 2024-12-18T02:58:30.3590489Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_overlap_1.1_e0a826f963696653_.log (deflated 58%) 2024-12-18T02:58:30.3591545Z adding: test/test-reports/distributed.checkpoint.test_planner_1.1_ffc376c01e92aab4_.log (deflated 68%) 2024-12-18T02:58:30.3592625Z adding: test/test-reports/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion_1.1_09e73bdf9e27a5e2_.log (deflated 88%) 2024-12-18T02:58:30.3593737Z adding: test/test-reports/distributed.checkpoint.test_state_dict_utils_1.1_f57de718209b4e31_.log (deflated 74%) 2024-12-18T02:58:30.3594718Z adding: test/test-reports/distributed.fsdp.test_shard_utils_1.1_972544faf9186e1c_.log (deflated 65%) 2024-12-18T02:58:30.3595701Z adding: test/test-reports/distributed.checkpoint.test_compatibility_1.1_0cf0457c18f43cdd_.log (deflated 65%) 2024-12-18T02:58:30.3596828Z adding: test/test-reports/distributed.checkpoint.test_hsdp_checkpoint_1.1_b1ab383ffea376f3_.log (deflated 92%) 2024-12-18T02:58:30.3597767Z adding: test/test-reports/distributed.test_nccl_1.1_91c90a830ac96836_.log (deflated 74%) 2024-12-18T02:58:30.3598636Z adding: test/test-reports/distributed.test_functional_api_1.1_df0011489f1d29aa_.log (deflated 86%) 2024-12-18T02:58:30.3599573Z adding: test/test-reports/distributed._tensor.test_optimizers_1.1_5dbf1fcfcbcd64c3_.log (deflated 83%) 2024-12-18T02:58:30.3600632Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_grad_scaler_1.1_6e9a6577b23aced9_.log (deflated 57%) 2024-12-18T02:58:30.3601672Z adding: test/test-reports/distributed.checkpoint.test_utils_1.1_0a5f786566a15efa_.log (deflated 65%) 2024-12-18T02:58:30.3602596Z adding: test/test-reports/distributed._tensor.test_utils_1.1_a3d950cb9d128b6f_.log (deflated 85%) 2024-12-18T02:58:30.3614305Z adding: test/test-reports/distributed.test_c10d_nccl_1.3_37b8a1b552bdd372_.log (deflated 93%) 2024-12-18T02:58:30.3615530Z adding: test/test-reports/distributed.fsdp.test_fsdp_overlap_1.1_d18592d87c2b6534_.log (deflated 68%) 2024-12-18T02:58:30.3651298Z adding: test/test-reports/distributed.fsdp.test_fsdp_optim_state_1.1_bd4971d31bd9df2c_.log (deflated 98%) 2024-12-18T02:58:30.3652681Z adding: test/test-reports/distributed.checkpoint.test_file_system_checkpoint_1.1_cca3bc9fbdea75c5_.log (deflated 90%) 2024-12-18T02:58:30.3654352Z adding: test/test-reports/distributed.fsdp.test_fsdp_apply_1.1_e1118e58e6b9e6c7_.log (deflated 81%) 2024-12-18T02:58:30.3655824Z adding: test/test-reports/distributed.fsdp.test_fsdp_flatten_params_1.1_1ff9b083961fc955_.log (deflated 83%) 2024-12-18T02:58:30.3663733Z adding: test/test-reports/distributed.optim.test_zero_redundancy_optimizer_1.1_fe9285a434702bb0_.log (deflated 95%) 2024-12-18T02:58:30.3664994Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_memory_1.1_922270446896e03e_.log (deflated 62%) 2024-12-18T02:58:30.3666206Z adding: test/test-reports/distributed.fsdp.test_checkpoint_wrapper_1.1_af9752128a8dbd8e_.log (deflated 70%) 2024-12-18T02:58:30.3667155Z adding: test/test-reports/distributed.fsdp.test_utils_1.1_0a2eb8c3934cec37_.log (deflated 66%) 2024-12-18T02:58:30.3669900Z adding: test/test-reports/distributed.fsdp.test_hsdp_dtensor_state_dict_1.1_c36bcfd4e91ef5bf_.log (deflated 94%) 2024-12-18T02:58:30.3670933Z adding: test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_c6e3b3eb4f45cc2d_.log (deflated 66%) 2024-12-18T02:58:30.3672130Z adding: test/test-reports/distributed.fsdp.test_fsdp_checkpoint_1.1_b376c86cc359d178_.log (deflated 90%) 2024-12-18T02:58:30.3673145Z adding: test/test-reports/distributed.algorithms.test_join_1.1_0f741634d511e795_.log (deflated 84%) 2024-12-18T02:58:30.3675345Z adding: test/test-reports/distributed.fsdp.test_fsdp_hybrid_shard_1.1_3ce571e7b20dadee_.log (deflated 91%) 2024-12-18T02:58:30.3678494Z adding: test/test-reports/distributed.checkpoint.test_file_system_checkpoint_cpu_1.1_1485b7711dd75d67_.log (deflated 94%) 2024-12-18T02:58:30.3680052Z adding: test/test-reports/distributed.test_data_parallel_1.1_8857beeb3b02e639_.log (deflated 87%) 2024-12-18T02:58:30.3692448Z adding: test/test-reports/distributed.fsdp.test_fsdp_core_1.2_c7d701e3e9880e09_.log (deflated 97%) 2024-12-18T02:58:30.3693391Z adding: test/test-reports/distributed.rpc.test_faulty_agent_1.1_07ed557f81a14759_.log (stored 0%) 2024-12-18T02:58:30.3694351Z adding: test/test-reports/distributed.fsdp.test_fsdp_fine_tune_1.1_8acaab5825bd30d4_.log (deflated 86%) 2024-12-18T02:58:30.3695804Z adding: test/test-reports/distributed._shard.sharded_optim.test_sharded_optim_1.1_970bc5344113e429_.log (deflated 92%) 2024-12-18T02:58:30.3698253Z adding: test/test-reports/distributed.fsdp.test_fsdp_unshard_params_1.1_f5b3e12ffc11afdc_.log (deflated 85%) 2024-12-18T02:58:30.3702993Z adding: test/test-reports/distributed.fsdp.test_fsdp_mixed_precision_1.1_970b5c2f2308582e_.log (deflated 93%) 2024-12-18T02:58:30.3704474Z adding: test/test-reports/distributed.test_c10d_spawn_gloo_1.1_3dd7b9dcc81c070e_.log (deflated 93%) 2024-12-18T02:58:30.3705810Z adding: test/test-reports/distributed.test_c10d_spawn_ucc_1.1_24352c650089b7ec_.log (deflated 89%) 2024-12-18T02:58:30.3707400Z adding: test/test-reports/distributed.test_c10d_spawn_nccl_1.1_6e28777d7d990ce1_.log (deflated 92%) 2024-12-18T02:58:30.3718434Z adding: test/test-reports/distributed.test_store_1.1_20ade82a32f16def_.log (deflated 95%) 2024-12-18T02:58:30.3719569Z adding: test/test-reports/distributed.elastic.timer.local_timer_example_1.1_f53082b28c197712_.log (deflated 68%) 2024-12-18T02:58:30.3722538Z adding: test/test-reports/distributed.elastic.utils.distributed_test_1.1_2d0b1e3224b2546b_.log (deflated 94%) 2024-12-18T02:58:30.3754888Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-12-18T02:58:30.3755481Z # Remove any previous debugging artifacts if they exist 2024-12-18T02:58:30.3755933Z rm -f debug-*.zip 2024-12-18T02:58:30.3756252Z if [ -d 'test/debug' ]; then 2024-12-18T02:58:30.3756661Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-12-18T02:58:30.3757049Z fi 2024-12-18T02:58:30.3762204Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:58:30.3762596Z env: 2024-12-18T02:58:30.3762832Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:30.3763193Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:30.3764099Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:30.3764914Z FILE_SUFFIX: test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_34566463052 2024-12-18T02:58:30.3765626Z ##[endgroup] 2024-12-18T02:58:30.3941627Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T02:58:30.3941987Z with: 2024-12-18T02:58:30.3942222Z s3-bucket: gha-artifacts 2024-12-18T02:58:30.3942564Z s3-prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:58:30.3942940Z retention-days: 14 2024-12-18T02:58:30.3943213Z if-no-files-found: warn 2024-12-18T02:58:30.3943505Z path: test-jsons-*.zip 2024-12-18T02:58:30.3943795Z name: artifact 2024-12-18T02:58:30.3944028Z region: us-east-1 2024-12-18T02:58:30.3944275Z env: 2024-12-18T02:58:30.3944501Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:30.3944857Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:30.3945461Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:30.3945986Z ##[endgroup] 2024-12-18T02:58:30.7762588Z NOTE: s3-prefix specified, ignoring name parameter 2024-12-18T02:58:30.7763417Z With the provided path, there will be 1 file uploaded 2024-12-18T02:58:30.7764126Z Uploading to s3 prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:58:30.7805802Z Starting upload of test-jsons-test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_34566463052.zip 2024-12-18T02:58:30.8993240Z Finished upload of test-jsons-test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_34566463052.zip 2024-12-18T02:58:30.9164777Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T02:58:30.9165371Z with: 2024-12-18T02:58:30.9165795Z s3-bucket: gha-artifacts 2024-12-18T02:58:30.9166186Z s3-prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:58:30.9166603Z retention-days: 14 2024-12-18T02:58:30.9166909Z if-no-files-found: error 2024-12-18T02:58:30.9167240Z path: test-reports-*.zip 2024-12-18T02:58:30.9167541Z name: artifact 2024-12-18T02:58:30.9167816Z region: us-east-1 2024-12-18T02:58:30.9168087Z env: 2024-12-18T02:58:30.9168343Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:30.9168740Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:30.9169408Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:30.9170017Z ##[endgroup] 2024-12-18T02:58:31.2618576Z NOTE: s3-prefix specified, ignoring name parameter 2024-12-18T02:58:31.2619138Z With the provided path, there will be 1 file uploaded 2024-12-18T02:58:31.2619669Z Uploading to s3 prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:58:31.2662641Z Starting upload of test-reports-test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_34566463052.zip 2024-12-18T02:58:31.3974451Z Finished upload of test-reports-test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_34566463052.zip 2024-12-18T02:58:31.4153391Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T02:58:31.4153772Z with: 2024-12-18T02:58:31.4154007Z s3-bucket: gha-artifacts 2024-12-18T02:58:31.4154374Z s3-prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:58:31.4154772Z retention-days: 14 2024-12-18T02:58:31.4155060Z if-no-files-found: ignore 2024-12-18T02:58:31.4155366Z path: logs-*.zip 2024-12-18T02:58:31.4155619Z name: artifact 2024-12-18T02:58:31.4155879Z region: us-east-1 2024-12-18T02:58:31.4156162Z env: 2024-12-18T02:58:31.4156403Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:31.4156781Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:31.4157408Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:31.4157990Z ##[endgroup] 2024-12-18T02:58:31.7586443Z NOTE: s3-prefix specified, ignoring name parameter 2024-12-18T02:58:31.7587660Z With the provided path, there will be 1 file uploaded 2024-12-18T02:58:31.7588232Z Uploading to s3 prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:58:31.7629865Z Starting upload of logs-test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_34566463052.zip 2024-12-18T02:58:31.9394788Z Finished upload of logs-test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_34566463052.zip 2024-12-18T02:58:31.9566642Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T02:58:31.9567044Z with: 2024-12-18T02:58:31.9567296Z s3-bucket: gha-artifacts 2024-12-18T02:58:31.9567682Z s3-prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:58:31.9568124Z retention-days: 14 2024-12-18T02:58:31.9568436Z if-no-files-found: ignore 2024-12-18T02:58:31.9568759Z path: debug-*.zip 2024-12-18T02:58:31.9569025Z name: artifact 2024-12-18T02:58:31.9569297Z region: us-east-1 2024-12-18T02:58:31.9569583Z env: 2024-12-18T02:58:31.9569840Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:31.9570239Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:31.9570903Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:31.9571515Z ##[endgroup] 2024-12-18T02:58:32.2935780Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-12-18T02:58:32.3118659Z ##[group]Run # shellcheck disable=SC2156 2024-12-18T02:58:32.3119071Z # shellcheck disable=SC2156 2024-12-18T02:58:32.3119729Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-12-18T02:58:32.3127049Z shell: /usr/bin/bash -e {0} 2024-12-18T02:58:32.3127383Z env: 2024-12-18T02:58:32.3127652Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:32.3128053Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:32.3128725Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:32.3129340Z ##[endgroup] 2024-12-18T02:58:32.5541413Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@release/2.6 2024-12-18T02:58:32.5541935Z with: 2024-12-18T02:58:32.5542165Z env: 2024-12-18T02:58:32.5542391Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:32.5542880Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:32.5543484Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:32.5544026Z ##[endgroup] 2024-12-18T02:58:32.5566312Z ##[group]Run set -eou pipefail 2024-12-18T02:58:32.5566698Z set -eou pipefail 2024-12-18T02:58:32.5567022Z  2024-12-18T02:58:32.5567605Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-12-18T02:58:32.5568162Z for _ in $(seq 1440); do 2024-12-18T02:58:32.5568563Z  # Break if no ssh session exists anymore 2024-12-18T02:58:32.5568977Z  if [ "$(who)" = "" ]; then 2024-12-18T02:58:32.5569330Z  break 2024-12-18T02:58:32.5569646Z  fi 2024-12-18T02:58:32.5569915Z  echo "." 2024-12-18T02:58:32.5570193Z  sleep 5 2024-12-18T02:58:32.5570477Z done 2024-12-18T02:58:32.5576665Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:58:32.5577205Z env: 2024-12-18T02:58:32.5577473Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:32.5577850Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:32.5578475Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:32.5579054Z ##[endgroup] 2024-12-18T02:58:32.5603092Z Holding runner for 2 hours until all ssh sessions have logged out 2024-12-18T02:58:32.5709525Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T02:58:32.5710122Z # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T02:58:32.5710582Z # shellcheck disable=SC2046 2024-12-18T02:58:32.5710937Z docker stop $(docker ps -q) || true 2024-12-18T02:58:32.5711307Z # Prune all of the docker images 2024-12-18T02:58:32.5711659Z docker system prune -af 2024-12-18T02:58:32.5716837Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:58:32.5717236Z env: 2024-12-18T02:58:32.5717466Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:58:32.5717811Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:58:32.5718415Z DOCKER_CONTAINER_ID: ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:32.5718955Z ##[endgroup] 2024-12-18T02:58:33.2707654Z ca3abfad5faf 2024-12-18T02:58:33.7018713Z Deleted Containers: 2024-12-18T02:58:33.7019251Z ca3abfad5faf9489fe0e0c64e029c00c86aaccc03bfbc9da77813a2a0577df0f 2024-12-18T02:58:33.7019651Z 2024-12-18T02:58:39.0154832Z Deleted Images: 2024-12-18T02:58:39.0155835Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T02:58:39.0157444Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9@sha256:52918bcdb0e370d1b9413099049464295e6c36e0a09cff6cff645b519e6da502 2024-12-18T02:58:39.0158691Z deleted: sha256:166dd14a00627fbd0d4e062a733b69b733b63b1ac12ec3bd5285db8a8befaaf5 2024-12-18T02:58:39.0159436Z deleted: sha256:2e828a4f851a71a08b9f06842643025da951b3d1a26bc40be945b5dc00f5015f 2024-12-18T02:58:39.0160166Z deleted: sha256:d27246a7406fc9214ce9db75df344bdd8e0a31300352beaf8ee0e0668a1ace5b 2024-12-18T02:58:39.0160921Z deleted: sha256:68c9fa6063c3f2de81e86b8de02d9c520f91cebed9343aecd84d4621681e108b 2024-12-18T02:58:39.0161692Z deleted: sha256:7fba561e060d45fd930ced565d615e1e1f5b5e1266212caaddcbdce3b6ce2157 2024-12-18T02:58:39.0162438Z deleted: sha256:5b5728232cece21786a91a2b7109215ddacbea370a452a5725759be3f9782f36 2024-12-18T02:58:39.0163170Z deleted: sha256:dc257cb01440638b647e8f88d00e791443a113161d0d4f469015690bab702c18 2024-12-18T02:58:39.0164524Z deleted: sha256:3ee1ba287c278eaf901c09c7e521f0a405bbb9fb8062aa51674fb9ea3ab18789 2024-12-18T02:58:39.0165471Z deleted: sha256:2d66661697c95587c70e4f77dc01731d10271a2a2d5ae52b240971a0554e6d0e 2024-12-18T02:58:39.0166229Z deleted: sha256:eaa304b04f52287b7bdefe4e276ed9444a9f2924aab757a4112d857162d00aae 2024-12-18T02:58:39.0166991Z deleted: sha256:4b999f756c8d5d7c8444b89e5466899b4bbd4f3adec20d788c443754a4cb2cf6 2024-12-18T02:58:39.0167753Z deleted: sha256:031bf045cac35b9a41c06ca4b85ca3419cf104dcaef931590111a4cbc99225d5 2024-12-18T02:58:39.0168510Z deleted: sha256:4a6da4e6593178794969f47a35ec4c667bd2a620051de682f4f6476200528ad7 2024-12-18T02:58:39.0169412Z deleted: sha256:d49018d0fd9477af622a59aae4a080650d82601c3ff10f5e413d54b61605cbc7 2024-12-18T02:58:39.0170174Z deleted: sha256:eaeb72d8e211a811359bbae6cdb944e506296e3593e02a7845d1de2b020db64d 2024-12-18T02:58:39.0170938Z deleted: sha256:4443a93278f4af70ebbc86c00611e1229086bcd73ba6f46cfbac959f5a3ea13e 2024-12-18T02:58:39.0171704Z deleted: sha256:7da4d3af5a970120b30198ce7c0d5758fd4180887250d2e1d20b1bb19a7ac558 2024-12-18T02:58:39.0172503Z deleted: sha256:51d1e0cf0c3b813663526e6d529bfb7f0374c2474f5a9581ccfd66f6dd722384 2024-12-18T02:58:39.0173263Z deleted: sha256:6d152e7f1f9932d4367ae21fbdcaf7e300d258e862c3c3104defca22e1f67f82 2024-12-18T02:58:39.0174020Z deleted: sha256:5f3d462f5b23f997262b6a535998f846bc071e1cbfb2039b507881fb8d873078 2024-12-18T02:58:39.0174777Z deleted: sha256:b5aa4897eb3166e0b560548c1ccbf716422a3b134bc63aa4da378db45a578506 2024-12-18T02:58:39.0175517Z deleted: sha256:19e8d1b6e7033c3cf97c2220cc96c9d4860bc5a09ad25b706a7260176331e59b 2024-12-18T02:58:39.0176383Z deleted: sha256:f82be94fab4a78f872875e44a2a54d01d6b7c7c9a73ad193593549037b0c9c6c 2024-12-18T02:58:39.0177121Z deleted: sha256:45e1395581d647e78b6fae987a785c08acea4b6176c97188b5a5010e9004d947 2024-12-18T02:58:39.0177845Z deleted: sha256:6a116d37e354cd470a6dee7600017fd2c240860da82a19939bd1451416af0972 2024-12-18T02:58:39.0178583Z deleted: sha256:96d8d1f880e7957a35c4060d1ae56eaa05c1e73ff5e7128cc5b42531a996ffeb 2024-12-18T02:58:39.0179319Z deleted: sha256:a186beea437cb709b244be99550467206b5bdcb190e032c091f15a6a065ddc36 2024-12-18T02:58:39.0180068Z deleted: sha256:e5f8bcff21a8fbb2d507150a82a1b26ebcec9caf9ee8fa7dce844574ef8da2ad 2024-12-18T02:58:39.0180808Z deleted: sha256:13091955acb9200fc139971ba2434c90657de11ff38ac954e35f3461fa4a6b8a 2024-12-18T02:58:39.0181535Z deleted: sha256:be10df8394bb253584f3f2f4c84a28bd53744e8c6892f1efcb4b12b67abe23da 2024-12-18T02:58:39.0182377Z deleted: sha256:ca58fcd63e3576eddbb8aaef07131e18631c53e4e9b6e8d9dbec5b8ddfa7726a 2024-12-18T02:58:39.0183093Z deleted: sha256:57c7d561a44e019b14a0f4aae3354ec1734b088b539cab6fb0e8ecca850ee714 2024-12-18T02:58:39.0183815Z deleted: sha256:a960411fa30719c90510a13403dcde638fbee3cbbb4ab71f14e52f40e9dd4533 2024-12-18T02:58:39.0184528Z deleted: sha256:8a2ef497a8ab2819802ad459d2dafe5195a85d11b38ef6d6d4d4fe872d9c6b6b 2024-12-18T02:58:39.0185517Z deleted: sha256:cbf589ae40cc4496ffa854c9108d58905a2ea5ac3162669b3b4606287b751d28 2024-12-18T02:58:39.0186229Z deleted: sha256:88c378f8d5104ea7a03bf49a1a7db4df53518519236345cf29bc2df76377d0ac 2024-12-18T02:58:39.0186949Z deleted: sha256:db2a844a68747daa51bb2b28c083df270c19bfbcb7e038723a1db1bacf20462e 2024-12-18T02:58:39.0187826Z deleted: sha256:f548fd4a27a545794a88afc9f9f3460001efb98677ae40f9737387bffedd901d 2024-12-18T02:58:39.0188647Z deleted: sha256:42f4cf4d61dad97d4c5f3bd5cb9374fe260e5166e36f4ae2d00a9c59127abb3a 2024-12-18T02:58:39.0189401Z deleted: sha256:1e465635bd263f73940e977a6d9ab8b3beceabc5eaa9d08c65a28aafdf72abf0 2024-12-18T02:58:39.0190137Z deleted: sha256:b890305c75102c7bd0a645b98b0e1893d45433319a1394c457afb7ffc3ac1070 2024-12-18T02:58:39.0190875Z deleted: sha256:c9d4902893199761be10cecc8c1de9ce1242b5df3c161bc452e5ffcf2d3bff38 2024-12-18T02:58:39.0191617Z deleted: sha256:254c14be940e16bbbe4d6da795ac3a987a11fde9842d614f31c1caae910c7263 2024-12-18T02:58:39.0192357Z deleted: sha256:137abe9e7f1c1726800d1ae709a40b92e48267f52dd7fa7b966be15345a94850 2024-12-18T02:58:39.0193211Z deleted: sha256:771d7f9e8bf6f532b16c3daa3a9167d4030f919eec29b18d99387ad1d1a277cb 2024-12-18T02:58:39.0193954Z deleted: sha256:22ee16d76a83cee720de0aafa52532ff1f3335a9048ff43877cee2aa6a370f5f 2024-12-18T02:58:39.0194694Z deleted: sha256:dbbad5d1e6d1785ee41424b56457a16a1d882ff1c09f41486b443b8c1e7c7eb4 2024-12-18T02:58:39.0195432Z deleted: sha256:df9a3657ce5f83f7ab775078ff74a472438fc5c21e1e0de53cc66509788e12bc 2024-12-18T02:58:39.0196173Z deleted: sha256:245eebe495c94ddc69ab83759089101d2edfdbf90474a3cb772e7bf1cce893bd 2024-12-18T02:58:39.0197140Z deleted: sha256:235e5b71d4fbb1eaf808e6fd789604752f653c1176b8acb1ba97604f35418396 2024-12-18T02:58:39.0197893Z deleted: sha256:c96d511cee1be7af8a8fedcab42cac5233cf66be05d188a2ba32ba4db1d2f715 2024-12-18T02:58:39.0198762Z deleted: sha256:704760c42e98a22e0f5ce40f10bdc3baec3e4ca679c2b046e90e6e12228a7164 2024-12-18T02:58:39.0199581Z deleted: sha256:78ec6b9f299f8114edc65d21c0c2e730f8423fcc496b819fe25df22af7477bd6 2024-12-18T02:58:39.0200270Z deleted: sha256:3d2201aa2c2a2db1e240c872ac8b84d4c92eed331c5c0053eaea4a8d292dd29a 2024-12-18T02:58:39.0200959Z deleted: sha256:2f8151f506d01c5be81b7547d7c20a36a2f30f6713adecaa91adb633fab9d809 2024-12-18T02:58:39.0201628Z deleted: sha256:d1f62ced0636553b623466ced7568917bb2b453997168ceb2635926f5c069ae6 2024-12-18T02:58:39.0202288Z deleted: sha256:bcb018d693dd3d8346bc43a253a42de37aad6ae11f42ee748dd9fdf1d3b259fc 2024-12-18T02:58:39.0203131Z deleted: sha256:40f224dcd4345d71537636d5c3bdb9c0659430771d9ac90c9e7cf7da30fabdd3 2024-12-18T02:58:39.0204110Z deleted: sha256:113e5e34c2859806f96efe4d47aa5e0d438e3f72211b7d539de8750bbfed7a98 2024-12-18T02:58:39.0204863Z deleted: sha256:a4786d6f71e2eec542ec8c4ff41191481de085b7e1b4e0139995017b6b7d6477 2024-12-18T02:58:39.0205631Z deleted: sha256:84faffca53886fb81b780b885baa8b6b4a4c16795bf93c9eed6f165c8d333545 2024-12-18T02:58:39.0206391Z deleted: sha256:22f20f90960f1663da6dbde7ea9b18026e0e1b03006890d56f64fc084a7550e3 2024-12-18T02:58:39.0207125Z deleted: sha256:d53c4a66cd75e088673985522b5ba099599960ac06350e59cfd330572327bff2 2024-12-18T02:58:39.0207883Z deleted: sha256:e95f4ec4da5fef7e54c22d270b51d9077a8c577a82bf0304e200fce309551084 2024-12-18T02:58:39.0208645Z deleted: sha256:ddbe476a0432522a73e2b052342ebc6f8610f7437c076ff6bf7e4501dc79efbc 2024-12-18T02:58:39.0209404Z deleted: sha256:ad7883a4c2954aebf555ea86e018fee1cc12d159b9e4d149263eb92c7cd21708 2024-12-18T02:58:39.0210178Z deleted: sha256:656ce67e45b8dab0cf77d1d21cd380f5ddfa64bfd486ebaa47b3af7c9b9df7b5 2024-12-18T02:58:39.0210955Z deleted: sha256:ae5754644efe96c6b01bbe838a7db9aac49333bf618c0a05160cdd0dd4b38fe8 2024-12-18T02:58:39.0211709Z deleted: sha256:23058ae084a55e5fd11db1aabe073f1886a1047be70eeb78dada3e6d4949c4d7 2024-12-18T02:58:39.0212479Z deleted: sha256:55c98b096fa43eb44df581de08d34a280f6eb359a04b1562fedccca5d56748e3 2024-12-18T02:58:39.0213243Z deleted: sha256:4a1d549deb2a6a9e8c52225d513aefeffbe0097e1eeb8217ca34963724db0b53 2024-12-18T02:58:39.0214003Z deleted: sha256:89e3b1c20a2d617b0ed103dc78296d410c1611d2044d8426fcc757b64c3113fe 2024-12-18T02:58:39.0214753Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-12-18T02:58:39.0215515Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-12-18T02:58:39.0216434Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-12-18T02:58:39.0217101Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-12-18T02:58:39.0217783Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-12-18T02:58:39.0218466Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-12-18T02:58:39.0219149Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-12-18T02:58:39.0219818Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-12-18T02:58:39.0220486Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-12-18T02:58:39.0221241Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-12-18T02:58:39.0221913Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-12-18T02:58:39.0222310Z 2024-12-18T02:58:39.0222438Z Total reclaimed space: 36.91GB 2024-12-18T02:58:39.0288052Z Post job cleanup. 2024-12-18T02:58:39.0338412Z Post job cleanup. 2024-12-18T02:58:39.1113771Z [command]/usr/bin/git version 2024-12-18T02:58:39.1149218Z git version 2.40.1 2024-12-18T02:58:39.1193991Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/31b25301-79db-49ea-9c38-4e856dbeaf18' before making global git config changes 2024-12-18T02:58:39.1195344Z Adding repository directory to the temporary git global config as a safe directory 2024-12-18T02:58:39.1196574Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T02:58:39.1226114Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-12-18T02:58:39.1253239Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-12-18T02:58:39.1571012Z Entering 'android/libs/fbjni' 2024-12-18T02:58:39.1629261Z Entering 'third_party/FP16' 2024-12-18T02:58:39.1684826Z Entering 'third_party/FXdiv' 2024-12-18T02:58:39.1742210Z Entering 'third_party/NNPACK' 2024-12-18T02:58:39.1802036Z Entering 'third_party/NVTX' 2024-12-18T02:58:39.1859275Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T02:58:39.1917592Z Entering 'third_party/XNNPACK' 2024-12-18T02:58:39.1991963Z Entering 'third_party/benchmark' 2024-12-18T02:58:39.2047140Z Entering 'third_party/composable_kernel' 2024-12-18T02:58:39.2111583Z Entering 'third_party/cpp-httplib' 2024-12-18T02:58:39.2167206Z Entering 'third_party/cpuinfo' 2024-12-18T02:58:39.2225494Z Entering 'third_party/cudnn_frontend' 2024-12-18T02:58:39.2281952Z Entering 'third_party/cutlass' 2024-12-18T02:58:39.2350592Z Entering 'third_party/eigen' 2024-12-18T02:58:39.2409834Z Entering 'third_party/fbgemm' 2024-12-18T02:58:39.2466364Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T02:58:39.2520672Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T02:58:39.2583060Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T02:58:39.2651082Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T02:58:39.2709131Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T02:58:39.2765485Z Entering 'third_party/flatbuffers' 2024-12-18T02:58:39.2826600Z Entering 'third_party/fmt' 2024-12-18T02:58:39.2885473Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T02:58:39.2943457Z Entering 'third_party/gloo' 2024-12-18T02:58:39.3000403Z Entering 'third_party/googletest' 2024-12-18T02:58:39.3055391Z Entering 'third_party/ideep' 2024-12-18T02:58:39.3115121Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T02:58:39.3185620Z Entering 'third_party/ittapi' 2024-12-18T02:58:39.3243964Z Entering 'third_party/kineto' 2024-12-18T02:58:39.3302214Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T02:58:39.3365636Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T02:58:39.3427336Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T02:58:39.3483981Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T02:58:39.3547614Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T02:58:39.3605633Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T02:58:39.3664361Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T02:58:39.3720162Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T02:58:39.3782072Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T02:58:39.3846057Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T02:58:39.3905684Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T02:58:39.3959800Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T02:58:39.4024366Z Entering 'third_party/mimalloc' 2024-12-18T02:58:39.4080218Z Entering 'third_party/nccl/nccl' 2024-12-18T02:58:39.4134730Z Entering 'third_party/nlohmann' 2024-12-18T02:58:39.4195462Z Entering 'third_party/onnx' 2024-12-18T02:58:39.4270268Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T02:58:39.4330721Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T02:58:39.4390565Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T02:58:39.4447742Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T02:58:39.4504316Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T02:58:39.4565667Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T02:58:39.4625544Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T02:58:39.4685707Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T02:58:39.4742418Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T02:58:39.4800199Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T02:58:39.4861959Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T02:58:39.4920535Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T02:58:39.5000166Z Entering 'third_party/pocketfft' 2024-12-18T02:58:39.5054395Z Entering 'third_party/protobuf' 2024-12-18T02:58:39.5117278Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T02:58:39.5173803Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T02:58:39.5232926Z Entering 'third_party/psimd' 2024-12-18T02:58:39.5288295Z Entering 'third_party/pthreadpool' 2024-12-18T02:58:39.5344457Z Entering 'third_party/pybind11' 2024-12-18T02:58:39.5402252Z Entering 'third_party/python-peachpy' 2024-12-18T02:58:39.5457539Z Entering 'third_party/sleef' 2024-12-18T02:58:39.5517492Z Entering 'third_party/tensorpipe' 2024-12-18T02:58:39.5581030Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T02:58:39.5636631Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T02:58:39.5697501Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T02:58:39.5757939Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T02:58:39.5811361Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T02:58:39.5884604Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-12-18T02:58:39.5905161Z http.https://github.com/.extraheader 2024-12-18T02:58:39.5912359Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-12-18T02:58:39.5939658Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-12-18T02:58:39.6245755Z Entering 'android/libs/fbjni' 2024-12-18T02:58:39.6284801Z http.https://github.com/.extraheader 2024-12-18T02:58:39.6323817Z Entering 'third_party/FP16' 2024-12-18T02:58:39.6361566Z http.https://github.com/.extraheader 2024-12-18T02:58:39.6397158Z Entering 'third_party/FXdiv' 2024-12-18T02:58:39.6436566Z http.https://github.com/.extraheader 2024-12-18T02:58:39.6477655Z Entering 'third_party/NNPACK' 2024-12-18T02:58:39.6519552Z http.https://github.com/.extraheader 2024-12-18T02:58:39.6555853Z Entering 'third_party/NVTX' 2024-12-18T02:58:39.6595159Z http.https://github.com/.extraheader 2024-12-18T02:58:39.6630414Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T02:58:39.6670888Z http.https://github.com/.extraheader 2024-12-18T02:58:39.6706725Z Entering 'third_party/XNNPACK' 2024-12-18T02:58:39.6744752Z http.https://github.com/.extraheader 2024-12-18T02:58:39.6798003Z Entering 'third_party/benchmark' 2024-12-18T02:58:39.6839876Z http.https://github.com/.extraheader 2024-12-18T02:58:39.6876080Z Entering 'third_party/composable_kernel' 2024-12-18T02:58:39.6916233Z http.https://github.com/.extraheader 2024-12-18T02:58:39.6958403Z Entering 'third_party/cpp-httplib' 2024-12-18T02:58:39.6996576Z http.https://github.com/.extraheader 2024-12-18T02:58:39.7030857Z Entering 'third_party/cpuinfo' 2024-12-18T02:58:39.7068468Z http.https://github.com/.extraheader 2024-12-18T02:58:39.7104855Z Entering 'third_party/cudnn_frontend' 2024-12-18T02:58:39.7142299Z http.https://github.com/.extraheader 2024-12-18T02:58:39.7180438Z Entering 'third_party/cutlass' 2024-12-18T02:58:39.7219441Z http.https://github.com/.extraheader 2024-12-18T02:58:39.7264531Z Entering 'third_party/eigen' 2024-12-18T02:58:39.7304435Z http.https://github.com/.extraheader 2024-12-18T02:58:39.7342561Z Entering 'third_party/fbgemm' 2024-12-18T02:58:39.7381607Z http.https://github.com/.extraheader 2024-12-18T02:58:39.7419421Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T02:58:39.7454656Z http.https://github.com/.extraheader 2024-12-18T02:58:39.7489384Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T02:58:39.7525472Z http.https://github.com/.extraheader 2024-12-18T02:58:39.7564885Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T02:58:39.7603636Z http.https://github.com/.extraheader 2024-12-18T02:58:39.7646178Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T02:58:39.7684635Z http.https://github.com/.extraheader 2024-12-18T02:58:39.7721049Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T02:58:39.7758955Z http.https://github.com/.extraheader 2024-12-18T02:58:39.7798317Z Entering 'third_party/flatbuffers' 2024-12-18T02:58:39.7840079Z http.https://github.com/.extraheader 2024-12-18T02:58:39.7882103Z Entering 'third_party/fmt' 2024-12-18T02:58:39.7920578Z http.https://github.com/.extraheader 2024-12-18T02:58:39.7956881Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T02:58:39.7995924Z http.https://github.com/.extraheader 2024-12-18T02:58:39.8030802Z Entering 'third_party/gloo' 2024-12-18T02:58:39.8067584Z http.https://github.com/.extraheader 2024-12-18T02:58:39.8104689Z Entering 'third_party/googletest' 2024-12-18T02:58:39.8142725Z http.https://github.com/.extraheader 2024-12-18T02:58:39.8180520Z Entering 'third_party/ideep' 2024-12-18T02:58:39.8219109Z http.https://github.com/.extraheader 2024-12-18T02:58:39.8252963Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T02:58:39.8290727Z http.https://github.com/.extraheader 2024-12-18T02:58:39.8345247Z Entering 'third_party/ittapi' 2024-12-18T02:58:39.8384547Z http.https://github.com/.extraheader 2024-12-18T02:58:39.8421025Z Entering 'third_party/kineto' 2024-12-18T02:58:39.8459775Z http.https://github.com/.extraheader 2024-12-18T02:58:39.8494788Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T02:58:39.8530710Z http.https://github.com/.extraheader 2024-12-18T02:58:39.8565944Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T02:58:39.8603426Z http.https://github.com/.extraheader 2024-12-18T02:58:39.8639871Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T02:58:39.8676239Z http.https://github.com/.extraheader 2024-12-18T02:58:39.8716876Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T02:58:39.8755469Z http.https://github.com/.extraheader 2024-12-18T02:58:39.8791719Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T02:58:39.8828090Z http.https://github.com/.extraheader 2024-12-18T02:58:39.8861933Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T02:58:39.8900313Z http.https://github.com/.extraheader 2024-12-18T02:58:39.8943714Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T02:58:39.8981830Z http.https://github.com/.extraheader 2024-12-18T02:58:39.9018800Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T02:58:39.9054164Z http.https://github.com/.extraheader 2024-12-18T02:58:39.9090423Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T02:58:39.9126667Z http.https://github.com/.extraheader 2024-12-18T02:58:39.9165491Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T02:58:39.9203126Z http.https://github.com/.extraheader 2024-12-18T02:58:39.9245191Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T02:58:39.9282826Z http.https://github.com/.extraheader 2024-12-18T02:58:39.9319570Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T02:58:39.9357466Z http.https://github.com/.extraheader 2024-12-18T02:58:39.9394489Z Entering 'third_party/mimalloc' 2024-12-18T02:58:39.9432878Z http.https://github.com/.extraheader 2024-12-18T02:58:39.9466884Z Entering 'third_party/nccl/nccl' 2024-12-18T02:58:39.9507327Z http.https://github.com/.extraheader 2024-12-18T02:58:39.9543776Z Entering 'third_party/nlohmann' 2024-12-18T02:58:39.9582189Z http.https://github.com/.extraheader 2024-12-18T02:58:39.9620978Z Entering 'third_party/onnx' 2024-12-18T02:58:39.9659939Z http.https://github.com/.extraheader 2024-12-18T02:58:39.9714213Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T02:58:39.9755839Z http.https://github.com/.extraheader 2024-12-18T02:58:39.9794292Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T02:58:39.9834007Z http.https://github.com/.extraheader 2024-12-18T02:58:39.9870095Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T02:58:39.9909714Z http.https://github.com/.extraheader 2024-12-18T02:58:39.9944432Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T02:58:39.9981314Z http.https://github.com/.extraheader 2024-12-18T02:58:40.0023303Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T02:58:40.0061125Z http.https://github.com/.extraheader 2024-12-18T02:58:40.0098968Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T02:58:40.0140646Z http.https://github.com/.extraheader 2024-12-18T02:58:40.0182515Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T02:58:40.0220739Z http.https://github.com/.extraheader 2024-12-18T02:58:40.0259254Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T02:58:40.0295655Z http.https://github.com/.extraheader 2024-12-18T02:58:40.0335955Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T02:58:40.0373660Z http.https://github.com/.extraheader 2024-12-18T02:58:40.0416366Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T02:58:40.0453657Z http.https://github.com/.extraheader 2024-12-18T02:58:40.0492261Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T02:58:40.0526052Z http.https://github.com/.extraheader 2024-12-18T02:58:40.0566386Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T02:58:40.0604439Z http.https://github.com/.extraheader 2024-12-18T02:58:40.0668817Z Entering 'third_party/pocketfft' 2024-12-18T02:58:40.0707398Z http.https://github.com/.extraheader 2024-12-18T02:58:40.0742752Z Entering 'third_party/protobuf' 2024-12-18T02:58:40.0782944Z http.https://github.com/.extraheader 2024-12-18T02:58:40.0823266Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T02:58:40.0860449Z http.https://github.com/.extraheader 2024-12-18T02:58:40.0894196Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T02:58:40.0930689Z http.https://github.com/.extraheader 2024-12-18T02:58:40.0970150Z Entering 'third_party/psimd' 2024-12-18T02:58:40.1007406Z http.https://github.com/.extraheader 2024-12-18T02:58:40.1044212Z Entering 'third_party/pthreadpool' 2024-12-18T02:58:40.1083721Z http.https://github.com/.extraheader 2024-12-18T02:58:40.1119912Z Entering 'third_party/pybind11' 2024-12-18T02:58:40.1158207Z http.https://github.com/.extraheader 2024-12-18T02:58:40.1194336Z Entering 'third_party/python-peachpy' 2024-12-18T02:58:40.1233077Z http.https://github.com/.extraheader 2024-12-18T02:58:40.1267206Z Entering 'third_party/sleef' 2024-12-18T02:58:40.1307318Z http.https://github.com/.extraheader 2024-12-18T02:58:40.1347457Z Entering 'third_party/tensorpipe' 2024-12-18T02:58:40.1387403Z http.https://github.com/.extraheader 2024-12-18T02:58:40.1423449Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T02:58:40.1461459Z http.https://github.com/.extraheader 2024-12-18T02:58:40.1505112Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T02:58:40.1542133Z http.https://github.com/.extraheader 2024-12-18T02:58:40.1576512Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T02:58:40.1613041Z http.https://github.com/.extraheader 2024-12-18T02:58:40.1648181Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T02:58:40.1687213Z http.https://github.com/.extraheader 2024-12-18T02:58:40.1720789Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T02:58:40.1757811Z http.https://github.com/.extraheader 2024-12-18T02:58:40.1893439Z A job completed hook has been configured by the self-hosted runner administrator 2024-12-18T02:58:40.1916298Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-12-18T02:58:40.1921492Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:58:40.1921905Z ##[endgroup] 2024-12-18T02:58:40.2006195Z [!ALERT!] Swap in detected! [!ALERT!] 2024-12-18T02:58:52.0025800Z [!ALERT!] Swap out detected [!ALERT!] 2024-12-18T02:59:10.3822518Z Cleaning up orphan processes