2024-11-01T16:31:43.3800550Z Current runner version: '2.320.0' 2024-11-01T16:31:43.3807052Z Runner name: 'i-07c6a72643395af71' 2024-11-01T16:31:43.3807874Z Runner group name: 'Default' 2024-11-01T16:31:43.3808680Z Machine name: 'ip-10-0-40-123' 2024-11-01T16:31:43.3825836Z Testing runner upgrade compatibility 2024-11-01T16:31:43.5068007Z ##[group]GITHUB_TOKEN Permissions 2024-11-01T16:31:43.5070339Z Actions: read 2024-11-01T16:31:43.5070852Z Attestations: read 2024-11-01T16:31:43.5071338Z Checks: read 2024-11-01T16:31:43.5071843Z Contents: read 2024-11-01T16:31:43.5072279Z Deployments: read 2024-11-01T16:31:43.5072866Z Discussions: read 2024-11-01T16:31:43.5073397Z Issues: read 2024-11-01T16:31:43.5073853Z Metadata: read 2024-11-01T16:31:43.5074393Z Packages: read 2024-11-01T16:31:43.5074858Z Pages: read 2024-11-01T16:31:43.5075309Z PullRequests: read 2024-11-01T16:31:43.5075834Z RepositoryProjects: read 2024-11-01T16:31:43.5076422Z SecurityEvents: read 2024-11-01T16:31:43.5076937Z Statuses: read 2024-11-01T16:31:43.5077632Z ##[endgroup] 2024-11-01T16:31:43.5081122Z Secret source: Actions 2024-11-01T16:31:43.5081889Z Prepare workflow directory 2024-11-01T16:31:43.6048004Z Prepare all required actions 2024-11-01T16:31:43.6224149Z Getting action download info 2024-11-01T16:31:43.8277917Z Download action repository 'pytorch/test-infra@main' (SHA:49fb39b5efb49007791d74b09885044cd64544bf) 2024-11-01T16:31:44.1781365Z Download action repository 'pytorch/pytorch@main' (SHA:b57b4b7f9b168389def15ea06a4dcf9e5f6f4f04) 2024-11-01T16:31:47.3232585Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-11-01T16:31:47.4350895Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-11-01T16:31:47.7193930Z Getting action download info 2024-11-01T16:31:47.8959926Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-11-01T16:31:48.0222461Z Getting action download info 2024-11-01T16:31:48.1186388Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2024-11-01T16:31:48.2597887Z Getting action download info 2024-11-01T16:31:48.3719084Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-11-01T16:31:48.4880227Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/138766/merge (368da7510f5d755034681777a4e5ae6d33c07b38) 2024-11-01T16:31:48.4882720Z ##[group] Inputs 2024-11-01T16:31:48.4883151Z build-environment: linux-jammy-py3.9-gcc11 2024-11-01T16:31:48.4886960Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "default", "shard": 2, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "default", "shard": 3, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "default", "shard": 4, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "default", "shard": 5, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "docs_test", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "jit_legacy", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "backwards_compat", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge"}]} 2024-11-01T16:31:48.4891286Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:31:48.4892300Z sync-tag: 2024-11-01T16:31:48.4893038Z timeout-minutes: 240 2024-11-01T16:31:48.4893376Z use-gha: 2024-11-01T16:31:48.4893663Z dashboard-tag: 2024-11-01T16:31:48.4893983Z s3-bucket: gha-artifacts 2024-11-01T16:31:48.4894329Z aws-role-to-assume: 2024-11-01T16:31:48.4894660Z ##[endgroup] 2024-11-01T16:31:48.4895635Z Complete job name: linux-jammy-py3.9-gcc11 / test (distributed, 1, 2, linux.2xlarge) 2024-11-01T16:31:48.5541366Z A job started hook has been configured by the self-hosted runner administrator 2024-11-01T16:31:48.5690313Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-11-01T16:31:48.5700003Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:31:48.5700548Z ##[endgroup] 2024-11-01T16:31:49.7025124Z Runner Type: linux.2xlarge 2024-11-01T16:31:49.7025632Z Instance Type: c5.2xlarge 2024-11-01T16:31:49.7025989Z AMI Name: unknown 2024-11-01T16:31:49.7026557Z AMI ID: ami-0fff1b9a61dec8a5f 2024-11-01T16:31:55.7263290Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-11-01T16:31:55.7263902Z with: 2024-11-01T16:31:55.7264611Z github-secret: *** 2024-11-01T16:31:55.7265588Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-11-01T16:31:55.7266704Z activate-with-label: false 2024-11-01T16:31:55.7267075Z label: with-ssh 2024-11-01T16:31:55.7267410Z remove-existing-keys: true 2024-11-01T16:31:55.7267785Z fail-silently: true 2024-11-01T16:31:55.7268087Z env: 2024-11-01T16:31:55.7268336Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:31:55.7268674Z ##[endgroup] 2024-11-01T16:31:55.8188473Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-11-01T16:31:56.0627731Z Grabbing public ssh keys from https://github.com/c00w.keys 2024-11-01T16:31:56.1129428Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-11-01T16:31:56.1146017Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-11-01T16:31:56.1171711Z Login using: ssh ec2-user@ec2-3-94-203-122.compute-1.amazonaws.com 2024-11-01T16:31:56.1172540Z All testing is done inside the container, to start an interactive session run: 2024-11-01T16:31:56.1173452Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-11-01T16:31:56.1295060Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-11-01T16:31:56.1295624Z with: 2024-11-01T16:31:56.1295899Z submodules: recursive 2024-11-01T16:31:56.1296230Z fetch-depth: 0 2024-11-01T16:31:56.1296508Z env: 2024-11-01T16:31:56.1296753Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:31:56.1297087Z ##[endgroup] 2024-11-01T16:31:56.1498504Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:31:56.1499880Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:31:56.1509736Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:31:56.1510257Z env: 2024-11-01T16:31:56.1510524Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:31:56.1510865Z ##[endgroup] 2024-11-01T16:31:56.1597287Z ##[group]Run retry () { 2024-11-01T16:31:56.1597698Z retry () { 2024-11-01T16:31:56.1598207Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-11-01T16:31:56.1598758Z } 2024-11-01T16:31:56.1599057Z echo "${GITHUB_WORKSPACE}" 2024-11-01T16:31:56.1599478Z if [ -z "${NO_SUDO}" ]; then 2024-11-01T16:31:56.1599955Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-11-01T16:31:56.1600417Z else 2024-11-01T16:31:56.1600783Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-11-01T16:31:56.1601255Z fi 2024-11-01T16:31:56.1601586Z mkdir "${GITHUB_WORKSPACE}" 2024-11-01T16:31:56.1607361Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:31:56.1607858Z env: 2024-11-01T16:31:56.1608124Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:31:56.1608464Z NO_SUDO: 2024-11-01T16:31:56.1608865Z ##[endgroup] 2024-11-01T16:31:56.1631863Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:31:56.2909394Z ##[group]Run malfet/checkout@silent-checkout 2024-11-01T16:31:56.2910005Z with: 2024-11-01T16:31:56.2910328Z ref: d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:31:56.2910774Z fetch-depth: 0 2024-11-01T16:31:56.2911079Z submodules: recursive 2024-11-01T16:31:56.2911409Z quiet-checkout: true 2024-11-01T16:31:56.2911747Z repository: pytorch/pytorch 2024-11-01T16:31:56.2912238Z token: *** 2024-11-01T16:31:56.2912515Z ssh-strict: true 2024-11-01T16:31:56.2912933Z persist-credentials: true 2024-11-01T16:31:56.2913288Z clean: true 2024-11-01T16:31:56.2913602Z sparse-checkout-cone-mode: true 2024-11-01T16:31:56.2913989Z lfs: false 2024-11-01T16:31:56.2914283Z set-safe-directory: true 2024-11-01T16:31:56.2914599Z env: 2024-11-01T16:31:56.2914857Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:31:56.2915194Z ##[endgroup] 2024-11-01T16:31:56.3919859Z Syncing repository: pytorch/pytorch 2024-11-01T16:31:56.3921511Z ##[group]Getting Git version info 2024-11-01T16:31:56.3922316Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-11-01T16:31:56.3923206Z [command]/usr/bin/git version 2024-11-01T16:31:56.3923576Z git version 2.40.1 2024-11-01T16:31:56.3924902Z ##[endgroup] 2024-11-01T16:31:56.3936436Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/3ba62717-e6d7-4df2-ba8a-dc8627701df0' before making global git config changes 2024-11-01T16:31:56.3937761Z Adding repository directory to the temporary git global config as a safe directory 2024-11-01T16:31:56.3938957Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:31:56.3975802Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-11-01T16:31:56.3979534Z ##[group]Initializing the repository 2024-11-01T16:31:56.3982460Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:31:56.4012238Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-11-01T16:31:56.4014137Z hint: is subject to change. To configure the initial branch name to use in all 2024-11-01T16:31:56.4014988Z hint: of your new repositories, which will suppress this warning, call: 2024-11-01T16:31:56.4015918Z hint: 2024-11-01T16:31:56.4016461Z hint: git config --global init.defaultBranch 2024-11-01T16:31:56.4016929Z hint: 2024-11-01T16:31:56.4017459Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-11-01T16:31:56.4018348Z hint: 'development'. The just-created branch can be renamed via this command: 2024-11-01T16:31:56.4019079Z hint: 2024-11-01T16:31:56.4019680Z hint: git branch -m 2024-11-01T16:31:56.4020478Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-11-01T16:31:56.4022654Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-11-01T16:31:56.4051047Z ##[endgroup] 2024-11-01T16:31:56.4051603Z ##[group]Disabling automatic garbage collection 2024-11-01T16:31:56.4054048Z [command]/usr/bin/git config --local gc.auto 0 2024-11-01T16:31:56.4083094Z ##[endgroup] 2024-11-01T16:31:56.4083589Z ##[group]Setting up auth 2024-11-01T16:31:56.4089226Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-11-01T16:31:56.4118570Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-11-01T16:31:56.4383796Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-11-01T16:31:56.4411656Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-11-01T16:31:56.4657319Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-11-01T16:31:56.4699359Z ##[endgroup] 2024-11-01T16:31:56.4700142Z ##[group]Fetching the repository 2024-11-01T16:31:56.4705864Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-11-01T16:31:59.3095721Z remote: Enumerating objects: 1053425 2024-11-01T16:31:59.3096333Z remote: Enumerating objects: 1053893, done. 2024-11-01T16:31:59.3097554Z remote: Counting objects: 0% (1/468) 2024-11-01T16:31:59.3098413Z remote: Counting objects: 1% (5/468) 2024-11-01T16:31:59.3099354Z remote: Counting objects: 2% (10/468) 2024-11-01T16:31:59.3100253Z remote: Counting objects: 3% (15/468) 2024-11-01T16:31:59.3101050Z remote: Counting objects: 4% (19/468) 2024-11-01T16:31:59.3101758Z remote: Counting objects: 5% (24/468) 2024-11-01T16:31:59.3102502Z remote: Counting objects: 6% (29/468) 2024-11-01T16:31:59.3103278Z remote: Counting objects: 7% (33/468) 2024-11-01T16:31:59.3104095Z remote: Counting objects: 8% (38/468) 2024-11-01T16:31:59.3104915Z remote: Counting objects: 9% (43/468) 2024-11-01T16:31:59.3105678Z remote: Counting objects: 10% (47/468) 2024-11-01T16:31:59.3106542Z remote: Counting objects: 11% (52/468) 2024-11-01T16:31:59.3107409Z remote: Counting objects: 12% (57/468) 2024-11-01T16:31:59.3108179Z remote: Counting objects: 13% (61/468) 2024-11-01T16:31:59.3108967Z remote: Counting objects: 14% (66/468) 2024-11-01T16:31:59.3109885Z remote: Counting objects: 15% (71/468) 2024-11-01T16:31:59.3110591Z remote: Counting objects: 16% (75/468) 2024-11-01T16:31:59.3111099Z remote: Counting objects: 17% (80/468) 2024-11-01T16:31:59.3111594Z remote: Counting objects: 18% (85/468) 2024-11-01T16:31:59.3112086Z remote: Counting objects: 19% (89/468) 2024-11-01T16:31:59.3112786Z remote: Counting objects: 20% (94/468) 2024-11-01T16:31:59.3114084Z remote: Counting objects: 21% (99/468) 2024-11-01T16:31:59.3114791Z remote: Counting objects: 22% (103/468) 2024-11-01T16:31:59.3115307Z remote: Counting objects: 23% (108/468) 2024-11-01T16:31:59.3115815Z remote: Counting objects: 24% (113/468) 2024-11-01T16:31:59.3116315Z remote: Counting objects: 25% (117/468) 2024-11-01T16:31:59.3116816Z remote: Counting objects: 26% (122/468) 2024-11-01T16:31:59.3117303Z remote: Counting objects: 27% (127/468) 2024-11-01T16:31:59.3117808Z remote: Counting objects: 28% (132/468) 2024-11-01T16:31:59.3118310Z remote: Counting objects: 29% (136/468) 2024-11-01T16:31:59.3118809Z remote: Counting objects: 30% (141/468) 2024-11-01T16:31:59.3119305Z remote: Counting objects: 31% (146/468) 2024-11-01T16:31:59.3119800Z remote: Counting objects: 32% (150/468) 2024-11-01T16:31:59.3120285Z remote: Counting objects: 33% (155/468) 2024-11-01T16:31:59.3120785Z remote: Counting objects: 34% (160/468) 2024-11-01T16:31:59.3121289Z remote: Counting objects: 35% (164/468) 2024-11-01T16:31:59.3121783Z remote: Counting objects: 36% (169/468) 2024-11-01T16:31:59.3122282Z remote: Counting objects: 37% (174/468) 2024-11-01T16:31:59.3122765Z remote: Counting objects: 38% (178/468) 2024-11-01T16:31:59.3123264Z remote: Counting objects: 39% (183/468) 2024-11-01T16:31:59.3123761Z remote: Counting objects: 40% (188/468) 2024-11-01T16:31:59.3124257Z remote: Counting objects: 41% (192/468) 2024-11-01T16:31:59.3124755Z remote: Counting objects: 42% (197/468) 2024-11-01T16:31:59.3125248Z remote: Counting objects: 43% (202/468) 2024-11-01T16:31:59.3125729Z remote: Counting objects: 44% (206/468) 2024-11-01T16:31:59.3126225Z remote: Counting objects: 45% (211/468) 2024-11-01T16:31:59.3126723Z remote: Counting objects: 46% (216/468) 2024-11-01T16:31:59.3127233Z remote: Counting objects: 47% (220/468) 2024-11-01T16:31:59.3127854Z remote: Counting objects: 48% (225/468) 2024-11-01T16:31:59.3128342Z remote: Counting objects: 49% (230/468) 2024-11-01T16:31:59.3128845Z remote: Counting objects: 50% (234/468) 2024-11-01T16:31:59.3129349Z remote: Counting objects: 51% (239/468) 2024-11-01T16:31:59.3129853Z remote: Counting objects: 52% (244/468) 2024-11-01T16:31:59.3130359Z remote: Counting objects: 53% (249/468) 2024-11-01T16:31:59.3130858Z remote: Counting objects: 54% (253/468) 2024-11-01T16:31:59.3131344Z remote: Counting objects: 55% (258/468) 2024-11-01T16:31:59.3131847Z remote: Counting objects: 56% (263/468) 2024-11-01T16:31:59.3132352Z remote: Counting objects: 57% (267/468) 2024-11-01T16:31:59.3132857Z remote: Counting objects: 58% (272/468) 2024-11-01T16:31:59.3133361Z remote: Counting objects: 59% (277/468) 2024-11-01T16:31:59.3133865Z remote: Counting objects: 60% (281/468) 2024-11-01T16:31:59.3134356Z remote: Counting objects: 61% (286/468) 2024-11-01T16:31:59.3134857Z remote: Counting objects: 62% (291/468) 2024-11-01T16:31:59.3135377Z remote: Counting objects: 63% (295/468) 2024-11-01T16:31:59.3135873Z remote: Counting objects: 64% (300/468) 2024-11-01T16:31:59.3136368Z remote: Counting objects: 65% (305/468) 2024-11-01T16:31:59.3136862Z remote: Counting objects: 66% (309/468) 2024-11-01T16:31:59.3137358Z remote: Counting objects: 67% (314/468) 2024-11-01T16:31:59.3137841Z remote: Counting objects: 68% (319/468) 2024-11-01T16:31:59.3138340Z remote: Counting objects: 69% (323/468) 2024-11-01T16:31:59.3138839Z remote: Counting objects: 70% (328/468) 2024-11-01T16:31:59.3139333Z remote: Counting objects: 71% (333/468) 2024-11-01T16:31:59.3139827Z remote: Counting objects: 72% (337/468) 2024-11-01T16:31:59.3140389Z remote: Counting objects: 73% (342/468) 2024-11-01T16:31:59.3140880Z remote: Counting objects: 74% (347/468) 2024-11-01T16:31:59.3141421Z remote: Counting objects: 75% (351/468) 2024-11-01T16:31:59.3141925Z remote: Counting objects: 76% (356/468) 2024-11-01T16:31:59.3142419Z remote: Counting objects: 77% (361/468) 2024-11-01T16:31:59.3142915Z remote: Counting objects: 78% (366/468) 2024-11-01T16:31:59.3143397Z remote: Counting objects: 79% (370/468) 2024-11-01T16:31:59.3143898Z remote: Counting objects: 80% (375/468) 2024-11-01T16:31:59.3144396Z remote: Counting objects: 81% (380/468) 2024-11-01T16:31:59.3144892Z remote: Counting objects: 82% (384/468) 2024-11-01T16:31:59.3145384Z remote: Counting objects: 83% (389/468) 2024-11-01T16:31:59.3145877Z remote: Counting objects: 84% (394/468) 2024-11-01T16:31:59.3146359Z remote: Counting objects: 85% (398/468) 2024-11-01T16:31:59.3146853Z remote: Counting objects: 86% (403/468) 2024-11-01T16:31:59.3147358Z remote: Counting objects: 87% (408/468) 2024-11-01T16:31:59.3147853Z remote: Counting objects: 88% (412/468) 2024-11-01T16:31:59.3148346Z remote: Counting objects: 89% (417/468) 2024-11-01T16:31:59.3148828Z remote: Counting objects: 90% (422/468) 2024-11-01T16:31:59.3149324Z remote: Counting objects: 91% (426/468) 2024-11-01T16:31:59.3149820Z remote: Counting objects: 92% (431/468) 2024-11-01T16:31:59.3150316Z remote: Counting objects: 93% (436/468) 2024-11-01T16:31:59.3150811Z remote: Counting objects: 94% (440/468) 2024-11-01T16:31:59.3151306Z remote: Counting objects: 95% (445/468) 2024-11-01T16:31:59.3151790Z remote: Counting objects: 96% (450/468) 2024-11-01T16:31:59.3152287Z remote: Counting objects: 97% (454/468) 2024-11-01T16:31:59.3152865Z remote: Counting objects: 98% (459/468) 2024-11-01T16:31:59.3153367Z remote: Counting objects: 99% (464/468) 2024-11-01T16:31:59.3153956Z remote: Counting objects: 100% (468/468) 2024-11-01T16:31:59.3154492Z remote: Counting objects: 100% (468/468), done. 2024-11-01T16:31:59.3155051Z remote: Compressing objects: 0% (1/252) 2024-11-01T16:31:59.3155577Z remote: Compressing objects: 1% (3/252) 2024-11-01T16:31:59.3161344Z remote: Compressing objects: 2% (6/252) 2024-11-01T16:31:59.3353184Z remote: Compressing objects: 3% (8/252) 2024-11-01T16:31:59.3445630Z remote: Compressing objects: 4% (11/252) 2024-11-01T16:31:59.3575593Z remote: Compressing objects: 5% (13/252) 2024-11-01T16:31:59.3637807Z remote: Compressing objects: 6% (16/252) 2024-11-01T16:31:59.3821352Z remote: Compressing objects: 7% (18/252) 2024-11-01T16:31:59.3906161Z remote: Compressing objects: 8% (21/252) 2024-11-01T16:31:59.4085054Z remote: Compressing objects: 9% (23/252) 2024-11-01T16:31:59.4156504Z remote: Compressing objects: 10% (26/252) 2024-11-01T16:31:59.4214249Z remote: Compressing objects: 11% (28/252) 2024-11-01T16:31:59.4229785Z remote: Compressing objects: 12% (31/252) 2024-11-01T16:31:59.4230740Z remote: Compressing objects: 13% (33/252) 2024-11-01T16:31:59.4231288Z remote: Compressing objects: 14% (36/252) 2024-11-01T16:31:59.4231848Z remote: Compressing objects: 15% (38/252) 2024-11-01T16:31:59.4232380Z remote: Compressing objects: 16% (41/252) 2024-11-01T16:31:59.4232969Z remote: Compressing objects: 17% (43/252) 2024-11-01T16:31:59.4233488Z remote: Compressing objects: 18% (46/252) 2024-11-01T16:31:59.4234023Z remote: Compressing objects: 19% (48/252) 2024-11-01T16:31:59.4234569Z remote: Compressing objects: 20% (51/252) 2024-11-01T16:31:59.4235203Z remote: Compressing objects: 21% (53/252) 2024-11-01T16:31:59.4237826Z remote: Compressing objects: 22% (56/252) 2024-11-01T16:31:59.4240798Z remote: Compressing objects: 23% (58/252) 2024-11-01T16:31:59.4242889Z remote: Compressing objects: 24% (61/252) 2024-11-01T16:31:59.4246807Z remote: Compressing objects: 25% (63/252) 2024-11-01T16:31:59.4247347Z remote: Compressing objects: 26% (66/252) 2024-11-01T16:31:59.4249246Z remote: Compressing objects: 27% (69/252) 2024-11-01T16:31:59.4249782Z remote: Compressing objects: 28% (71/252) 2024-11-01T16:31:59.4252063Z remote: Compressing objects: 29% (74/252) 2024-11-01T16:31:59.4252598Z remote: Compressing objects: 30% (76/252) 2024-11-01T16:31:59.4254500Z remote: Compressing objects: 31% (79/252) 2024-11-01T16:31:59.4255043Z remote: Compressing objects: 32% (81/252) 2024-11-01T16:31:59.4255580Z remote: Compressing objects: 33% (84/252) 2024-11-01T16:31:59.4259020Z remote: Compressing objects: 34% (86/252) 2024-11-01T16:31:59.4261449Z remote: Compressing objects: 35% (89/252) 2024-11-01T16:31:59.4266635Z remote: Compressing objects: 36% (91/252) 2024-11-01T16:31:59.4267178Z remote: Compressing objects: 37% (94/252) 2024-11-01T16:31:59.4270329Z remote: Compressing objects: 38% (96/252) 2024-11-01T16:31:59.4272963Z remote: Compressing objects: 39% (99/252) 2024-11-01T16:31:59.4275627Z remote: Compressing objects: 40% (101/252) 2024-11-01T16:31:59.4281544Z remote: Compressing objects: 41% (104/252) 2024-11-01T16:31:59.4287706Z remote: Compressing objects: 42% (106/252) 2024-11-01T16:31:59.4290685Z remote: Compressing objects: 43% (109/252) 2024-11-01T16:31:59.4293508Z remote: Compressing objects: 44% (111/252) 2024-11-01T16:31:59.4294039Z remote: Compressing objects: 45% (114/252) 2024-11-01T16:31:59.4295855Z remote: Compressing objects: 46% (116/252) 2024-11-01T16:31:59.4296394Z remote: Compressing objects: 47% (119/252) 2024-11-01T16:31:59.4298395Z remote: Compressing objects: 48% (121/252) 2024-11-01T16:31:59.4301156Z remote: Compressing objects: 49% (124/252) 2024-11-01T16:31:59.4301855Z remote: Compressing objects: 50% (126/252) 2024-11-01T16:31:59.4305817Z remote: Compressing objects: 51% (129/252) 2024-11-01T16:31:59.4307932Z remote: Compressing objects: 52% (132/252) 2024-11-01T16:31:59.4309394Z remote: Compressing objects: 53% (134/252) 2024-11-01T16:31:59.4310474Z remote: Compressing objects: 54% (137/252) 2024-11-01T16:31:59.4312409Z remote: Compressing objects: 55% (139/252) 2024-11-01T16:31:59.4313473Z remote: Compressing objects: 56% (142/252) 2024-11-01T16:31:59.4314120Z remote: Compressing objects: 57% (144/252) 2024-11-01T16:31:59.4315045Z remote: Compressing objects: 58% (147/252) 2024-11-01T16:31:59.4315761Z remote: Compressing objects: 59% (149/252) 2024-11-01T16:31:59.4316720Z remote: Compressing objects: 60% (152/252) 2024-11-01T16:31:59.4317685Z remote: Compressing objects: 61% (154/252) 2024-11-01T16:31:59.4318611Z remote: Compressing objects: 62% (157/252) 2024-11-01T16:31:59.4319405Z remote: Compressing objects: 63% (159/252) 2024-11-01T16:31:59.4320141Z remote: Compressing objects: 64% (162/252) 2024-11-01T16:31:59.4321054Z remote: Compressing objects: 65% (164/252) 2024-11-01T16:31:59.4321735Z remote: Compressing objects: 66% (167/252) 2024-11-01T16:31:59.4322590Z remote: Compressing objects: 67% (169/252) 2024-11-01T16:31:59.4323525Z remote: Compressing objects: 68% (172/252) 2024-11-01T16:31:59.4324212Z remote: Compressing objects: 69% (174/252) 2024-11-01T16:31:59.4325045Z remote: Compressing objects: 70% (177/252) 2024-11-01T16:31:59.4325967Z remote: Compressing objects: 71% (179/252) 2024-11-01T16:31:59.4326659Z remote: Compressing objects: 72% (182/252) 2024-11-01T16:31:59.4327490Z remote: Compressing objects: 73% (184/252) 2024-11-01T16:31:59.4328497Z remote: Compressing objects: 74% (187/252) 2024-11-01T16:31:59.4329335Z remote: Compressing objects: 75% (189/252) 2024-11-01T16:31:59.4330076Z remote: Compressing objects: 76% (192/252) 2024-11-01T16:31:59.4330987Z remote: Compressing objects: 77% (195/252) 2024-11-01T16:31:59.4331802Z remote: Compressing objects: 78% (197/252) 2024-11-01T16:31:59.4332520Z remote: Compressing objects: 79% (200/252) 2024-11-01T16:31:59.4333471Z remote: Compressing objects: 80% (202/252) 2024-11-01T16:31:59.4334160Z remote: Compressing objects: 81% (205/252) 2024-11-01T16:31:59.4335015Z remote: Compressing objects: 82% (207/252) 2024-11-01T16:31:59.4335862Z remote: Compressing objects: 83% (210/252) 2024-11-01T16:31:59.4336589Z remote: Compressing objects: 84% (212/252) 2024-11-01T16:31:59.4337470Z remote: Compressing objects: 85% (215/252) 2024-11-01T16:31:59.4338297Z remote: Compressing objects: 86% (217/252) 2024-11-01T16:31:59.4339058Z remote: Compressing objects: 87% (220/252) 2024-11-01T16:31:59.4339911Z remote: Compressing objects: 88% (222/252) 2024-11-01T16:31:59.4340442Z remote: Compressing objects: 89% (225/252) 2024-11-01T16:31:59.4340970Z remote: Compressing objects: 90% (227/252) 2024-11-01T16:31:59.4341529Z remote: Compressing objects: 91% (230/252) 2024-11-01T16:31:59.4342115Z remote: Compressing objects: 92% (232/252) 2024-11-01T16:31:59.4342646Z remote: Compressing objects: 93% (235/252) 2024-11-01T16:31:59.4343190Z remote: Compressing objects: 94% (237/252) 2024-11-01T16:31:59.4343724Z remote: Compressing objects: 95% (240/252) 2024-11-01T16:31:59.4344282Z remote: Compressing objects: 96% (242/252) 2024-11-01T16:31:59.4344835Z remote: Compressing objects: 97% (245/252) 2024-11-01T16:31:59.4345610Z remote: Compressing objects: 98% (247/252) 2024-11-01T16:31:59.4346443Z remote: Compressing objects: 99% (250/252) 2024-11-01T16:31:59.4347389Z remote: Compressing objects: 100% (252/252) 2024-11-01T16:31:59.4348257Z remote: Compressing objects: 100% (252/252), done. 2024-11-01T16:32:20.9927686Z remote: Total 1053893 (delta 269), reused 369 (delta 215), pack-reused 1053425 (from 1) 2024-11-01T16:32:49.7848711Z [command]/usr/bin/git rev-parse --verify --quiet d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea^{object} 2024-11-01T16:32:49.7876158Z d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:32:49.7881693Z ##[endgroup] 2024-11-01T16:32:49.7882296Z ##[group]Determining the checkout info 2024-11-01T16:32:49.7882932Z ##[endgroup] 2024-11-01T16:32:49.7883444Z ##[group]Checking out the ref 2024-11-01T16:32:49.7885842Z [command]/usr/bin/git checkout --quiet --force d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:32:51.1840511Z ##[endgroup] 2024-11-01T16:32:51.1841507Z ##[group]Setting up auth for fetching submodules 2024-11-01T16:32:51.1843017Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-11-01T16:32:51.1862997Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-11-01T16:32:51.1893044Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-11-01T16:32:51.1923380Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-11-01T16:32:51.1949306Z ##[endgroup] 2024-11-01T16:32:51.1949827Z ##[group]Fetching submodules 2024-11-01T16:32:51.1953160Z [command]/usr/bin/git submodule sync --recursive 2024-11-01T16:32:51.2226629Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-11-01T16:32:51.2496760Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-11-01T16:32:51.2499827Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-11-01T16:32:51.2503188Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-11-01T16:32:51.2506143Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-11-01T16:32:51.2508865Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2024-11-01T16:32:51.2513880Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-11-01T16:32:51.2517504Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-11-01T16:32:51.2520843Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-11-01T16:32:51.2526325Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2024-11-01T16:32:51.2530354Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-11-01T16:32:51.2535028Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-11-01T16:32:51.2540032Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-11-01T16:32:51.2544594Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-11-01T16:32:51.2549520Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-11-01T16:32:51.2554638Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-11-01T16:32:51.2560630Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-11-01T16:32:51.2567889Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-11-01T16:32:51.2573705Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:32:51.2578861Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-11-01T16:32:51.2584765Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-11-01T16:32:51.2590144Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-11-01T16:32:51.2596077Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-11-01T16:32:51.2601842Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-11-01T16:32:51.2607992Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-11-01T16:32:51.2613906Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-11-01T16:32:51.2620137Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-11-01T16:32:51.2626250Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-11-01T16:32:51.2633479Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-11-01T16:32:51.2639711Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-11-01T16:32:51.2646719Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-11-01T16:32:51.2653405Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-11-01T16:32:51.2660721Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-11-01T16:32:51.2667644Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-11-01T16:32:51.2676444Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-11-01T16:32:51.2683399Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-11-01T16:32:51.2690873Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-11-01T16:32:51.2717608Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-11-01T16:32:51.5947202Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-11-01T16:32:51.7917452Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-11-01T16:32:51.9814119Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-11-01T16:32:52.2573539Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2024-11-01T16:32:52.5638313Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-11-01T16:32:54.8024194Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-11-01T16:33:08.8327688Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-11-01T16:33:09.2656112Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2024-11-01T16:33:11.0536493Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-11-01T16:33:11.6722440Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-11-01T16:33:12.2813653Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-11-01T16:33:13.5367286Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-11-01T16:33:15.8451216Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-11-01T16:33:21.6329754Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-11-01T16:33:23.6735438Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-11-01T16:33:25.5059384Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-11-01T16:33:27.0886378Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-11-01T16:33:27.5530473Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-11-01T16:33:27.9117954Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-11-01T16:33:29.0509057Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-11-01T16:33:29.4102327Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-11-01T16:33:29.7034704Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-11-01T16:33:31.2549799Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-11-01T16:33:32.0734257Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-11-01T16:33:32.4888623Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-11-01T16:33:38.9717422Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-11-01T16:33:41.4200727Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-11-01T16:33:51.3414276Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-11-01T16:33:51.5719109Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-11-01T16:34:02.6386454Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-11-01T16:34:02.8424371Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-11-01T16:34:03.0421562Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-11-01T16:34:04.1916057Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-11-01T16:34:04.5020724Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-11-01T16:34:05.2738545Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-11-01T16:34:05.6984081Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-11-01T16:34:05.7096450Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-11-01T16:34:05.7180760Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-11-01T16:34:05.7414365Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-11-01T16:34:05.7702583Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1' 2024-11-01T16:34:05.8081169Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-11-01T16:34:06.5838125Z Submodule path 'third_party/XNNPACK': checked out '87ee0b46b834f67bad9025d4a82ed5654f3403d3' 2024-11-01T16:34:06.6067685Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-11-01T16:34:06.8092521Z Submodule path 'third_party/composable_kernel': checked out 'cedccd59c94cb0c74e7ec0d0f6c791aed081febc' 2024-11-01T16:34:06.8561756Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-11-01T16:34:06.9513344Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99' 2024-11-01T16:34:06.9838367Z Submodule path 'third_party/cudnn_frontend': checked out '936021bfed8c91dc416af1588b2c4eca631a9e45' 2024-11-01T16:34:07.4725115Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-11-01T16:34:07.7203004Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-11-01T16:34:07.8091958Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-11-01T16:34:07.8108532Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:34:07.8110937Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:34:07.8113600Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:34:07.8116202Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:34:07.8118979Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:34:07.8143167Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-11-01T16:34:09.0232165Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-11-01T16:34:09.6503971Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-11-01T16:34:11.9291549Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-11-01T16:34:13.0771783Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-11-01T16:34:13.4357468Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-11-01T16:34:13.5293401Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-11-01T16:34:13.9195467Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-11-01T16:34:13.9815213Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-11-01T16:34:13.9933264Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-11-01T16:34:14.1076552Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-11-01T16:34:14.1453862Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-11-01T16:34:14.1835011Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-11-01T16:34:14.2079288Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-11-01T16:34:14.2516018Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-11-01T16:34:14.2635929Z Submodule path 'third_party/ideep': checked out '41d636c2bbcea6bff0faf97cdb65a48cdde987af' 2024-11-01T16:34:14.2651553Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-11-01T16:34:14.2674276Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-11-01T16:34:28.0661359Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af' 2024-11-01T16:34:28.0834465Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-11-01T16:34:28.1686634Z Submodule path 'third_party/kineto': checked out 'ed052ea024b9468908d558b15cd3f7584fb0f492' 2024-11-01T16:34:28.1703027Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:34:28.1706389Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:34:28.1708456Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:34:28.1733356Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-11-01T16:34:28.7827597Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-11-01T16:34:30.1507862Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-11-01T16:34:31.3626188Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-11-01T16:34:31.3642049Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:34:31.3645112Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:34:31.3647756Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:34:31.3650806Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:34:31.3653927Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:34:31.3657279Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:34:31.3661891Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:34:31.3665066Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:34:31.3691414Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-11-01T16:34:32.2719725Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-11-01T16:34:32.6766117Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-11-01T16:34:34.0330353Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-11-01T16:34:34.3339950Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-11-01T16:34:34.9678693Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-11-01T16:34:36.1155779Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-11-01T16:34:43.5939278Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-11-01T16:34:43.9755279Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-11-01T16:34:43.9934279Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-11-01T16:34:44.0289986Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-11-01T16:34:44.0418692Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-11-01T16:34:44.0434779Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:34:44.0457976Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-11-01T16:34:44.3570928Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-11-01T16:34:44.3746948Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-11-01T16:34:44.4135548Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-11-01T16:34:44.5128463Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-11-01T16:34:44.5284496Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-11-01T16:34:44.5662082Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-11-01T16:34:44.6240420Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-11-01T16:34:44.6598295Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-11-01T16:34:44.6883406Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-11-01T16:34:44.7891983Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-11-01T16:34:45.1344862Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c' 2024-11-01T16:34:45.1377253Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-11-01T16:34:45.1402282Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-11-01T16:34:46.3498483Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-11-01T16:34:46.4156038Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-11-01T16:34:46.4175411Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:34:46.4177710Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:34:46.4179877Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:34:46.4182283Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:34:46.4184787Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:34:46.4187422Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:34:46.4190253Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:34:46.4193194Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:34:46.4218502Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-11-01T16:34:46.8613696Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-11-01T16:34:48.0345087Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-11-01T16:34:48.4162836Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-11-01T16:34:55.0099392Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-11-01T16:34:55.3627270Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-11-01T16:34:55.5919635Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-11-01T16:34:55.8989239Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-11-01T16:35:03.4884733Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-11-01T16:35:03.5271440Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-11-01T16:35:03.5424587Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-11-01T16:35:03.6379069Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-11-01T16:35:03.6501526Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-11-01T16:35:03.6637315Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-11-01T16:35:03.6783596Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-11-01T16:35:03.6798007Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:35:03.6800443Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:35:03.6823641Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-11-01T16:35:05.8930775Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-11-01T16:35:07.3133621Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-11-01T16:35:07.3587741Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-11-01T16:35:07.8013933Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-11-01T16:35:07.8129806Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-11-01T16:35:08.0898452Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-11-01T16:35:08.0919169Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:35:08.0920975Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-11-01T16:35:08.0947080Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-11-01T16:35:08.5809967Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-11-01T16:35:09.8660837Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-11-01T16:35:09.9370693Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-11-01T16:35:09.9457699Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-11-01T16:35:09.9572070Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-11-01T16:35:09.9912925Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2024-11-01T16:35:10.0188719Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-11-01T16:35:10.0593496Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-11-01T16:35:10.0842528Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-11-01T16:35:10.0857651Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:35:10.0860714Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:35:10.0863266Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:35:10.0866236Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:35:10.0889420Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-11-01T16:35:11.2640243Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-11-01T16:35:11.5254634Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-11-01T16:35:12.9036202Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-11-01T16:35:14.0180568Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-11-01T16:35:14.0329593Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-11-01T16:35:14.1010860Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-11-01T16:35:14.1287755Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-11-01T16:35:14.1301333Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:35:14.1325872Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-11-01T16:35:14.3362591Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-11-01T16:35:14.3394467Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-11-01T16:35:14.3668370Z Entering 'android/libs/fbjni' 2024-11-01T16:35:14.3706323Z Entering 'third_party/FP16' 2024-11-01T16:35:14.3744176Z Entering 'third_party/FXdiv' 2024-11-01T16:35:14.3781752Z Entering 'third_party/NNPACK' 2024-11-01T16:35:14.3819110Z Entering 'third_party/NVTX' 2024-11-01T16:35:14.3856770Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:35:14.3894656Z Entering 'third_party/XNNPACK' 2024-11-01T16:35:14.3948128Z Entering 'third_party/benchmark' 2024-11-01T16:35:14.3985725Z Entering 'third_party/composable_kernel' 2024-11-01T16:35:14.4029672Z Entering 'third_party/cpp-httplib' 2024-11-01T16:35:14.4067681Z Entering 'third_party/cpuinfo' 2024-11-01T16:35:14.4105157Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:35:14.4142119Z Entering 'third_party/cutlass' 2024-11-01T16:35:14.4188745Z Entering 'third_party/eigen' 2024-11-01T16:35:14.4230840Z Entering 'third_party/fbgemm' 2024-11-01T16:35:14.4269094Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:35:14.4305938Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:35:14.4342589Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:35:14.4386948Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:35:14.4423738Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:35:14.4463410Z Entering 'third_party/flatbuffers' 2024-11-01T16:35:14.4503271Z Entering 'third_party/fmt' 2024-11-01T16:35:14.4540824Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:35:14.4579847Z Entering 'third_party/gloo' 2024-11-01T16:35:14.4617100Z Entering 'third_party/googletest' 2024-11-01T16:35:14.4654378Z Entering 'third_party/ideep' 2024-11-01T16:35:14.4691242Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:35:14.4735948Z Entering 'third_party/ittapi' 2024-11-01T16:35:14.4774064Z Entering 'third_party/kineto' 2024-11-01T16:35:14.4811305Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:35:14.4848354Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:35:14.4887881Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:35:14.4925988Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:35:14.4963915Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:35:14.5000873Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:35:14.5039627Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:35:14.5078418Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:35:14.5115408Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:35:14.5152718Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:35:14.5192248Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:35:14.5229418Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:35:14.5268088Z Entering 'third_party/mimalloc' 2024-11-01T16:35:14.5307038Z Entering 'third_party/nccl/nccl' 2024-11-01T16:35:14.5344693Z Entering 'third_party/nlohmann' 2024-11-01T16:35:14.5384734Z Entering 'third_party/onnx' 2024-11-01T16:35:14.5437813Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:35:14.5477736Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:35:14.5518929Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:35:14.5556849Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:35:14.5593387Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:35:14.5631648Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:35:14.5671834Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:35:14.5712247Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:35:14.5750015Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:35:14.5788265Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:35:14.5827420Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:35:14.5866289Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:35:14.5923453Z Entering 'third_party/pocketfft' 2024-11-01T16:35:14.5962444Z Entering 'third_party/protobuf' 2024-11-01T16:35:14.6004049Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:35:14.6041340Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:35:14.6082579Z Entering 'third_party/psimd' 2024-11-01T16:35:14.6120483Z Entering 'third_party/pthreadpool' 2024-11-01T16:35:14.6157220Z Entering 'third_party/pybind11' 2024-11-01T16:35:14.6195470Z Entering 'third_party/python-peachpy' 2024-11-01T16:35:14.6232121Z Entering 'third_party/sleef' 2024-11-01T16:35:14.6269935Z Entering 'third_party/tensorpipe' 2024-11-01T16:35:14.6308232Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:35:14.6345942Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:35:14.6384009Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:35:14.6423017Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:35:14.6461472Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:35:14.6515088Z ##[endgroup] 2024-11-01T16:35:14.6516089Z ##[group]Persisting credentials for submodules 2024-11-01T16:35:14.6520379Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-11-01T16:35:14.6818599Z Entering 'android/libs/fbjni' 2024-11-01T16:35:14.6871536Z Entering 'third_party/FP16' 2024-11-01T16:35:14.6922996Z Entering 'third_party/FXdiv' 2024-11-01T16:35:14.6974225Z Entering 'third_party/NNPACK' 2024-11-01T16:35:14.7025209Z Entering 'third_party/NVTX' 2024-11-01T16:35:14.7080142Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:35:14.7130853Z Entering 'third_party/XNNPACK' 2024-11-01T16:35:14.7198444Z Entering 'third_party/benchmark' 2024-11-01T16:35:14.7252029Z Entering 'third_party/composable_kernel' 2024-11-01T16:35:14.7307129Z Entering 'third_party/cpp-httplib' 2024-11-01T16:35:14.7356607Z Entering 'third_party/cpuinfo' 2024-11-01T16:35:14.7405591Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:35:14.7454360Z Entering 'third_party/cutlass' 2024-11-01T16:35:14.7510211Z Entering 'third_party/eigen' 2024-11-01T16:35:14.7562794Z Entering 'third_party/fbgemm' 2024-11-01T16:35:14.7613363Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:35:14.7665473Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:35:14.7715111Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:35:14.7770793Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:35:14.7819506Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:35:14.7870690Z Entering 'third_party/flatbuffers' 2024-11-01T16:35:14.7922206Z Entering 'third_party/fmt' 2024-11-01T16:35:14.7971410Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:35:14.8020532Z Entering 'third_party/gloo' 2024-11-01T16:35:14.8070284Z Entering 'third_party/googletest' 2024-11-01T16:35:14.8120491Z Entering 'third_party/ideep' 2024-11-01T16:35:14.8169117Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:35:14.8226521Z Entering 'third_party/ittapi' 2024-11-01T16:35:14.8276603Z Entering 'third_party/kineto' 2024-11-01T16:35:14.8326141Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:35:14.8376983Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:35:14.8427197Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:35:14.8477684Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:35:14.8528140Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:35:14.8577913Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:35:14.8631010Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:35:14.8680834Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:35:14.8730302Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:35:14.8781384Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:35:14.8831547Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:35:14.8881104Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:35:14.8932944Z Entering 'third_party/mimalloc' 2024-11-01T16:35:14.8983637Z Entering 'third_party/nccl/nccl' 2024-11-01T16:35:14.9033703Z Entering 'third_party/nlohmann' 2024-11-01T16:35:14.9089138Z Entering 'third_party/onnx' 2024-11-01T16:35:14.9153853Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:35:14.9206454Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:35:14.9257274Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:35:14.9307278Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:35:14.9355923Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:35:14.9405944Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:35:14.9456644Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:35:14.9506216Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:35:14.9554642Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:35:14.9602904Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:35:14.9656493Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:35:14.9708203Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:35:14.9776947Z Entering 'third_party/pocketfft' 2024-11-01T16:35:14.9827175Z Entering 'third_party/protobuf' 2024-11-01T16:35:14.9878476Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:35:14.9925998Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:35:14.9975318Z Entering 'third_party/psimd' 2024-11-01T16:35:15.0026086Z Entering 'third_party/pthreadpool' 2024-11-01T16:35:15.0075964Z Entering 'third_party/pybind11' 2024-11-01T16:35:15.0126874Z Entering 'third_party/python-peachpy' 2024-11-01T16:35:15.0178733Z Entering 'third_party/sleef' 2024-11-01T16:35:15.0228650Z Entering 'third_party/tensorpipe' 2024-11-01T16:35:15.0279472Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:35:15.0331661Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:35:15.0381104Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:35:15.0429129Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:35:15.0477858Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:35:15.0551918Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-11-01T16:35:15.0836525Z Entering 'android/libs/fbjni' 2024-11-01T16:35:15.0885225Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-11-01T16:35:15.0902088Z Entering 'third_party/FP16' 2024-11-01T16:35:15.0952651Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-11-01T16:35:15.0968832Z Entering 'third_party/FXdiv' 2024-11-01T16:35:15.1016249Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-11-01T16:35:15.1030718Z Entering 'third_party/NNPACK' 2024-11-01T16:35:15.1079995Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-11-01T16:35:15.1096501Z Entering 'third_party/NVTX' 2024-11-01T16:35:15.1146885Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2024-11-01T16:35:15.1165173Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:35:15.1214612Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-11-01T16:35:15.1230544Z Entering 'third_party/XNNPACK' 2024-11-01T16:35:15.1280704Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-11-01T16:35:15.1313240Z Entering 'third_party/benchmark' 2024-11-01T16:35:15.1362118Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-11-01T16:35:15.1377443Z Entering 'third_party/composable_kernel' 2024-11-01T16:35:15.1428068Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2024-11-01T16:35:15.1449110Z Entering 'third_party/cpp-httplib' 2024-11-01T16:35:15.1497348Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-11-01T16:35:15.1512389Z Entering 'third_party/cpuinfo' 2024-11-01T16:35:15.1561156Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-11-01T16:35:15.1577428Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:35:15.1625952Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-11-01T16:35:15.1642010Z Entering 'third_party/cutlass' 2024-11-01T16:35:15.1690672Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-11-01T16:35:15.1714317Z Entering 'third_party/eigen' 2024-11-01T16:35:15.1764249Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-11-01T16:35:15.1781818Z Entering 'third_party/fbgemm' 2024-11-01T16:35:15.1831355Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-11-01T16:35:15.1847557Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:35:15.1893517Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-11-01T16:35:15.1907033Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:35:15.1955774Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-11-01T16:35:15.1970830Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:35:15.2020303Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-11-01T16:35:15.2040617Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:35:15.2089287Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:35:15.2103350Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:35:15.2153027Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-11-01T16:35:15.2168951Z Entering 'third_party/flatbuffers' 2024-11-01T16:35:15.2214592Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-11-01T16:35:15.2231730Z Entering 'third_party/fmt' 2024-11-01T16:35:15.2279639Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-11-01T16:35:15.2295321Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:35:15.2341369Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-11-01T16:35:15.2356931Z Entering 'third_party/gloo' 2024-11-01T16:35:15.2405480Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-11-01T16:35:15.2421576Z Entering 'third_party/googletest' 2024-11-01T16:35:15.2471599Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:35:15.2487884Z Entering 'third_party/ideep' 2024-11-01T16:35:15.2537152Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-11-01T16:35:15.2551168Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:35:15.2597353Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-11-01T16:35:15.2620634Z Entering 'third_party/ittapi' 2024-11-01T16:35:15.2670447Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-11-01T16:35:15.2685341Z Entering 'third_party/kineto' 2024-11-01T16:35:15.2733000Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-11-01T16:35:15.2747424Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:35:15.2796751Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-11-01T16:35:15.2810735Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:35:15.2857751Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-11-01T16:35:15.2872606Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:35:15.2921434Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-11-01T16:35:15.2936348Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:35:15.2982656Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-11-01T16:35:15.2996779Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:35:15.3047196Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-11-01T16:35:15.3061973Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:35:15.3107895Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-11-01T16:35:15.3123617Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:35:15.3171631Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-11-01T16:35:15.3186058Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:35:15.3231967Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:35:15.3246586Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:35:15.3293248Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-11-01T16:35:15.3308820Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:35:15.3355822Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-11-01T16:35:15.3374666Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:35:15.3420477Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-11-01T16:35:15.3434716Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:35:15.3484685Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-11-01T16:35:15.3500003Z Entering 'third_party/mimalloc' 2024-11-01T16:35:15.3547434Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-11-01T16:35:15.3562860Z Entering 'third_party/nccl/nccl' 2024-11-01T16:35:15.3608925Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-11-01T16:35:15.3624298Z Entering 'third_party/nlohmann' 2024-11-01T16:35:15.3670132Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-11-01T16:35:15.3685716Z Entering 'third_party/onnx' 2024-11-01T16:35:15.3731556Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-11-01T16:35:15.3763638Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:35:15.3810030Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-11-01T16:35:15.3826963Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:35:15.3874695Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-11-01T16:35:15.3891786Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:35:15.3937415Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-11-01T16:35:15.3951763Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:35:15.3999126Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:35:15.4013209Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:35:15.4058946Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-11-01T16:35:15.4073138Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:35:15.4118801Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-11-01T16:35:15.4134477Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:35:15.4180289Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-11-01T16:35:15.4195977Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:35:15.4241454Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-11-01T16:35:15.4255426Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:35:15.4301743Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-11-01T16:35:15.4315174Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:35:15.4364532Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-11-01T16:35:15.4380529Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:35:15.4429072Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-11-01T16:35:15.4444759Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:35:15.4491554Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-11-01T16:35:15.4526802Z Entering 'third_party/pocketfft' 2024-11-01T16:35:15.4574820Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-11-01T16:35:15.4589310Z Entering 'third_party/protobuf' 2024-11-01T16:35:15.4635276Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-11-01T16:35:15.4652029Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:35:15.4697514Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-11-01T16:35:15.4711715Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:35:15.4759014Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:35:15.4774502Z Entering 'third_party/psimd' 2024-11-01T16:35:15.4824029Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-11-01T16:35:15.4838549Z Entering 'third_party/pthreadpool' 2024-11-01T16:35:15.4888434Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-11-01T16:35:15.4903635Z Entering 'third_party/pybind11' 2024-11-01T16:35:15.4948636Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-11-01T16:35:15.4964245Z Entering 'third_party/python-peachpy' 2024-11-01T16:35:15.5010774Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-11-01T16:35:15.5025361Z Entering 'third_party/sleef' 2024-11-01T16:35:15.5071496Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-11-01T16:35:15.5085960Z Entering 'third_party/tensorpipe' 2024-11-01T16:35:15.5132142Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-11-01T16:35:15.5146046Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:35:15.5193554Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:35:15.5209458Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:35:15.5256463Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-11-01T16:35:15.5270443Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:35:15.5318343Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-11-01T16:35:15.5333235Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:35:15.5381638Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-11-01T16:35:15.5396631Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:35:15.5442367Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-11-01T16:35:15.6414062Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-11-01T16:35:15.6693815Z Entering 'android/libs/fbjni' 2024-11-01T16:35:15.6728931Z Entering 'third_party/FP16' 2024-11-01T16:35:15.6766335Z Entering 'third_party/FXdiv' 2024-11-01T16:35:15.6803638Z Entering 'third_party/NNPACK' 2024-11-01T16:35:15.6841065Z Entering 'third_party/NVTX' 2024-11-01T16:35:15.6879072Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:35:15.6916339Z Entering 'third_party/XNNPACK' 2024-11-01T16:35:15.6970639Z Entering 'third_party/benchmark' 2024-11-01T16:35:15.7007702Z Entering 'third_party/composable_kernel' 2024-11-01T16:35:15.7052348Z Entering 'third_party/cpp-httplib' 2024-11-01T16:35:15.7089699Z Entering 'third_party/cpuinfo' 2024-11-01T16:35:15.7127833Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:35:15.7165708Z Entering 'third_party/cutlass' 2024-11-01T16:35:15.7209167Z Entering 'third_party/eigen' 2024-11-01T16:35:15.7248761Z Entering 'third_party/fbgemm' 2024-11-01T16:35:15.7286493Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:35:15.7322676Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:35:15.7359849Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:35:15.7403694Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:35:15.7439869Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:35:15.7478332Z Entering 'third_party/flatbuffers' 2024-11-01T16:35:15.7517880Z Entering 'third_party/fmt' 2024-11-01T16:35:15.7555270Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:35:15.7592419Z Entering 'third_party/gloo' 2024-11-01T16:35:15.7629900Z Entering 'third_party/googletest' 2024-11-01T16:35:15.7667617Z Entering 'third_party/ideep' 2024-11-01T16:35:15.7703862Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:35:15.7747464Z Entering 'third_party/ittapi' 2024-11-01T16:35:15.7785392Z Entering 'third_party/kineto' 2024-11-01T16:35:15.7822747Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:35:15.7861446Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:35:15.7898722Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:35:15.7936180Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:35:15.7973617Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:35:15.8011060Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:35:15.8049466Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:35:15.8086521Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:35:15.8123927Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:35:15.8165468Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:35:15.8203896Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:35:15.8240592Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:35:15.8278498Z Entering 'third_party/mimalloc' 2024-11-01T16:35:15.8316151Z Entering 'third_party/nccl/nccl' 2024-11-01T16:35:15.8353525Z Entering 'third_party/nlohmann' 2024-11-01T16:35:15.8391917Z Entering 'third_party/onnx' 2024-11-01T16:35:15.8444150Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:35:15.8483687Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:35:15.8523315Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:35:15.8559808Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:35:15.8597247Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:35:15.8633441Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:35:15.8675785Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:35:15.8712128Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:35:15.8747895Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:35:15.8784210Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:35:15.8822119Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:35:15.8860578Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:35:15.8916291Z Entering 'third_party/pocketfft' 2024-11-01T16:35:15.8953349Z Entering 'third_party/protobuf' 2024-11-01T16:35:15.8994623Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:35:15.9032202Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:35:15.9070809Z Entering 'third_party/psimd' 2024-11-01T16:35:15.9107522Z Entering 'third_party/pthreadpool' 2024-11-01T16:35:15.9144235Z Entering 'third_party/pybind11' 2024-11-01T16:35:15.9181777Z Entering 'third_party/python-peachpy' 2024-11-01T16:35:15.9218044Z Entering 'third_party/sleef' 2024-11-01T16:35:15.9254738Z Entering 'third_party/tensorpipe' 2024-11-01T16:35:15.9292110Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:35:15.9329099Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:35:15.9366089Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:35:15.9402819Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:35:15.9438882Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:35:15.9494191Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-11-01T16:35:15.9766687Z Entering 'android/libs/fbjni' 2024-11-01T16:35:15.9804853Z Entering 'third_party/FP16' 2024-11-01T16:35:15.9842287Z Entering 'third_party/FXdiv' 2024-11-01T16:35:15.9880159Z Entering 'third_party/NNPACK' 2024-11-01T16:35:15.9917711Z Entering 'third_party/NVTX' 2024-11-01T16:35:15.9955671Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:35:15.9995080Z Entering 'third_party/XNNPACK' 2024-11-01T16:35:16.0049436Z Entering 'third_party/benchmark' 2024-11-01T16:35:16.0087571Z Entering 'third_party/composable_kernel' 2024-11-01T16:35:16.0130515Z Entering 'third_party/cpp-httplib' 2024-11-01T16:35:16.0169155Z Entering 'third_party/cpuinfo' 2024-11-01T16:35:16.0206749Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:35:16.0245778Z Entering 'third_party/cutlass' 2024-11-01T16:35:16.0294577Z Entering 'third_party/eigen' 2024-11-01T16:35:16.0334351Z Entering 'third_party/fbgemm' 2024-11-01T16:35:16.0373123Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:35:16.0409780Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:35:16.0446767Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:35:16.0491094Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:35:16.0528105Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:35:16.0581793Z Entering 'third_party/flatbuffers' 2024-11-01T16:35:16.0608351Z Entering 'third_party/fmt' 2024-11-01T16:35:16.0646008Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:35:16.0684751Z Entering 'third_party/gloo' 2024-11-01T16:35:16.0722669Z Entering 'third_party/googletest' 2024-11-01T16:35:16.0762943Z Entering 'third_party/ideep' 2024-11-01T16:35:16.0799643Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:35:16.0843329Z Entering 'third_party/ittapi' 2024-11-01T16:35:16.0881777Z Entering 'third_party/kineto' 2024-11-01T16:35:16.0919741Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:35:16.0957113Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:35:16.0996563Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:35:16.1033748Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:35:16.1071223Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:35:16.1109072Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:35:16.1147946Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:35:16.1185259Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:35:16.1222466Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:35:16.1260824Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:35:16.1300361Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:35:16.1337273Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:35:16.1375879Z Entering 'third_party/mimalloc' 2024-11-01T16:35:16.1413755Z Entering 'third_party/nccl/nccl' 2024-11-01T16:35:16.1451226Z Entering 'third_party/nlohmann' 2024-11-01T16:35:16.1490174Z Entering 'third_party/onnx' 2024-11-01T16:35:16.1542752Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:35:16.1585316Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:35:16.1625362Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:35:16.1662614Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:35:16.1699440Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:35:16.1736106Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:35:16.1774627Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:35:16.1811387Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:35:16.1848346Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:35:16.2122781Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:35:16.2162913Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:35:16.2202495Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:35:16.2261835Z Entering 'third_party/pocketfft' 2024-11-01T16:35:16.2299787Z Entering 'third_party/protobuf' 2024-11-01T16:35:16.2340588Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:35:16.2378082Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:35:16.2417432Z Entering 'third_party/psimd' 2024-11-01T16:35:16.2456362Z Entering 'third_party/pthreadpool' 2024-11-01T16:35:16.2495122Z Entering 'third_party/pybind11' 2024-11-01T16:35:16.2533009Z Entering 'third_party/python-peachpy' 2024-11-01T16:35:16.2570571Z Entering 'third_party/sleef' 2024-11-01T16:35:16.2607765Z Entering 'third_party/tensorpipe' 2024-11-01T16:35:16.2645472Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:35:16.2685648Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:35:16.2723140Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:35:16.2759995Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:35:16.2796469Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:35:16.2848120Z ##[endgroup] 2024-11-01T16:35:16.2893954Z [command]/usr/bin/git log -1 --format='%H' 2024-11-01T16:35:16.2922693Z 'd1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea' 2024-11-01T16:35:16.3121938Z Prepare all required actions 2024-11-01T16:35:16.3122397Z Getting action download info 2024-11-01T16:35:16.4503746Z ##[group]Run ./.github/actions/setup-linux 2024-11-01T16:35:16.4504174Z env: 2024-11-01T16:35:16.4504442Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:35:16.4504800Z ##[endgroup] 2024-11-01T16:35:16.4570578Z ##[group]Run set -euo pipefail 2024-11-01T16:35:16.4570999Z set -euo pipefail 2024-11-01T16:35:16.4571391Z function get_ec2_metadata() { 2024-11-01T16:35:16.4571935Z  # Pulled from instance metadata endpoint for EC2 2024-11-01T16:35:16.4572866Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-11-01T16:35:16.4573678Z  category=$1 2024-11-01T16:35:16.4574213Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-11-01T16:35:16.4574863Z  runner_name_str=i-07c6a72643395af71 2024-11-01T16:35:16.4575362Z  if [[ -f /.inarc ]]; then 2024-11-01T16:35:16.4575843Z  echo "ARC Runner, no info on ec2 metadata" 2024-11-01T16:35:16.4576418Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-11-01T16:35:16.4577128Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-11-01T16:35:16.4577759Z  else 2024-11-01T16:35:16.4579046Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-11-01T16:35:16.4580416Z  fi 2024-11-01T16:35:16.4580688Z } 2024-11-01T16:35:16.4581028Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-11-01T16:35:16.4581624Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-11-01T16:35:16.4582295Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-11-01T16:35:16.4582870Z echo "system info $(uname -a)" 2024-11-01T16:35:16.4591945Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:35:16.4592456Z env: 2024-11-01T16:35:16.4592706Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:35:16.4593155Z ##[endgroup] 2024-11-01T16:35:16.4736019Z ami-id: ami-0fff1b9a61dec8a5f 2024-11-01T16:35:16.4825492Z instance-id: i-07c6a72643395af71 2024-11-01T16:35:16.4930458Z instance-type: c5.2xlarge 2024-11-01T16:35:16.4939951Z system info Linux ip-10-0-40-123.ec2.internal 6.1.109-118.189.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 10 08:59:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-11-01T16:35:16.4964844Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:35:16.4966204Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:35:16.4972504Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:35:16.4973026Z env: 2024-11-01T16:35:16.4973294Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:35:16.4973637Z ##[endgroup] 2024-11-01T16:35:16.5025860Z ##[group]Run if systemctl is-active --quiet docker; then 2024-11-01T16:35:16.5026460Z if systemctl is-active --quiet docker; then 2024-11-01T16:35:16.5027125Z  echo "Docker daemon is running..."; 2024-11-01T16:35:16.5027576Z else 2024-11-01T16:35:16.5028066Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-11-01T16:35:16.5028661Z fi 2024-11-01T16:35:16.5034309Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:35:16.5034799Z env: 2024-11-01T16:35:16.5035059Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:35:16.5035395Z ##[endgroup] 2024-11-01T16:35:16.5108265Z Docker daemon is running... 2024-11-01T16:35:16.5179784Z ##[group]Run nick-fields/retry@v3.0.0 2024-11-01T16:35:16.5180168Z with: 2024-11-01T16:35:16.5180422Z shell: bash 2024-11-01T16:35:16.5180864Z timeout_minutes: 5 2024-11-01T16:35:16.5181185Z max_attempts: 3 2024-11-01T16:35:16.5181495Z retry_wait_seconds: 30 2024-11-01T16:35:16.5184744Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-11-01T16:35:16.5188048Z polling_interval_seconds: 1 2024-11-01T16:35:16.5188420Z warning_on_retry: true 2024-11-01T16:35:16.5188745Z continue_on_error: false 2024-11-01T16:35:16.5189069Z env: 2024-11-01T16:35:16.5189328Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:35:16.5189677Z AWS_RETRY_MODE: standard 2024-11-01T16:35:16.5190014Z AWS_MAX_ATTEMPTS: 5 2024-11-01T16:35:16.5190332Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:35:16.5190697Z ##[endgroup] 2024-11-01T16:35:17.7239118Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-11-01T16:35:17.7240426Z Configure a credential helper to remove this warning. See 2024-11-01T16:35:17.7241738Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-11-01T16:35:17.7242315Z 2024-11-01T16:35:17.7242422Z Login Succeeded 2024-11-01T16:35:18.6022912Z Command completed after 1 attempt(s). 2024-11-01T16:35:18.6085942Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-11-01T16:35:18.6086678Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-11-01T16:35:18.6087341Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-11-01T16:35:18.6094807Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:35:18.6095308Z env: 2024-11-01T16:35:18.6095572Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:35:18.6095910Z ##[endgroup] 2024-11-01T16:35:18.6181941Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T16:35:18.6183145Z # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T16:35:18.6183745Z # shellcheck disable=SC2046 2024-11-01T16:35:18.6184199Z docker stop $(docker ps -q) || true 2024-11-01T16:35:18.6184674Z # Prune all of the docker images 2024-11-01T16:35:18.6185121Z docker system prune -af 2024-11-01T16:35:18.6190606Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:35:18.6191096Z env: 2024-11-01T16:35:18.6191362Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:35:18.6191703Z ##[endgroup] 2024-11-01T16:35:18.6440246Z "docker stop" requires at least 1 argument. 2024-11-01T16:35:18.6441093Z See 'docker stop --help'. 2024-11-01T16:35:18.6441331Z 2024-11-01T16:35:18.6441581Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-11-01T16:35:18.6441966Z 2024-11-01T16:35:18.6442117Z Stop one or more running containers 2024-11-01T16:35:18.6605118Z Total reclaimed space: 0B 2024-11-01T16:35:18.6659339Z ##[group]Run set +e 2024-11-01T16:35:18.6659690Z set +e 2024-11-01T16:35:18.6659991Z set -x 2024-11-01T16:35:18.6660281Z  2024-11-01T16:35:18.6660590Z PT_DOMAIN=download.pytorch.org 2024-11-01T16:35:18.6661420Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-11-01T16:35:18.6662566Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-11-01T16:35:18.6663361Z # one is returned at random 2024-11-01T16:35:18.6663938Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-11-01T16:35:18.6664469Z  2024-11-01T16:35:18.6664916Z if [ -z "${RESOLVED_IP}" ]; then 2024-11-01T16:35:18.6671680Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-11-01T16:35:18.6672484Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-11-01T16:35:18.6673162Z  2024-11-01T16:35:18.6673478Z  if [ -z "${RESOLVED_IP}" ]; then 2024-11-01T16:35:18.6674036Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-11-01T16:35:18.6674546Z  exit 1 2024-11-01T16:35:18.6674847Z  fi 2024-11-01T16:35:18.6675105Z fi 2024-11-01T16:35:18.6675366Z  2024-11-01T16:35:18.6696352Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-11-01T16:35:18.6696855Z  # Clean up any old records first 2024-11-01T16:35:18.6697340Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-11-01T16:35:18.6697761Z fi 2024-11-01T16:35:18.6698001Z  2024-11-01T16:35:18.6698412Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-11-01T16:35:18.6698945Z cat /etc/hosts 2024-11-01T16:35:18.6704653Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:35:18.6705146Z env: 2024-11-01T16:35:18.6705386Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:35:18.6705717Z ##[endgroup] 2024-11-01T16:35:18.6727748Z + PT_DOMAIN=download.pytorch.org 2024-11-01T16:35:18.6733171Z ++ dig -4 +short download.pytorch.org 2024-11-01T16:35:18.6733733Z ++ tail -n1 2024-11-01T16:35:18.7386169Z + RESOLVED_IP=18.160.10.28 2024-11-01T16:35:18.7386806Z + '[' -z 18.160.10.28 ']' 2024-11-01T16:35:18.7387241Z + grep -r download.pytorch.org /etc/hosts 2024-11-01T16:35:18.7401010Z + echo '18.160.10.28 download.pytorch.org' 2024-11-01T16:35:18.7401591Z + sudo tee -a /etc/hosts 2024-11-01T16:35:18.9644380Z 18.160.10.28 download.pytorch.org 2024-11-01T16:35:18.9660091Z + cat /etc/hosts 2024-11-01T16:35:18.9669250Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-11-01T16:35:18.9682848Z ::1 localhost6 localhost6.localdomain6 2024-11-01T16:35:18.9683424Z 18.160.10.28 download.pytorch.org 2024-11-01T16:35:18.9886759Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-11-01T16:35:18.9887390Z with: 2024-11-01T16:35:18.9888329Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:35:18.9889399Z docker-build-dir: .ci/docker 2024-11-01T16:35:18.9889772Z working-directory: . 2024-11-01T16:35:18.9890246Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:35:18.9890773Z force-push: false 2024-11-01T16:35:18.9891066Z env: 2024-11-01T16:35:18.9891327Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:35:18.9891665Z ##[endgroup] 2024-11-01T16:35:18.9914363Z ##[group]Run set -ex 2024-11-01T16:35:18.9914711Z set -ex 2024-11-01T16:35:18.9914994Z  2024-11-01T16:35:18.9915558Z # If the docker build directory or the build script doesn't exist, the action will 2024-11-01T16:35:18.9916617Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-11-01T16:35:18.9917714Z # job could then download the pre-built image as usual 2024-11-01T16:35:18.9918611Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-11-01T16:35:18.9919294Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-11-01T16:35:18.9919947Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:35:18.9920522Z  2024-11-01T16:35:18.9921038Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-11-01T16:35:18.9921682Z  exit 0 2024-11-01T16:35:18.9921975Z else 2024-11-01T16:35:18.9922320Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-11-01T16:35:18.9922773Z fi 2024-11-01T16:35:18.9923034Z  2024-11-01T16:35:18.9923499Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-11-01T16:35:18.9924393Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-11-01T16:35:18.9925198Z  # use it as it is, but first let's extract the tag 2024-11-01T16:35:18.9925913Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-11-01T16:35:18.9926658Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:35:18.9927378Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:35:18.9927951Z else 2024-11-01T16:35:18.9928385Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-11-01T16:35:18.9929063Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:35:18.9929998Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:35:18.9930777Z fi 2024-11-01T16:35:18.9938694Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:35:18.9939190Z env: 2024-11-01T16:35:18.9939456Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:35:18.9939804Z REPO_NAME: pytorch 2024-11-01T16:35:18.9940772Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:35:18.9941826Z DOCKER_BUILD_DIR: .ci/docker 2024-11-01T16:35:18.9942327Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:35:18.9942841Z ##[endgroup] 2024-11-01T16:35:18.9967988Z + [[ ! -d .ci/docker ]] 2024-11-01T16:35:18.9968402Z + [[ ! -f .ci/docker/build.sh ]] 2024-11-01T16:35:18.9968791Z + echo skip=false 2024-11-01T16:35:18.9970483Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-11-01T16:35:18.9978017Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:35:18.9979105Z ++ awk -F '[:,]' '{print $2}' 2024-11-01T16:35:18.9998120Z + DOCKER_TAG=bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:35:18.9999191Z + echo docker-tag=bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:35:19.0001223Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:35:19.0029126Z ##[group]Run set +e 2024-11-01T16:35:19.0029488Z set +e 2024-11-01T16:35:19.0029778Z set -x 2024-11-01T16:35:19.0030056Z  2024-11-01T16:35:19.0030320Z login() { 2024-11-01T16:35:19.0030979Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-11-01T16:35:19.0031724Z } 2024-11-01T16:35:19.0031987Z  2024-11-01T16:35:19.0032250Z retry () { 2024-11-01T16:35:19.0032637Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-11-01T16:35:19.0033182Z } 2024-11-01T16:35:19.0033428Z  2024-11-01T16:35:19.0033861Z retry login "${DOCKER_REGISTRY}" 2024-11-01T16:35:19.0034275Z  2024-11-01T16:35:19.0034748Z # Check if image already exists, if it does then skip building it 2024-11-01T16:35:19.0035476Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-11-01T16:35:19.0035970Z  exit 0 2024-11-01T16:35:19.0036258Z fi 2024-11-01T16:35:19.0036520Z  2024-11-01T16:35:19.0037028Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-11-01T16:35:19.0037885Z # be empty. The default action would be to continue rebuild the image 2024-11-01T16:35:19.0038644Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-11-01T16:35:19.0039319Z  # if we're on the base branch then use the parent commit 2024-11-01T16:35:19.0039921Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-11-01T16:35:19.0040350Z else 2024-11-01T16:35:19.0040811Z  # otherwise we're on a PR, so use the most recent base commit 2024-11-01T16:35:19.0041523Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-11-01T16:35:19.0042029Z fi 2024-11-01T16:35:19.0042277Z  2024-11-01T16:35:19.0042578Z if [[ -z "${MERGE_BASE}" ]]; then 2024-11-01T16:35:19.0043080Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-11-01T16:35:19.0043533Z  2024-11-01T16:35:19.0044202Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-11-01T16:35:19.0044999Z  exit 0 2024-11-01T16:35:19.0045272Z fi 2024-11-01T16:35:19.0045530Z  2024-11-01T16:35:19.0045958Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-11-01T16:35:19.0046977Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-11-01T16:35:19.0047823Z  exit 1 2024-11-01T16:35:19.0048115Z fi 2024-11-01T16:35:19.0048364Z  2024-11-01T16:35:19.0048858Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-11-01T16:35:19.0049858Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-11-01T16:35:19.0050759Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-11-01T16:35:19.0051771Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-11-01T16:35:19.0052923Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-11-01T16:35:19.0053582Z fi 2024-11-01T16:35:19.0053846Z  2024-11-01T16:35:19.0054192Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-11-01T16:35:19.0060492Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:35:19.0061003Z env: 2024-11-01T16:35:19.0061269Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:35:19.0061622Z DOCKER_BUILD_DIR: .ci/docker 2024-11-01T16:35:19.0062094Z BASE_REVISION: 2055d3c8dc8b93c52eda6b9c07d716448d9e28c9 2024-11-01T16:35:19.0063207Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:35:19.0064301Z DOCKER_TAG: bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:35:19.0064912Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:35:19.0065436Z ##[endgroup] 2024-11-01T16:35:19.0088502Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:35:19.0089169Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:35:19.0091216Z + aws ecr get-login-password --region us-east-1 2024-11-01T16:35:19.0092599Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:35:19.5539494Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-11-01T16:35:19.5540867Z Configure a credential helper to remove this warning. See 2024-11-01T16:35:19.5541764Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-11-01T16:35:19.5542324Z 2024-11-01T16:35:19.5542433Z Login Succeeded 2024-11-01T16:35:19.5551635Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:35:19.7446319Z { 2024-11-01T16:35:19.7446914Z "schemaVersion": 2, 2024-11-01T16:35:19.7447778Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-11-01T16:35:19.7448688Z "config": { 2024-11-01T16:35:19.7449469Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-11-01T16:35:19.7450490Z "size": 45977, 2024-11-01T16:35:19.7451599Z "digest": "sha256:acd5ab414b6754b1ba1fbfceff9d37aa2af65d736e970c726d477963e8596d49" 2024-11-01T16:35:19.7452703Z }, 2024-11-01T16:35:19.7453089Z "layers": [ 2024-11-01T16:35:19.7453566Z { 2024-11-01T16:35:19.7454356Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7455311Z "size": 30439933, 2024-11-01T16:35:19.7456248Z "digest": "sha256:7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059" 2024-11-01T16:35:19.7457324Z }, 2024-11-01T16:35:19.7457736Z { 2024-11-01T16:35:19.7458488Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7459525Z "size": 1822, 2024-11-01T16:35:19.7460291Z "digest": "sha256:ad0341f900ba4169726d094fc9b175b444e43083e7a41767b567d258755520df" 2024-11-01T16:35:19.7461370Z }, 2024-11-01T16:35:19.7461749Z { 2024-11-01T16:35:19.7462399Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7463407Z "size": 313176109, 2024-11-01T16:35:19.7464353Z "digest": "sha256:be63e049eed381057c26b89dbe1a5af32980caf444535353763f260883778b0f" 2024-11-01T16:35:19.7465224Z }, 2024-11-01T16:35:19.7465593Z { 2024-11-01T16:35:19.7466207Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7467047Z "size": 863, 2024-11-01T16:35:19.7467814Z "digest": "sha256:7b58778274e4907eb77a53e6b0cf6463f3c8bfc16fdf98d82689b6b6587d4eb2" 2024-11-01T16:35:19.7468967Z }, 2024-11-01T16:35:19.7469390Z { 2024-11-01T16:35:19.7470199Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7471258Z "size": 106, 2024-11-01T16:35:19.7472167Z "digest": "sha256:5252fbf945ae2f66a546f585125adcdd4d649a9d52affc5aa0c3dbc8562383d4" 2024-11-01T16:35:19.7473325Z }, 2024-11-01T16:35:19.7473739Z { 2024-11-01T16:35:19.7474360Z + exit 0 2024-11-01T16:35:19.7475055Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7476124Z "size": 703, 2024-11-01T16:35:19.7477481Z "digest": "sha256:f88e6e245d17df41885b0e0b50d0bbf5fcf08cb77bbe5f9abdfe4160a41c941f" 2024-11-01T16:35:19.7478304Z }, 2024-11-01T16:35:19.7478604Z { 2024-11-01T16:35:19.7479240Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7480136Z "size": 1218, 2024-11-01T16:35:19.7481000Z "digest": "sha256:d2fc2f6d6c5f1f51f1f9f6dbf384df57f88442d4aa80b19db8df5c57f6816eb0" 2024-11-01T16:35:19.7482120Z }, 2024-11-01T16:35:19.7482574Z { 2024-11-01T16:35:19.7483172Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7483748Z "size": 484, 2024-11-01T16:35:19.7484372Z "digest": "sha256:8db79f4cc8cc3ddbbcfbe4298ca789f53943cbe08b801b0ead12437dfa1578bc" 2024-11-01T16:35:19.7485041Z }, 2024-11-01T16:35:19.7485258Z { 2024-11-01T16:35:19.7485689Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7486256Z "size": 110322453, 2024-11-01T16:35:19.7486833Z "digest": "sha256:8f875d71a923483cee22a59531af7746a05a5761d41cabea5514660597dc9099" 2024-11-01T16:35:19.7487461Z }, 2024-11-01T16:35:19.7487693Z { 2024-11-01T16:35:19.7488111Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7488838Z "size": 4119, 2024-11-01T16:35:19.7489398Z "digest": "sha256:d4816a7d80f80b60067bbbfb0445c6df4e0a2f6e69bb501787127ccc2d0a4e03" 2024-11-01T16:35:19.7490048Z }, 2024-11-01T16:35:19.7490280Z { 2024-11-01T16:35:19.7490703Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7491271Z "size": 1801, 2024-11-01T16:35:19.7491826Z "digest": "sha256:2d6bd9e42551a4a156f25c19a7fe61e68252cc88d3f6452e3cb28bebf1357b03" 2024-11-01T16:35:19.7492468Z }, 2024-11-01T16:35:19.7492699Z { 2024-11-01T16:35:19.7493118Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7493682Z "size": 701, 2024-11-01T16:35:19.7494224Z "digest": "sha256:3750343ed4e2963edebf1c1501754874e2e25387d2c1f0bc536995ea7a714449" 2024-11-01T16:35:19.7494863Z }, 2024-11-01T16:35:19.7495098Z { 2024-11-01T16:35:19.7495516Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7496089Z "size": 2803789884, 2024-11-01T16:35:19.7496657Z "digest": "sha256:96047d905613a9bad3fff00cab771685ec4ed5d6811502b45be68418b37368a4" 2024-11-01T16:35:19.7497289Z }, 2024-11-01T16:35:19.7497518Z { 2024-11-01T16:35:19.7497934Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7498495Z "size": 32, 2024-11-01T16:35:19.7499050Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:35:19.7499692Z }, 2024-11-01T16:35:19.7499923Z { 2024-11-01T16:35:19.7500356Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7500905Z "size": 379, 2024-11-01T16:35:19.7501451Z "digest": "sha256:b79cecd6800959fe47b950e969ab4799d268cd7b822143bd7e4c2ffdf173d14f" 2024-11-01T16:35:19.7502088Z }, 2024-11-01T16:35:19.7502325Z { 2024-11-01T16:35:19.7502751Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7503301Z "size": 232174, 2024-11-01T16:35:19.7503875Z "digest": "sha256:efe1e385dc9e6129ba1ce09f274a24e288c5fcb30f5a9ad0cac51d4f9674b2ec" 2024-11-01T16:35:19.7504527Z }, 2024-11-01T16:35:19.7504758Z { 2024-11-01T16:35:19.7505188Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7505742Z "size": 230, 2024-11-01T16:35:19.7506289Z "digest": "sha256:adeab13bac13b551938d4b48819331e51815c24629fe276f34937c1570a904ed" 2024-11-01T16:35:19.7506923Z }, 2024-11-01T16:35:19.7507153Z { 2024-11-01T16:35:19.7507585Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7508140Z "size": 3235670, 2024-11-01T16:35:19.7508703Z "digest": "sha256:24b41e0168fafa36f581e5fa51df4d5c48fb686867ae1f184860c8eee7537483" 2024-11-01T16:35:19.7509342Z }, 2024-11-01T16:35:19.7509573Z { 2024-11-01T16:35:19.7510101Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7510673Z "size": 1954, 2024-11-01T16:35:19.7511189Z "digest": "sha256:707cc0109119630d83e1c884c0250a24d99900309456ae11e6e8b3755af86b2a" 2024-11-01T16:35:19.7511810Z }, 2024-11-01T16:35:19.7512044Z { 2024-11-01T16:35:19.7512475Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7513147Z "size": 105, 2024-11-01T16:35:19.7513688Z "digest": "sha256:c5fa30c2a598a04cd4c417b6187230e12ddbe39c9dc38338e425414c5c148d4a" 2024-11-01T16:35:19.7514326Z }, 2024-11-01T16:35:19.7514558Z { 2024-11-01T16:35:19.7514990Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7515553Z "size": 565, 2024-11-01T16:35:19.7516100Z "digest": "sha256:7c311ff0f8ed17f85cce86bf63f9e9587521fa8ac6c5f21d6face105b4e7703c" 2024-11-01T16:35:19.7516750Z }, 2024-11-01T16:35:19.7516981Z { 2024-11-01T16:35:19.7517419Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7517984Z "size": 32, 2024-11-01T16:35:19.7518522Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:35:19.7519246Z }, 2024-11-01T16:35:19.7519474Z { 2024-11-01T16:35:19.7519906Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7520469Z "size": 104, 2024-11-01T16:35:19.7521010Z "digest": "sha256:6d067a971ea00018d0503daa88fb7473d272e19b039874532b929452d24df868" 2024-11-01T16:35:19.7521652Z }, 2024-11-01T16:35:19.7521885Z { 2024-11-01T16:35:19.7522305Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7522871Z "size": 504, 2024-11-01T16:35:19.7523417Z "digest": "sha256:32a0e0f8d213a1427ff7110069e3d4d3b90cb24b57ede9a30e7b573d649de137" 2024-11-01T16:35:19.7524058Z }, 2024-11-01T16:35:19.7524291Z { 2024-11-01T16:35:19.7524721Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7525279Z "size": 101109882, 2024-11-01T16:35:19.7525842Z "digest": "sha256:6759707dde2383b2b9b6e94894c3acf5619203cc57661e577906f3529c552a57" 2024-11-01T16:35:19.7526471Z }, 2024-11-01T16:35:19.7526700Z { 2024-11-01T16:35:19.7527133Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7527685Z "size": 109, 2024-11-01T16:35:19.7528223Z "digest": "sha256:42a959f7585ea3d548af853d04f831753759ae7793e5066ca18b973ce2402ebe" 2024-11-01T16:35:19.7528844Z }, 2024-11-01T16:35:19.7529073Z { 2024-11-01T16:35:19.7529503Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7530053Z "size": 488, 2024-11-01T16:35:19.7530588Z "digest": "sha256:103e959619f3a92484107a7b88198fff73e2a6c72daa7eec906208b38f69fafe" 2024-11-01T16:35:19.7531210Z }, 2024-11-01T16:35:19.7531440Z { 2024-11-01T16:35:19.7531869Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7532422Z "size": 338, 2024-11-01T16:35:19.7532967Z "digest": "sha256:0d21b1ebe1a48164142d075480095b1e36f294ea8a7f4f147a11bc4576222137" 2024-11-01T16:35:19.7533597Z }, 2024-11-01T16:35:19.7533830Z { 2024-11-01T16:35:19.7534260Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7534823Z "size": 103, 2024-11-01T16:35:19.7535365Z "digest": "sha256:c2036a0c8e1113e3c017bbe6cf0a2b646618ee33c6ffb7ce135d6a2e4a13ab01" 2024-11-01T16:35:19.7536005Z }, 2024-11-01T16:35:19.7536237Z { 2024-11-01T16:35:19.7536666Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7537227Z "size": 1472, 2024-11-01T16:35:19.7537784Z "digest": "sha256:f566c3af6db7fc704e0bbdde98ba341c239a7e11d721b83acb0ce6ab45621b5e" 2024-11-01T16:35:19.7538435Z }, 2024-11-01T16:35:19.7538665Z { 2024-11-01T16:35:19.7539099Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7539664Z "size": 445359487, 2024-11-01T16:35:19.7540283Z "digest": "sha256:177321563b8eb0ca51527060730909bd6da52c0c28ac039ed370521202e36506" 2024-11-01T16:35:19.7540904Z }, 2024-11-01T16:35:19.7541133Z { 2024-11-01T16:35:19.7541571Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7542139Z "size": 164, 2024-11-01T16:35:19.7542662Z "digest": "sha256:088d9767f2d82f854b9b1fc4bda8783868820638188776fb797d65f81e7be245" 2024-11-01T16:35:19.7543284Z }, 2024-11-01T16:35:19.7543516Z { 2024-11-01T16:35:19.7543946Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7544514Z "size": 2211, 2024-11-01T16:35:19.7545077Z "digest": "sha256:180d2736fbf96c3dcbbffce71922e73d97bf2eb7745a6d97cd055ffc2e629329" 2024-11-01T16:35:19.7545713Z }, 2024-11-01T16:35:19.7545943Z { 2024-11-01T16:35:19.7546374Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7546938Z "size": 152, 2024-11-01T16:35:19.7547495Z "digest": "sha256:730a2f77686b27c61998a36afbfa83302fc71af61aeed8721ab2898fde71325f" 2024-11-01T16:35:19.7548118Z }, 2024-11-01T16:35:19.7548346Z { 2024-11-01T16:35:19.7548779Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7549417Z "size": 647, 2024-11-01T16:35:19.7549966Z "digest": "sha256:1af136be95e0e0ea34471cf3d78cd9a3244ff6111018a3f88318558cac12e637" 2024-11-01T16:35:19.7550590Z }, 2024-11-01T16:35:19.7550824Z { 2024-11-01T16:35:19.7551254Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7551817Z "size": 32, 2024-11-01T16:35:19.7552368Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:35:19.7553074Z }, 2024-11-01T16:35:19.7553307Z { 2024-11-01T16:35:19.7553741Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7554308Z "size": 168, 2024-11-01T16:35:19.7554842Z "digest": "sha256:3eb706825457be768f552ea178ead7c52884c4237894b6bb70a23f1787e21b03" 2024-11-01T16:35:19.7555470Z }, 2024-11-01T16:35:19.7555688Z { 2024-11-01T16:35:19.7556114Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7556689Z "size": 109, 2024-11-01T16:35:19.7557241Z "digest": "sha256:67d83ed868e60c0f3717e8bcb8b07ccb0eac05f472908d47be1b7e03c22c5b12" 2024-11-01T16:35:19.7557883Z }, 2024-11-01T16:35:19.7558102Z { 2024-11-01T16:35:19.7558672Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7559275Z "size": 102, 2024-11-01T16:35:19.7559826Z "digest": "sha256:30d4de16074139d72c51e19aa214aec3cc4c988772facbc49e0aea501d886b82" 2024-11-01T16:35:19.7560467Z }, 2024-11-01T16:35:19.7560685Z { 2024-11-01T16:35:19.7561117Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7561683Z "size": 422, 2024-11-01T16:35:19.7562248Z "digest": "sha256:d0d44efe8bc344eb11d0e857603abfbcbb7955f97a120a095f91fecf2c6d89ad" 2024-11-01T16:35:19.7562900Z }, 2024-11-01T16:35:19.7563116Z { 2024-11-01T16:35:19.7563553Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7564119Z "size": 32, 2024-11-01T16:35:19.7564674Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:35:19.7565318Z }, 2024-11-01T16:35:19.7565548Z { 2024-11-01T16:35:19.7565964Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7566525Z "size": 111, 2024-11-01T16:35:19.7567078Z "digest": "sha256:be6faa013cc8e66cb07fff68f375022557fb83272e5840b89b57397b34c59ddd" 2024-11-01T16:35:19.7567720Z }, 2024-11-01T16:35:19.7567950Z { 2024-11-01T16:35:19.7568370Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7568936Z "size": 473, 2024-11-01T16:35:19.7569481Z "digest": "sha256:72dc36b20429f9963dfd9bed81119228749e60f2df3c23aed442f6f53de061d9" 2024-11-01T16:35:19.7570114Z }, 2024-11-01T16:35:19.7570344Z { 2024-11-01T16:35:19.7570926Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7571495Z "size": 32, 2024-11-01T16:35:19.7572043Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:35:19.7572692Z }, 2024-11-01T16:35:19.7572922Z { 2024-11-01T16:35:19.7573338Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7573901Z "size": 112, 2024-11-01T16:35:19.7574442Z "digest": "sha256:058580a8278bc76e35b78df1e3648c77e807dc63bdd3dc90045c4cd9987fd5b9" 2024-11-01T16:35:19.7575074Z }, 2024-11-01T16:35:19.7575306Z { 2024-11-01T16:35:19.7575727Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7576290Z "size": 565, 2024-11-01T16:35:19.7576820Z "digest": "sha256:aae1352538670573fcb31c13d39345fd08ab18f33a30401080b0e409a62c5091" 2024-11-01T16:35:19.7577444Z }, 2024-11-01T16:35:19.7577674Z { 2024-11-01T16:35:19.7578104Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7578769Z "size": 43163839, 2024-11-01T16:35:19.7579340Z "digest": "sha256:3b05bde4486416e4a13919dbf361ae09a65caca35cce09c4b59d3deeb11d547b" 2024-11-01T16:35:19.7580066Z }, 2024-11-01T16:35:19.7580285Z { 2024-11-01T16:35:19.7580720Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7581290Z "size": 106, 2024-11-01T16:35:19.7581817Z "digest": "sha256:292e740c61872884758b278457310312e91b4c53c83f72add38b6b01592e2f74" 2024-11-01T16:35:19.7582427Z }, 2024-11-01T16:35:19.7582646Z { 2024-11-01T16:35:19.7583080Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7583645Z "size": 294, 2024-11-01T16:35:19.7584188Z "digest": "sha256:d715265d8db72d2d32d52bd703d48a4e1108f4d0e13d316639d21b174e541a9a" 2024-11-01T16:35:19.7584821Z }, 2024-11-01T16:35:19.7585042Z { 2024-11-01T16:35:19.7585471Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7586039Z "size": 32, 2024-11-01T16:35:19.7586599Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:35:19.7587236Z }, 2024-11-01T16:35:19.7628300Z { 2024-11-01T16:35:19.7628879Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7629464Z "size": 106, 2024-11-01T16:35:19.7630122Z "digest": "sha256:3634fe4cf0c4e625f72a5c2443656cd99d3a8faa8f547092a68e4efeb76761f8" 2024-11-01T16:35:19.7630788Z }, 2024-11-01T16:35:19.7631045Z { 2024-11-01T16:35:19.7631470Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7632031Z "size": 425, 2024-11-01T16:35:19.7632584Z "digest": "sha256:da04921cbec53df59b0cd9df3ff3c8747096a5f7f4b8fe9d0bbb87613ab4ca00" 2024-11-01T16:35:19.7633345Z }, 2024-11-01T16:35:19.7633556Z { 2024-11-01T16:35:19.7633965Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7634520Z "size": 19308710, 2024-11-01T16:35:19.7635084Z "digest": "sha256:55b1e8dc281d64beab6693bf2031e444dc01e92b93c1d7b5ceecd7318b0c880a" 2024-11-01T16:35:19.7635711Z }, 2024-11-01T16:35:19.7635918Z { 2024-11-01T16:35:19.7636331Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7636877Z "size": 108, 2024-11-01T16:35:19.7637401Z "digest": "sha256:595dea37c8f8e777f19f790f768d0b68787b02d34eee74c8fc16d783c63ecd38" 2024-11-01T16:35:19.7638014Z }, 2024-11-01T16:35:19.7638241Z { 2024-11-01T16:35:19.7638649Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7639186Z "size": 437, 2024-11-01T16:35:19.7639728Z "digest": "sha256:1edfdc5afa7b44c396a07cb196a816e1c1660b731e6346b0cba3ed62d672b4d7" 2024-11-01T16:35:19.7640371Z }, 2024-11-01T16:35:19.7640590Z { 2024-11-01T16:35:19.7641003Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7641544Z "size": 701, 2024-11-01T16:35:19.7642184Z "digest": "sha256:3750343ed4e2963edebf1c1501754874e2e25387d2c1f0bc536995ea7a714449" 2024-11-01T16:35:19.7642793Z }, 2024-11-01T16:35:19.7643004Z { 2024-11-01T16:35:19.7643412Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7643953Z "size": 141, 2024-11-01T16:35:19.7644483Z "digest": "sha256:f64e657da3cf04d6bc43e8de98abe0d33741737767ed0bc295e37b1dc9751ce3" 2024-11-01T16:35:19.7645114Z }, 2024-11-01T16:35:19.7645323Z { 2024-11-01T16:35:19.7645740Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7646277Z "size": 135, 2024-11-01T16:35:19.7646808Z "digest": "sha256:de853ca4b926f76e26c46b6be0e816947e2e0ed767ea71164bf5795dd68610e4" 2024-11-01T16:35:19.7647430Z }, 2024-11-01T16:35:19.7647637Z { 2024-11-01T16:35:19.7648046Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7648596Z "size": 32, 2024-11-01T16:35:19.7649134Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:35:19.7649760Z }, 2024-11-01T16:35:19.7649986Z { 2024-11-01T16:35:19.7650398Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7651032Z "size": 195, 2024-11-01T16:35:19.7651574Z "digest": "sha256:71d440813d6d606e2d2fee5dfad585658b402b3fbe579c98fb79dccc9dd27a0c" 2024-11-01T16:35:19.7652196Z }, 2024-11-01T16:35:19.7652403Z { 2024-11-01T16:35:19.7652813Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7653349Z "size": 1402, 2024-11-01T16:35:19.7653884Z "digest": "sha256:cee9a121085f4b19d27c1952ee154e16ce0a0eb6d47848030a92de7b6c668118" 2024-11-01T16:35:19.7654509Z }, 2024-11-01T16:35:19.7654719Z { 2024-11-01T16:35:19.7655133Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7655684Z "size": 701, 2024-11-01T16:35:19.7656195Z "digest": "sha256:3750343ed4e2963edebf1c1501754874e2e25387d2c1f0bc536995ea7a714449" 2024-11-01T16:35:19.7656804Z }, 2024-11-01T16:35:19.7657025Z { 2024-11-01T16:35:19.7657446Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7658002Z "size": 136, 2024-11-01T16:35:19.7658722Z "digest": "sha256:f749ccdf4d4636ca276233c3403c73c5c3f9059f77de1f220af09ae0d402f1db" 2024-11-01T16:35:19.7659357Z }, 2024-11-01T16:35:19.7659586Z { 2024-11-01T16:35:19.7660012Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7660571Z "size": 3691884935, 2024-11-01T16:35:19.7661141Z "digest": "sha256:1ffba2c07a780d4ab29bb558e28915358b9661875d01c7a570f6c4eda134c457" 2024-11-01T16:35:19.7661765Z }, 2024-11-01T16:35:19.7661985Z { 2024-11-01T16:35:19.7662403Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7662960Z "size": 156, 2024-11-01T16:35:19.7663507Z "digest": "sha256:d474c4eb5d81b5758eda41d6981795dbc39db496d327b1cba41762898c5f9112" 2024-11-01T16:35:19.7664129Z }, 2024-11-01T16:35:19.7664354Z { 2024-11-01T16:35:19.7664776Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7665324Z "size": 1402, 2024-11-01T16:35:19.7665863Z "digest": "sha256:cee9a121085f4b19d27c1952ee154e16ce0a0eb6d47848030a92de7b6c668118" 2024-11-01T16:35:19.7666492Z }, 2024-11-01T16:35:19.7666719Z { 2024-11-01T16:35:19.7667148Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7667711Z "size": 701, 2024-11-01T16:35:19.7668249Z "digest": "sha256:3750343ed4e2963edebf1c1501754874e2e25387d2c1f0bc536995ea7a714449" 2024-11-01T16:35:19.7668874Z }, 2024-11-01T16:35:19.7669093Z { 2024-11-01T16:35:19.7669520Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7670083Z "size": 140, 2024-11-01T16:35:19.7670600Z "digest": "sha256:b30192601766675f418ce429c021330541e583716511b8c110a80f8b4512d1a3" 2024-11-01T16:35:19.7671204Z }, 2024-11-01T16:35:19.7671424Z { 2024-11-01T16:35:19.7671985Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7672553Z "size": 32, 2024-11-01T16:35:19.7673180Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:35:19.7673831Z }, 2024-11-01T16:35:19.7674049Z { 2024-11-01T16:35:19.7674480Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7675045Z "size": 161, 2024-11-01T16:35:19.7675593Z "digest": "sha256:e927cdba5b8a8610a7c6a0219be646c3177e4e6638587da97c2d8ad0e49e6be1" 2024-11-01T16:35:19.7676232Z }, 2024-11-01T16:35:19.7676449Z { 2024-11-01T16:35:19.7676878Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7677443Z "size": 849, 2024-11-01T16:35:19.7677986Z "digest": "sha256:859f5d9a55a371b64c430b8ccbe2a96ae24550f9fbba87228c14452276b1b1f8" 2024-11-01T16:35:19.7678624Z }, 2024-11-01T16:35:19.7678840Z { 2024-11-01T16:35:19.7679269Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7679836Z "size": 701, 2024-11-01T16:35:19.7680367Z "digest": "sha256:3750343ed4e2963edebf1c1501754874e2e25387d2c1f0bc536995ea7a714449" 2024-11-01T16:35:19.7681081Z }, 2024-11-01T16:35:19.7681313Z { 2024-11-01T16:35:19.7681729Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7682295Z "size": 141, 2024-11-01T16:35:19.7682847Z "digest": "sha256:e992e1c9030386aaccc6a4f0f3b1e81b9aee1474f51b629b585b5994eecbc829" 2024-11-01T16:35:19.7683484Z }, 2024-11-01T16:35:19.7683715Z { 2024-11-01T16:35:19.7684132Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7684694Z "size": 32, 2024-11-01T16:35:19.7685243Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:35:19.7685867Z }, 2024-11-01T16:35:19.7686135Z { 2024-11-01T16:35:19.7686550Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7687114Z "size": 160, 2024-11-01T16:35:19.7687670Z "digest": "sha256:ded8f639d8e87a4510e9b4105897df92d8dd54138801ba7eb1fcdbf84b8d2abf" 2024-11-01T16:35:19.7688315Z }, 2024-11-01T16:35:19.7688545Z { 2024-11-01T16:35:19.7688982Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7689536Z "size": 908, 2024-11-01T16:35:19.7690091Z "digest": "sha256:6dc1988800dbe97274eba8f1acb6c4ab80a79e00c8cf3b30f06d8d8420ee559a" 2024-11-01T16:35:19.7690738Z }, 2024-11-01T16:35:19.7690968Z { 2024-11-01T16:35:19.7691395Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7691944Z "size": 701, 2024-11-01T16:35:19.7692481Z "digest": "sha256:3750343ed4e2963edebf1c1501754874e2e25387d2c1f0bc536995ea7a714449" 2024-11-01T16:35:19.7693111Z }, 2024-11-01T16:35:19.7693340Z { 2024-11-01T16:35:19.7693772Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7694325Z "size": 135, 2024-11-01T16:35:19.7694876Z "digest": "sha256:6b07caec55f88a858b123d2e3915be8715e74f071293bcf582062fd5e03a822b" 2024-11-01T16:35:19.7695512Z }, 2024-11-01T16:35:19.7695742Z { 2024-11-01T16:35:19.7696170Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7696726Z "size": 32, 2024-11-01T16:35:19.7697280Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:35:19.7697926Z }, 2024-11-01T16:35:19.7698155Z { 2024-11-01T16:35:19.7698586Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7699151Z "size": 159, 2024-11-01T16:35:19.7699683Z "digest": "sha256:fe72746ef78d12232e6562062d423a61aad2fb25ef96a9c056ba01be982b4cc6" 2024-11-01T16:35:19.7700319Z }, 2024-11-01T16:35:19.7700548Z { 2024-11-01T16:35:19.7700975Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7701535Z "size": 1520, 2024-11-01T16:35:19.7702049Z "digest": "sha256:41244b047648e4071fe28c5fc86d3499897770370d8c822224af271c51dd3e40" 2024-11-01T16:35:19.7702739Z }, 2024-11-01T16:35:19.7702978Z { 2024-11-01T16:35:19.7703409Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7703982Z "size": 32, 2024-11-01T16:35:19.7704525Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:35:19.7705169Z }, 2024-11-01T16:35:19.7705401Z { 2024-11-01T16:35:19.7705829Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7706394Z "size": 137, 2024-11-01T16:35:19.7706953Z "digest": "sha256:c5ca3a0d4a1103ec8bbdca66eb94f1b68bf19868aafeb16435a54b347a668dbd" 2024-11-01T16:35:19.7707609Z }, 2024-11-01T16:35:19.7707837Z { 2024-11-01T16:35:19.7708267Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7708831Z "size": 379, 2024-11-01T16:35:19.7709383Z "digest": "sha256:1f3eab9dfbb47a90acc80106d65b4e2be05b1c63fe0b4e66216e0fc00db041df" 2024-11-01T16:35:19.7710037Z }, 2024-11-01T16:35:19.7710271Z { 2024-11-01T16:35:19.7710703Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7711263Z "size": 32, 2024-11-01T16:35:19.7711868Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:35:19.7712499Z }, 2024-11-01T16:35:19.7712729Z { 2024-11-01T16:35:19.7713254Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7713820Z "size": 104, 2024-11-01T16:35:19.7714388Z "digest": "sha256:805d5faba3b6ebd33be5a3de08e0cfbfba236d4b103d466b47269c4841849812" 2024-11-01T16:35:19.7715032Z }, 2024-11-01T16:35:19.7715265Z { 2024-11-01T16:35:19.7715694Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7716258Z "size": 2048, 2024-11-01T16:35:19.7716832Z "digest": "sha256:b37c2b2cc48af121a14c5d384cd7ead2eec0bf8e82def2c7823cfc0e71e48e21" 2024-11-01T16:35:19.7717475Z }, 2024-11-01T16:35:19.7717705Z { 2024-11-01T16:35:19.7718137Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7718701Z "size": 197353626, 2024-11-01T16:35:19.7719292Z "digest": "sha256:a58aebee5440abfdb53c19e05ff4daf75f3d8b3cef88cc3007a3103d37997582" 2024-11-01T16:35:19.7719951Z }, 2024-11-01T16:35:19.7720169Z { 2024-11-01T16:35:19.7720596Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7721159Z "size": 106, 2024-11-01T16:35:19.7721713Z "digest": "sha256:aa5c2183b5a66c0efc47604dfa96218eb476bdddf5403a33534c7160147a916c" 2024-11-01T16:35:19.7722353Z }, 2024-11-01T16:35:19.7722571Z { 2024-11-01T16:35:19.7723000Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7723563Z "size": 164, 2024-11-01T16:35:19.7724108Z "digest": "sha256:0c906cd3d76d4a13859d1d59c1a43b875b1c4a16e8b80bb47844de697e5d00cd" 2024-11-01T16:35:19.7724741Z }, 2024-11-01T16:35:19.7724958Z { 2024-11-01T16:35:19.7725391Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7725950Z "size": 7943, 2024-11-01T16:35:19.7726520Z "digest": "sha256:4fad3dce2eb75ae56cd22ed1352a6b8dc7a618a652098ead52eec7769a56be1d" 2024-11-01T16:35:19.7727182Z }, 2024-11-01T16:35:19.7727395Z { 2024-11-01T16:35:19.7727824Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7728384Z "size": 8069, 2024-11-01T16:35:19.7728920Z "digest": "sha256:3ee82f0066227fb3291568703c059c81d519b068cb58c4d39dff639c46865bce" 2024-11-01T16:35:19.7729542Z }, 2024-11-01T16:35:19.7729762Z { 2024-11-01T16:35:19.7730191Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7730749Z "size": 303, 2024-11-01T16:35:19.7731305Z "digest": "sha256:dbf98406efc0952d4ca62e3e458eb17ad02e4590f683b5ad5906b4dd7f2f0836" 2024-11-01T16:35:19.7731946Z }, 2024-11-01T16:35:19.7732178Z { 2024-11-01T16:35:19.7732595Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7733658Z "size": 32, 2024-11-01T16:35:19.7734221Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:35:19.7734875Z }, 2024-11-01T16:35:19.7735108Z { 2024-11-01T16:35:19.7735526Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7736088Z "size": 108, 2024-11-01T16:35:19.7736655Z "digest": "sha256:9cb7ede94ead5eb859cb6a83ce084d0675ada6c535cfa7c0ac18e1aca2034256" 2024-11-01T16:35:19.7737317Z }, 2024-11-01T16:35:19.7737546Z { 2024-11-01T16:35:19.7737966Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7738528Z "size": 54145661, 2024-11-01T16:35:19.7739105Z "digest": "sha256:81ba6fd7879d26a34e11f209f0ac8bacb2edb20bd8c295699b1d71f701e857ce" 2024-11-01T16:35:19.7739750Z }, 2024-11-01T16:35:19.7739979Z { 2024-11-01T16:35:19.7740395Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:35:19.7740958Z "size": 32, 2024-11-01T16:35:19.7741513Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:35:19.7742155Z } 2024-11-01T16:35:19.7742446Z ] 2024-11-01T16:35:19.7742662Z } 2024-11-01T16:35:19.7862698Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-11-01T16:35:19.7863179Z tag=${ECR_DOCKER_IMAGE##*/} 2024-11-01T16:35:19.7863732Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-11-01T16:35:19.7870035Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:35:19.7870525Z env: 2024-11-01T16:35:19.7870790Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:35:19.7871773Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:35:19.7872873Z ##[endgroup] 2024-11-01T16:35:19.7897861Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:35:19.7962929Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-11-01T16:35:19.7963508Z with: 2024-11-01T16:35:19.7964397Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:35:19.7965802Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:35:19.7966330Z env: 2024-11-01T16:35:19.7966595Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:35:19.7966921Z ##[endgroup] 2024-11-01T16:35:19.7985354Z ##[group]Run set -x 2024-11-01T16:35:19.7985694Z set -x 2024-11-01T16:35:19.7985985Z set +e 2024-11-01T16:35:19.7986270Z  2024-11-01T16:35:19.7986535Z login() { 2024-11-01T16:35:19.7987210Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-11-01T16:35:19.7987954Z } 2024-11-01T16:35:19.7988201Z  2024-11-01T16:35:19.7988506Z retry () { 2024-11-01T16:35:19.7988886Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-11-01T16:35:19.7989318Z } 2024-11-01T16:35:19.7989576Z  2024-11-01T16:35:19.7989881Z retry login "${DOCKER_REGISTRY}" 2024-11-01T16:35:19.7990551Z  2024-11-01T16:35:19.7990814Z set -e 2024-11-01T16:35:19.7991278Z # ignore output since only exit code is used for conditional 2024-11-01T16:35:19.7992003Z # only pull docker image if it's not available locally 2024-11-01T16:35:19.7992865Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-11-01T16:35:19.7993588Z  retry docker pull "${DOCKER_IMAGE}" 2024-11-01T16:35:19.7994034Z fi 2024-11-01T16:35:19.7999403Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:35:19.7999910Z env: 2024-11-01T16:35:19.8000162Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:35:19.8001132Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:35:19.8002277Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:35:19.8002808Z ##[endgroup] 2024-11-01T16:35:19.8024254Z + set +e 2024-11-01T16:35:19.8024846Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:35:19.8025513Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:35:19.8027770Z + aws ecr get-login-password --region us-east-1 2024-11-01T16:35:19.8028896Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:35:20.3479579Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-11-01T16:35:20.3480886Z Configure a credential helper to remove this warning. See 2024-11-01T16:35:20.3481842Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-11-01T16:35:20.3482402Z 2024-11-01T16:35:20.3482528Z Login Succeeded 2024-11-01T16:35:20.3491201Z + set -e 2024-11-01T16:35:20.3492434Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:35:20.3614761Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:35:20.3616603Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:35:20.5802206Z bdd298b12da59246147f016e0693ffd722419941: Pulling from pytorch/pytorch-linux-jammy-py3.9-gcc11 2024-11-01T16:35:20.5803446Z 7478e0ac0f23: Pulling fs layer 2024-11-01T16:35:20.5804098Z ad0341f900ba: Pulling fs layer 2024-11-01T16:35:20.5804710Z be63e049eed3: Pulling fs layer 2024-11-01T16:35:20.5805343Z 7b58778274e4: Pulling fs layer 2024-11-01T16:35:20.5805999Z 5252fbf945ae: Pulling fs layer 2024-11-01T16:35:20.5807038Z f88e6e245d17: Pulling fs layer 2024-11-01T16:35:20.5807748Z d2fc2f6d6c5f: Pulling fs layer 2024-11-01T16:35:20.5808423Z 8db79f4cc8cc: Pulling fs layer 2024-11-01T16:35:20.5809080Z 8f875d71a923: Pulling fs layer 2024-11-01T16:35:20.5809760Z d4816a7d80f8: Pulling fs layer 2024-11-01T16:35:20.5810328Z 7b58778274e4: Waiting 2024-11-01T16:35:20.5810868Z 2d6bd9e42551: Pulling fs layer 2024-11-01T16:35:20.5811553Z 3750343ed4e2: Pulling fs layer 2024-11-01T16:35:20.5812213Z 96047d905613: Pulling fs layer 2024-11-01T16:35:20.5812792Z 4f4fb700ef54: Pulling fs layer 2024-11-01T16:35:20.5813411Z b79cecd68009: Pulling fs layer 2024-11-01T16:35:20.5814015Z efe1e385dc9e: Pulling fs layer 2024-11-01T16:35:20.5814655Z adeab13bac13: Pulling fs layer 2024-11-01T16:35:20.5815393Z 5252fbf945ae: Waiting 2024-11-01T16:35:20.5815994Z 24b41e0168fa: Pulling fs layer 2024-11-01T16:35:20.5816674Z 707cc0109119: Pulling fs layer 2024-11-01T16:35:20.5817365Z c5fa30c2a598: Pulling fs layer 2024-11-01T16:35:20.5817891Z 7c311ff0f8ed: Pulling fs layer 2024-11-01T16:35:20.5818258Z 6d067a971ea0: Pulling fs layer 2024-11-01T16:35:20.5818629Z 32a0e0f8d213: Pulling fs layer 2024-11-01T16:35:20.5818996Z 6759707dde23: Pulling fs layer 2024-11-01T16:35:20.5819362Z 42a959f7585e: Pulling fs layer 2024-11-01T16:35:20.5819732Z 103e959619f3: Pulling fs layer 2024-11-01T16:35:20.5820084Z 0d21b1ebe1a4: Pulling fs layer 2024-11-01T16:35:20.5820470Z c2036a0c8e11: Pulling fs layer 2024-11-01T16:35:20.5820849Z f566c3af6db7: Pulling fs layer 2024-11-01T16:35:20.5821203Z f88e6e245d17: Waiting 2024-11-01T16:35:20.5821517Z d2fc2f6d6c5f: Waiting 2024-11-01T16:35:20.5821852Z 177321563b8e: Pulling fs layer 2024-11-01T16:35:20.5822207Z 8db79f4cc8cc: Waiting 2024-11-01T16:35:20.5822531Z 088d9767f2d8: Pulling fs layer 2024-11-01T16:35:20.5822898Z 180d2736fbf9: Pulling fs layer 2024-11-01T16:35:20.5823261Z 730a2f77686b: Pulling fs layer 2024-11-01T16:35:20.5823598Z 8f875d71a923: Waiting 2024-11-01T16:35:20.5823922Z 1af136be95e0: Pulling fs layer 2024-11-01T16:35:20.5824278Z d4816a7d80f8: Waiting 2024-11-01T16:35:20.5824599Z 3eb706825457: Pulling fs layer 2024-11-01T16:35:20.5824969Z 67d83ed868e6: Pulling fs layer 2024-11-01T16:35:20.5825314Z b79cecd68009: Waiting 2024-11-01T16:35:20.5825622Z 2d6bd9e42551: Waiting 2024-11-01T16:35:20.5825933Z 24b41e0168fa: Waiting 2024-11-01T16:35:20.5826253Z 30d4de160741: Pulling fs layer 2024-11-01T16:35:20.5826613Z efe1e385dc9e: Waiting 2024-11-01T16:35:20.5826907Z 707cc0109119: Waiting 2024-11-01T16:35:20.5827209Z 180d2736fbf9: Waiting 2024-11-01T16:35:20.5827515Z adeab13bac13: Waiting 2024-11-01T16:35:20.5827820Z 730a2f77686b: Waiting 2024-11-01T16:35:20.5828123Z c5fa30c2a598: Waiting 2024-11-01T16:35:20.5828432Z d0d44efe8bc3: Pulling fs layer 2024-11-01T16:35:20.5828797Z 0d21b1ebe1a4: Waiting 2024-11-01T16:35:20.5829105Z 1af136be95e0: Waiting 2024-11-01T16:35:20.5829410Z 7c311ff0f8ed: Waiting 2024-11-01T16:35:20.5829715Z c2036a0c8e11: Waiting 2024-11-01T16:35:20.5830025Z be6faa013cc8: Pulling fs layer 2024-11-01T16:35:20.5830384Z 3eb706825457: Waiting 2024-11-01T16:35:20.5830690Z f566c3af6db7: Waiting 2024-11-01T16:35:20.5830992Z 3750343ed4e2: Waiting 2024-11-01T16:35:20.5831309Z 72dc36b20429: Pulling fs layer 2024-11-01T16:35:20.5831653Z 67d83ed868e6: Waiting 2024-11-01T16:35:20.5832144Z 6d067a971ea0: Waiting 2024-11-01T16:35:20.5832497Z 177321563b8e: Waiting 2024-11-01T16:35:20.5832884Z 30d4de160741: Waiting 2024-11-01T16:35:20.5833384Z 058580a8278b: Pulling fs layer 2024-11-01T16:35:20.5833825Z d0d44efe8bc3: Waiting 2024-11-01T16:35:20.5834147Z 088d9767f2d8: Waiting 2024-11-01T16:35:20.5834494Z 32a0e0f8d213: Waiting 2024-11-01T16:35:20.5834820Z aae135253867: Pulling fs layer 2024-11-01T16:35:20.5835243Z 103e959619f3: Waiting 2024-11-01T16:35:20.5835629Z 3b05bde44864: Pulling fs layer 2024-11-01T16:35:20.5835988Z 42a959f7585e: Waiting 2024-11-01T16:35:20.5836325Z 6759707dde23: Waiting 2024-11-01T16:35:20.5836641Z 292e740c6187: Pulling fs layer 2024-11-01T16:35:20.5837015Z d715265d8db7: Pulling fs layer 2024-11-01T16:35:20.5837372Z be6faa013cc8: Waiting 2024-11-01T16:35:20.5837808Z 72dc36b20429: Waiting 2024-11-01T16:35:20.5838133Z 3634fe4cf0c4: Pulling fs layer 2024-11-01T16:35:20.5838510Z da04921cbec5: Pulling fs layer 2024-11-01T16:35:20.5838889Z 55b1e8dc281d: Pulling fs layer 2024-11-01T16:35:20.5839295Z 058580a8278b: Waiting 2024-11-01T16:35:20.5839601Z 595dea37c8f8: Pulling fs layer 2024-11-01T16:35:20.5839955Z 4f4fb700ef54: Waiting 2024-11-01T16:35:20.5840261Z aae135253867: Waiting 2024-11-01T16:35:20.5840567Z d715265d8db7: Waiting 2024-11-01T16:35:20.5840891Z 1edfdc5afa7b: Pulling fs layer 2024-11-01T16:35:20.5841240Z 3b05bde44864: Waiting 2024-11-01T16:35:20.5841542Z 292e740c6187: Waiting 2024-11-01T16:35:20.5841843Z da04921cbec5: Waiting 2024-11-01T16:35:20.5842166Z f64e657da3cf: Pulling fs layer 2024-11-01T16:35:20.5842522Z 55b1e8dc281d: Waiting 2024-11-01T16:35:20.5842814Z 595dea37c8f8: Waiting 2024-11-01T16:35:20.5843119Z 1edfdc5afa7b: Waiting 2024-11-01T16:35:20.5843443Z de853ca4b926: Pulling fs layer 2024-11-01T16:35:20.5843808Z f64e657da3cf: Waiting 2024-11-01T16:35:20.5844112Z de853ca4b926: Waiting 2024-11-01T16:35:20.5844417Z 71d440813d6d: Pulling fs layer 2024-11-01T16:35:20.5844791Z cee9a121085f: Pulling fs layer 2024-11-01T16:35:20.5845167Z f749ccdf4d46: Pulling fs layer 2024-11-01T16:35:20.5845545Z 1ffba2c07a78: Pulling fs layer 2024-11-01T16:35:20.5845918Z d474c4eb5d81: Pulling fs layer 2024-11-01T16:35:20.5846259Z cee9a121085f: Waiting 2024-11-01T16:35:20.5846569Z 1ffba2c07a78: Waiting 2024-11-01T16:35:20.5846877Z 71d440813d6d: Waiting 2024-11-01T16:35:20.5847181Z f749ccdf4d46: Waiting 2024-11-01T16:35:20.5847501Z b30192601766: Pulling fs layer 2024-11-01T16:35:20.5847858Z e927cdba5b8a: Pulling fs layer 2024-11-01T16:35:20.5848213Z b30192601766: Waiting 2024-11-01T16:35:20.5848533Z 859f5d9a55a3: Pulling fs layer 2024-11-01T16:35:20.5848889Z e927cdba5b8a: Waiting 2024-11-01T16:35:20.5849211Z e992e1c90303: Pulling fs layer 2024-11-01T16:35:20.5849551Z e992e1c90303: Waiting 2024-11-01T16:35:20.5849875Z ded8f639d8e8: Pulling fs layer 2024-11-01T16:35:20.5850253Z 6dc1988800db: Pulling fs layer 2024-11-01T16:35:20.5850626Z 6b07caec55f8: Pulling fs layer 2024-11-01T16:35:20.5850998Z fe72746ef78d: Pulling fs layer 2024-11-01T16:35:20.5851351Z 41244b047648: Pulling fs layer 2024-11-01T16:35:20.5851712Z ded8f639d8e8: Waiting 2024-11-01T16:35:20.5852037Z c5ca3a0d4a11: Pulling fs layer 2024-11-01T16:35:20.5852389Z 6dc1988800db: Waiting 2024-11-01T16:35:20.5852714Z 1f3eab9dfbb4: Pulling fs layer 2024-11-01T16:35:20.5853063Z fe72746ef78d: Waiting 2024-11-01T16:35:20.5853388Z 805d5faba3b6: Pulling fs layer 2024-11-01T16:35:20.5853760Z b37c2b2cc48a: Pulling fs layer 2024-11-01T16:35:20.5854134Z a58aebee5440: Pulling fs layer 2024-11-01T16:35:20.5854504Z aa5c2183b5a6: Pulling fs layer 2024-11-01T16:35:20.5854858Z 0c906cd3d76d: Pulling fs layer 2024-11-01T16:35:20.5855231Z 4fad3dce2eb7: Pulling fs layer 2024-11-01T16:35:20.5855601Z 3ee82f006622: Pulling fs layer 2024-11-01T16:35:20.5855969Z dbf98406efc0: Pulling fs layer 2024-11-01T16:35:20.5856332Z 1f3eab9dfbb4: Waiting 2024-11-01T16:35:20.5856631Z 6b07caec55f8: Waiting 2024-11-01T16:35:20.5856938Z aa5c2183b5a6: Waiting 2024-11-01T16:35:20.5857262Z 9cb7ede94ead: Pulling fs layer 2024-11-01T16:35:20.5857624Z b37c2b2cc48a: Waiting 2024-11-01T16:35:20.5858036Z c5ca3a0d4a11: Waiting 2024-11-01T16:35:20.5858334Z a58aebee5440: Waiting 2024-11-01T16:35:20.5858868Z 81ba6fd7879d: Pulling fs layer 2024-11-01T16:35:20.5859233Z 3ee82f006622: Waiting 2024-11-01T16:35:20.5859544Z 9cb7ede94ead: Waiting 2024-11-01T16:35:20.5859857Z dbf98406efc0: Waiting 2024-11-01T16:35:20.5860152Z 81ba6fd7879d: Waiting 2024-11-01T16:35:20.5860464Z 4fad3dce2eb7: Waiting 2024-11-01T16:35:20.5860770Z 0c906cd3d76d: Waiting 2024-11-01T16:35:20.6449272Z ad0341f900ba: Verifying Checksum 2024-11-01T16:35:20.6449829Z ad0341f900ba: Download complete 2024-11-01T16:35:20.7639429Z 7b58778274e4: Verifying Checksum 2024-11-01T16:35:20.7640097Z 7b58778274e4: Download complete 2024-11-01T16:35:20.8501085Z 5252fbf945ae: Verifying Checksum 2024-11-01T16:35:20.8502034Z 5252fbf945ae: Download complete 2024-11-01T16:35:20.9305907Z 7478e0ac0f23: Verifying Checksum 2024-11-01T16:35:20.9306955Z 7478e0ac0f23: Download complete 2024-11-01T16:35:20.9519607Z f88e6e245d17: Download complete 2024-11-01T16:35:21.0105234Z d2fc2f6d6c5f: Download complete 2024-11-01T16:35:21.0152115Z 8db79f4cc8cc: Verifying Checksum 2024-11-01T16:35:21.0152982Z 8db79f4cc8cc: Download complete 2024-11-01T16:35:21.0849701Z d4816a7d80f8: Download complete 2024-11-01T16:35:21.1726650Z 2d6bd9e42551: Download complete 2024-11-01T16:35:21.2845398Z 3750343ed4e2: Verifying Checksum 2024-11-01T16:35:21.2846174Z 3750343ed4e2: Download complete 2024-11-01T16:35:21.8750791Z 7478e0ac0f23: Pull complete 2024-11-01T16:35:21.8938004Z ad0341f900ba: Pull complete 2024-11-01T16:35:22.1690823Z 8f875d71a923: Verifying Checksum 2024-11-01T16:35:22.1691466Z 8f875d71a923: Download complete 2024-11-01T16:35:22.1759543Z 4f4fb700ef54: Verifying Checksum 2024-11-01T16:35:22.1760156Z 4f4fb700ef54: Download complete 2024-11-01T16:35:22.2441867Z b79cecd68009: Download complete 2024-11-01T16:35:22.3300382Z efe1e385dc9e: Download complete 2024-11-01T16:35:22.4023228Z adeab13bac13: Verifying Checksum 2024-11-01T16:35:22.4023772Z adeab13bac13: Download complete 2024-11-01T16:35:22.4865221Z 24b41e0168fa: Verifying Checksum 2024-11-01T16:35:22.4865759Z 24b41e0168fa: Download complete 2024-11-01T16:35:22.5489004Z 707cc0109119: Verifying Checksum 2024-11-01T16:35:22.5489536Z 707cc0109119: Download complete 2024-11-01T16:35:22.6167180Z c5fa30c2a598: Verifying Checksum 2024-11-01T16:35:22.6167841Z c5fa30c2a598: Download complete 2024-11-01T16:35:22.6803803Z 7c311ff0f8ed: Download complete 2024-11-01T16:35:22.7517528Z 6d067a971ea0: Verifying Checksum 2024-11-01T16:35:22.7518212Z 6d067a971ea0: Download complete 2024-11-01T16:35:22.8498869Z 32a0e0f8d213: Download complete 2024-11-01T16:35:23.7740073Z be63e049eed3: Verifying Checksum 2024-11-01T16:35:23.7740771Z be63e049eed3: Download complete 2024-11-01T16:35:23.8410427Z 42a959f7585e: Verifying Checksum 2024-11-01T16:35:23.9062332Z 103e959619f3: Verifying Checksum 2024-11-01T16:35:23.9063144Z 103e959619f3: Download complete 2024-11-01T16:35:23.9226542Z 6759707dde23: Verifying Checksum 2024-11-01T16:35:23.9227272Z 6759707dde23: Download complete 2024-11-01T16:35:23.9690831Z 0d21b1ebe1a4: Download complete 2024-11-01T16:35:24.0028908Z c2036a0c8e11: Verifying Checksum 2024-11-01T16:35:24.0029459Z c2036a0c8e11: Download complete 2024-11-01T16:35:24.0359194Z f566c3af6db7: Verifying Checksum 2024-11-01T16:35:24.0360021Z f566c3af6db7: Download complete 2024-11-01T16:35:24.1084336Z 088d9767f2d8: Download complete 2024-11-01T16:35:24.1732717Z 180d2736fbf9: Verifying Checksum 2024-11-01T16:35:24.1733322Z 180d2736fbf9: Download complete 2024-11-01T16:35:24.2625401Z 730a2f77686b: Verifying Checksum 2024-11-01T16:35:24.2625978Z 730a2f77686b: Download complete 2024-11-01T16:35:24.3267118Z 1af136be95e0: Verifying Checksum 2024-11-01T16:35:24.3267887Z 1af136be95e0: Download complete 2024-11-01T16:35:24.3921218Z 3eb706825457: Verifying Checksum 2024-11-01T16:35:24.3921802Z 3eb706825457: Download complete 2024-11-01T16:35:24.4795107Z 67d83ed868e6: Verifying Checksum 2024-11-01T16:35:24.4795656Z 67d83ed868e6: Download complete 2024-11-01T16:35:24.5620867Z 30d4de160741: Verifying Checksum 2024-11-01T16:35:24.5621719Z 30d4de160741: Download complete 2024-11-01T16:35:24.6831887Z d0d44efe8bc3: Verifying Checksum 2024-11-01T16:35:24.6832580Z d0d44efe8bc3: Download complete 2024-11-01T16:35:24.7643025Z be6faa013cc8: Verifying Checksum 2024-11-01T16:35:24.7643762Z be6faa013cc8: Download complete 2024-11-01T16:35:24.8316528Z 72dc36b20429: Verifying Checksum 2024-11-01T16:35:24.8317250Z 72dc36b20429: Download complete 2024-11-01T16:35:24.9147044Z 058580a8278b: Download complete 2024-11-01T16:35:24.9796144Z aae135253867: Verifying Checksum 2024-11-01T16:35:24.9796861Z aae135253867: Download complete 2024-11-01T16:35:25.4527541Z 3b05bde44864: Verifying Checksum 2024-11-01T16:35:25.4528708Z 3b05bde44864: Download complete 2024-11-01T16:35:25.5285220Z 292e740c6187: Download complete 2024-11-01T16:35:25.6078137Z d715265d8db7: Verifying Checksum 2024-11-01T16:35:25.6079202Z d715265d8db7: Download complete 2024-11-01T16:35:25.6751427Z 3634fe4cf0c4: Verifying Checksum 2024-11-01T16:35:25.6752263Z 3634fe4cf0c4: Download complete 2024-11-01T16:35:25.7505393Z da04921cbec5: Verifying Checksum 2024-11-01T16:35:25.7506021Z da04921cbec5: Download complete 2024-11-01T16:35:25.9983201Z 55b1e8dc281d: Verifying Checksum 2024-11-01T16:35:25.9984262Z 55b1e8dc281d: Download complete 2024-11-01T16:35:26.0801998Z 595dea37c8f8: Verifying Checksum 2024-11-01T16:35:26.1541677Z 595dea37c8f8: Download complete 2024-11-01T16:35:26.1542531Z 1edfdc5afa7b: Download complete 2024-11-01T16:35:26.2231501Z f64e657da3cf: Verifying Checksum 2024-11-01T16:35:26.2232310Z f64e657da3cf: Download complete 2024-11-01T16:35:26.2888380Z de853ca4b926: Verifying Checksum 2024-11-01T16:35:26.2889085Z de853ca4b926: Download complete 2024-11-01T16:35:26.3534430Z 71d440813d6d: Verifying Checksum 2024-11-01T16:35:26.3535227Z 71d440813d6d: Download complete 2024-11-01T16:35:26.4141604Z cee9a121085f: Verifying Checksum 2024-11-01T16:35:26.4142220Z cee9a121085f: Download complete 2024-11-01T16:35:26.5035443Z f749ccdf4d46: Verifying Checksum 2024-11-01T16:35:26.5036163Z f749ccdf4d46: Download complete 2024-11-01T16:35:28.5040234Z 177321563b8e: Verifying Checksum 2024-11-01T16:35:28.5041254Z 177321563b8e: Download complete 2024-11-01T16:35:28.5769275Z d474c4eb5d81: Verifying Checksum 2024-11-01T16:35:28.5770129Z d474c4eb5d81: Download complete 2024-11-01T16:35:28.6498815Z b30192601766: Verifying Checksum 2024-11-01T16:35:28.6499629Z b30192601766: Download complete 2024-11-01T16:35:28.7195429Z e927cdba5b8a: Verifying Checksum 2024-11-01T16:35:28.7196188Z e927cdba5b8a: Download complete 2024-11-01T16:35:28.7977131Z 859f5d9a55a3: Download complete 2024-11-01T16:35:28.8628701Z e992e1c90303: Verifying Checksum 2024-11-01T16:35:28.8629492Z e992e1c90303: Download complete 2024-11-01T16:35:28.9286057Z ded8f639d8e8: Download complete 2024-11-01T16:35:28.9964261Z 6dc1988800db: Verifying Checksum 2024-11-01T16:35:28.9965021Z 6dc1988800db: Download complete 2024-11-01T16:35:29.0922133Z 6b07caec55f8: Download complete 2024-11-01T16:35:29.1674742Z fe72746ef78d: Verifying Checksum 2024-11-01T16:35:29.1675570Z fe72746ef78d: Download complete 2024-11-01T16:35:29.2277921Z 41244b047648: Verifying Checksum 2024-11-01T16:35:29.2278607Z 41244b047648: Download complete 2024-11-01T16:35:29.3082873Z c5ca3a0d4a11: Verifying Checksum 2024-11-01T16:35:29.3083594Z c5ca3a0d4a11: Download complete 2024-11-01T16:35:29.3683410Z 1f3eab9dfbb4: Download complete 2024-11-01T16:35:29.4345476Z 805d5faba3b6: Download complete 2024-11-01T16:35:29.4998271Z b37c2b2cc48a: Verifying Checksum 2024-11-01T16:35:29.4998906Z b37c2b2cc48a: Download complete 2024-11-01T16:35:31.6365173Z a58aebee5440: Verifying Checksum 2024-11-01T16:35:31.6365915Z a58aebee5440: Download complete 2024-11-01T16:35:31.7131884Z aa5c2183b5a6: Download complete 2024-11-01T16:35:31.7837098Z 0c906cd3d76d: Download complete 2024-11-01T16:35:31.8475633Z 4fad3dce2eb7: Verifying Checksum 2024-11-01T16:35:31.8476300Z 4fad3dce2eb7: Download complete 2024-11-01T16:35:31.9224188Z 3ee82f006622: Verifying Checksum 2024-11-01T16:35:31.9225153Z 3ee82f006622: Download complete 2024-11-01T16:35:31.9981057Z dbf98406efc0: Verifying Checksum 2024-11-01T16:35:31.9981794Z dbf98406efc0: Download complete 2024-11-01T16:35:32.0681907Z 9cb7ede94ead: Verifying Checksum 2024-11-01T16:35:32.0682483Z 9cb7ede94ead: Download complete 2024-11-01T16:35:32.4596940Z be63e049eed3: Pull complete 2024-11-01T16:35:32.5890112Z 7b58778274e4: Pull complete 2024-11-01T16:35:32.7560748Z 5252fbf945ae: Pull complete 2024-11-01T16:35:32.8568773Z 81ba6fd7879d: Verifying Checksum 2024-11-01T16:35:32.8569538Z 81ba6fd7879d: Download complete 2024-11-01T16:35:32.9126923Z f88e6e245d17: Pull complete 2024-11-01T16:35:33.0661014Z d2fc2f6d6c5f: Pull complete 2024-11-01T16:35:33.1958342Z 8db79f4cc8cc: Pull complete 2024-11-01T16:35:55.7652329Z 8f875d71a923: Pull complete 2024-11-01T16:35:55.9753893Z d4816a7d80f8: Pull complete 2024-11-01T16:35:56.1861415Z 2d6bd9e42551: Pull complete 2024-11-01T16:35:56.3774292Z 3750343ed4e2: Pull complete 2024-11-01T16:36:03.5506106Z 96047d905613: Verifying Checksum 2024-11-01T16:36:03.5506789Z 96047d905613: Download complete 2024-11-01T16:36:17.4858470Z 1ffba2c07a78: Verifying Checksum 2024-11-01T16:36:17.4859144Z 1ffba2c07a78: Download complete 2024-11-01T16:37:35.1257079Z 96047d905613: Pull complete 2024-11-01T16:37:35.1477391Z 4f4fb700ef54: Pull complete 2024-11-01T16:37:35.1872673Z b79cecd68009: Pull complete 2024-11-01T16:37:35.2335548Z efe1e385dc9e: Pull complete 2024-11-01T16:37:35.2543108Z adeab13bac13: Pull complete 2024-11-01T16:37:35.3277553Z 24b41e0168fa: Pull complete 2024-11-01T16:37:35.3490747Z 707cc0109119: Pull complete 2024-11-01T16:37:35.3705461Z c5fa30c2a598: Pull complete 2024-11-01T16:37:35.3917201Z 7c311ff0f8ed: Pull complete 2024-11-01T16:37:35.4297834Z 6d067a971ea0: Pull complete 2024-11-01T16:37:35.4520901Z 32a0e0f8d213: Pull complete 2024-11-01T16:37:37.9086308Z 6759707dde23: Pull complete 2024-11-01T16:37:38.0125979Z 42a959f7585e: Pull complete 2024-11-01T16:37:38.0316846Z 103e959619f3: Pull complete 2024-11-01T16:37:38.0526473Z 0d21b1ebe1a4: Pull complete 2024-11-01T16:37:38.0710324Z c2036a0c8e11: Pull complete 2024-11-01T16:37:38.0913043Z f566c3af6db7: Pull complete 2024-11-01T16:37:50.7549828Z 177321563b8e: Pull complete 2024-11-01T16:37:50.7741578Z 088d9767f2d8: Pull complete 2024-11-01T16:37:50.7951968Z 180d2736fbf9: Pull complete 2024-11-01T16:37:50.8135755Z 730a2f77686b: Pull complete 2024-11-01T16:37:50.8314466Z 1af136be95e0: Pull complete 2024-11-01T16:37:50.8689691Z 3eb706825457: Pull complete 2024-11-01T16:37:50.8873348Z 67d83ed868e6: Pull complete 2024-11-01T16:37:50.9069557Z 30d4de160741: Pull complete 2024-11-01T16:37:50.9250971Z d0d44efe8bc3: Pull complete 2024-11-01T16:37:50.9617852Z be6faa013cc8: Pull complete 2024-11-01T16:37:50.9800743Z 72dc36b20429: Pull complete 2024-11-01T16:37:51.0177490Z 058580a8278b: Pull complete 2024-11-01T16:37:51.0362432Z aae135253867: Pull complete 2024-11-01T16:37:52.5488065Z 3b05bde44864: Pull complete 2024-11-01T16:37:52.5669370Z 292e740c6187: Pull complete 2024-11-01T16:37:52.5865353Z d715265d8db7: Pull complete 2024-11-01T16:37:52.6246907Z 3634fe4cf0c4: Pull complete 2024-11-01T16:37:52.6442426Z da04921cbec5: Pull complete 2024-11-01T16:37:52.8591621Z 55b1e8dc281d: Pull complete 2024-11-01T16:37:52.8776591Z 595dea37c8f8: Pull complete 2024-11-01T16:37:52.8973613Z 1edfdc5afa7b: Pull complete 2024-11-01T16:37:52.9373991Z f64e657da3cf: Pull complete 2024-11-01T16:37:52.9554779Z de853ca4b926: Pull complete 2024-11-01T16:37:52.9913717Z 71d440813d6d: Pull complete 2024-11-01T16:37:53.0089478Z cee9a121085f: Pull complete 2024-11-01T16:37:53.0454131Z f749ccdf4d46: Pull complete 2024-11-01T16:39:46.0117308Z 1ffba2c07a78: Pull complete 2024-11-01T16:39:46.2174294Z d474c4eb5d81: Pull complete 2024-11-01T16:39:46.4444520Z b30192601766: Pull complete 2024-11-01T16:39:46.5783670Z e927cdba5b8a: Pull complete 2024-11-01T16:39:46.6637462Z 859f5d9a55a3: Pull complete 2024-11-01T16:39:46.7812151Z e992e1c90303: Pull complete 2024-11-01T16:39:47.1173459Z ded8f639d8e8: Pull complete 2024-11-01T16:39:47.1975715Z 6dc1988800db: Pull complete 2024-11-01T16:39:47.3856374Z 6b07caec55f8: Pull complete 2024-11-01T16:39:47.5252912Z fe72746ef78d: Pull complete 2024-11-01T16:39:47.7115880Z 41244b047648: Pull complete 2024-11-01T16:39:48.0817291Z c5ca3a0d4a11: Pull complete 2024-11-01T16:39:48.1615816Z 1f3eab9dfbb4: Pull complete 2024-11-01T16:39:48.3456571Z 805d5faba3b6: Pull complete 2024-11-01T16:39:48.4331128Z b37c2b2cc48a: Pull complete 2024-11-01T16:39:57.2227312Z a58aebee5440: Pull complete 2024-11-01T16:39:57.4213362Z aa5c2183b5a6: Pull complete 2024-11-01T16:39:57.4777511Z 0c906cd3d76d: Pull complete 2024-11-01T16:39:57.4987323Z 4fad3dce2eb7: Pull complete 2024-11-01T16:39:57.5206787Z 3ee82f006622: Pull complete 2024-11-01T16:39:57.5421341Z dbf98406efc0: Pull complete 2024-11-01T16:39:57.5839136Z 9cb7ede94ead: Pull complete 2024-11-01T16:39:59.0726506Z 81ba6fd7879d: Pull complete 2024-11-01T16:39:59.1039064Z Digest: sha256:68d308b5ab9729d840ef6882af267b00db6e15ae40d6c8de9da61fef478f8a18 2024-11-01T16:39:59.1224374Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:39:59.1226231Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:39:59.1263468Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:39:59.1264871Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:39:59.1272326Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:39:59.1272924Z env: 2024-11-01T16:39:59.1273199Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:39:59.1273544Z ##[endgroup] 2024-11-01T16:39:59.1399463Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-11-01T16:39:59.1400846Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-11-01T16:39:59.1402039Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-11-01T16:39:59.1403229Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:39:59.1412113Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:39:59.1413011Z env: 2024-11-01T16:39:59.1413441Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:39:59.1413988Z ##[endgroup] 2024-11-01T16:39:59.6240251Z Defaulting to user installation because normal site-packages is not writeable 2024-11-01T16:40:00.0728628Z Collecting psutil==5.9.1 2024-11-01T16:40:00.1093168Z Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-11-01T16:40:00.1613138Z Collecting nvidia-ml-py==11.525.84 2024-11-01T16:40:00.1658267Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-11-01T16:40:00.2598815Z Installing collected packages: psutil, nvidia-ml-py 2024-11-01T16:40:00.4270683Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-11-01T16:40:00.6363626Z Prepare all required actions 2024-11-01T16:40:00.6364171Z Getting action download info 2024-11-01T16:40:00.7619765Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-11-01T16:40:00.9531977Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-11-01T16:40:01.0654797Z ##[group]Run ./.github/actions/download-build-artifacts 2024-11-01T16:40:01.0655285Z with: 2024-11-01T16:40:01.0655698Z name: linux-jammy-py3.9-gcc11 2024-11-01T16:40:01.0656079Z s3-bucket: gha-artifacts 2024-11-01T16:40:01.0656414Z env: 2024-11-01T16:40:01.0656679Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:40:01.0657014Z ##[endgroup] 2024-11-01T16:40:01.0694177Z ##[group]Run seemethere/download-artifact-s3@v4 2024-11-01T16:40:01.0694630Z with: 2024-11-01T16:40:01.0695037Z name: linux-jammy-py3.9-gcc11 2024-11-01T16:40:01.0695424Z s3-bucket: gha-artifacts 2024-11-01T16:40:01.0695821Z region: us-east-1 2024-11-01T16:40:01.0696112Z env: 2024-11-01T16:40:01.0696357Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:40:01.0696693Z ##[endgroup] 2024-11-01T16:40:01.5902850Z (node:39793) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-11-01T16:40:01.5903593Z 2024-11-01T16:40:01.5903856Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-11-01T16:40:01.5904851Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-11-01T16:40:01.5906261Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-11-01T16:40:01.6690388Z Found 1 objects with prefix pytorch/pytorch/11632514903/linux-jammy-py3.9-gcc11/ 2024-11-01T16:40:01.6691469Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-11-01T16:40:05.4766630Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-11-01T16:40:05.4773370Z Artifact download has finished successfully 2024-11-01T16:40:05.4936868Z ##[group]Run unzip -o artifacts.zip 2024-11-01T16:40:05.4937306Z unzip -o artifacts.zip 2024-11-01T16:40:05.4943673Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:40:05.4944182Z env: 2024-11-01T16:40:05.4944450Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:40:05.4944792Z ##[endgroup] 2024-11-01T16:40:05.5009323Z Archive: artifacts.zip 2024-11-01T16:40:05.5010260Z creating: dist/ 2024-11-01T16:40:06.4590974Z inflating: dist/torch-2.6.0a0+gitd1aa4ef-cp39-cp39-linux_x86_64.whl 2024-11-01T16:40:06.4591703Z creating: build/custom_test_artifacts/ 2024-11-01T16:40:06.4592321Z creating: build/custom_test_artifacts/custom-op-build/ 2024-11-01T16:40:06.4593135Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-11-01T16:40:06.4594428Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-11-01T16:40:06.4596134Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-11-01T16:40:06.4597141Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-11-01T16:40:06.4598562Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-11-01T16:40:06.4600278Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-11-01T16:40:06.4601329Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-11-01T16:40:06.4603084Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-11-01T16:40:06.4604802Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-11-01T16:40:06.4606396Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-11-01T16:40:06.4608263Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-11-01T16:40:06.4610974Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-11-01T16:40:06.4613394Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-11-01T16:40:06.4614813Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-11-01T16:40:06.4616053Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-11-01T16:40:06.4617301Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-11-01T16:40:06.4618566Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-11-01T16:40:06.4619621Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-11-01T16:40:06.4620756Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-11-01T16:40:06.4621761Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-11-01T16:40:06.4622875Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-11-01T16:40:06.4624114Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-11-01T16:40:06.4625326Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-11-01T16:40:06.4626512Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-11-01T16:40:06.4627678Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-11-01T16:40:06.4628845Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-11-01T16:40:06.4630016Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-11-01T16:40:06.4631177Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-11-01T16:40:06.4632766Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-11-01T16:40:06.4634048Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-11-01T16:40:06.4763555Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-11-01T16:40:06.4765599Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-11-01T16:40:06.4767840Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-11-01T16:40:06.4769418Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-11-01T16:40:06.4770706Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-11-01T16:40:06.4771909Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-11-01T16:40:06.4773138Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-11-01T16:40:06.4774382Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-11-01T16:40:06.4775620Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-11-01T16:40:06.4776849Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-11-01T16:40:06.4778176Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-11-01T16:40:06.4787182Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-11-01T16:40:06.4863591Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-11-01T16:40:06.4865902Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-11-01T16:40:06.4867655Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-11-01T16:40:06.4869364Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-11-01T16:40:06.4870349Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-11-01T16:40:06.4871317Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-11-01T16:40:06.4872242Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-11-01T16:40:06.4873159Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-11-01T16:40:06.4874020Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-11-01T16:40:06.4983364Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-11-01T16:40:06.5043087Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-11-01T16:40:06.5044253Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-11-01T16:40:06.5044987Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-11-01T16:40:06.5045855Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-11-01T16:40:06.5047560Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-11-01T16:40:06.5048843Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-11-01T16:40:06.5050189Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-11-01T16:40:06.5051661Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-11-01T16:40:06.5052935Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-11-01T16:40:06.5054528Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-11-01T16:40:06.5056125Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-11-01T16:40:06.5057845Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-11-01T16:40:06.5059467Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-11-01T16:40:06.5061790Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-11-01T16:40:06.5064081Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-11-01T16:40:06.5066106Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-11-01T16:40:06.5067342Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-11-01T16:40:06.5068601Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-11-01T16:40:06.5069860Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-11-01T16:40:06.5070911Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-11-01T16:40:06.5071877Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-11-01T16:40:06.5072969Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-11-01T16:40:06.5074131Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-11-01T16:40:06.5075421Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-11-01T16:40:06.5076669Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-11-01T16:40:06.5078007Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-11-01T16:40:06.5079219Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-11-01T16:40:06.5080424Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-11-01T16:40:06.5081634Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-11-01T16:40:06.5082837Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-11-01T16:40:06.5084426Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-11-01T16:40:06.5085728Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-11-01T16:40:06.5144394Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-11-01T16:40:06.5146665Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-11-01T16:40:06.5148406Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-11-01T16:40:06.5150066Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-11-01T16:40:06.5151034Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-11-01T16:40:06.5151990Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-11-01T16:40:06.5152958Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-11-01T16:40:06.5153777Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-11-01T16:40:06.5154622Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-11-01T16:40:06.5198315Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-11-01T16:40:06.5199510Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-11-01T16:40:06.5200318Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-11-01T16:40:06.5201287Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-11-01T16:40:06.5203346Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-11-01T16:40:06.5204420Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-11-01T16:40:06.5205926Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-11-01T16:40:06.5207708Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-11-01T16:40:06.5208901Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-11-01T16:40:06.5210912Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-11-01T16:40:06.5212879Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-11-01T16:40:06.5214430Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-11-01T16:40:06.5216519Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-11-01T16:40:06.5219016Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-11-01T16:40:06.5221226Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-11-01T16:40:06.5222520Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-11-01T16:40:06.5223833Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-11-01T16:40:06.5225752Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-11-01T16:40:06.5227304Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-11-01T16:40:06.5228439Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-11-01T16:40:06.5229508Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-11-01T16:40:06.5230603Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-11-01T16:40:06.5231820Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-11-01T16:40:06.5233262Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-11-01T16:40:06.5234605Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-11-01T16:40:06.5235976Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-11-01T16:40:06.5237284Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-11-01T16:40:06.5238584Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-11-01T16:40:06.5239893Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-11-01T16:40:06.5241193Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-11-01T16:40:06.5242466Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-11-01T16:40:06.5243968Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-11-01T16:40:06.5338910Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-11-01T16:40:06.5341122Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-11-01T16:40:06.5343608Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-11-01T16:40:06.5345226Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-11-01T16:40:06.5346615Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-11-01T16:40:06.5347924Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-11-01T16:40:06.5349270Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-11-01T16:40:06.5350796Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-11-01T16:40:06.5352170Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-11-01T16:40:06.5353601Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-11-01T16:40:06.5354944Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-11-01T16:40:06.5363161Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-11-01T16:40:06.5414509Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-11-01T16:40:06.5417108Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-11-01T16:40:06.5418999Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-11-01T16:40:06.5420673Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-11-01T16:40:06.5421737Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-11-01T16:40:06.5422780Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-11-01T16:40:06.5423771Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-11-01T16:40:06.5424752Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-11-01T16:40:06.5425676Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-11-01T16:40:06.5517978Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-11-01T16:40:06.5559506Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-11-01T16:40:06.5560813Z creating: build/lib/ 2024-11-01T16:40:06.5561184Z inflating: build/lib/libclog.a 2024-11-01T16:40:06.5569570Z inflating: build/lib/libpthreadpool.a 2024-11-01T16:40:06.5577427Z inflating: build/lib/libcpuinfo.a 2024-11-01T16:40:06.5584995Z inflating: build/lib/libcpuinfo_internals.a 2024-11-01T16:40:06.5648413Z inflating: build/lib/libgtest.a 2024-11-01T16:40:06.5655733Z inflating: build/lib/libittnotify.a 2024-11-01T16:40:06.5735296Z inflating: build/lib/libprotobuf-lite.a 2024-11-01T16:40:06.5805807Z inflating: build/lib/libbenchmark.a 2024-11-01T16:40:06.5832544Z inflating: build/lib/libtensorpipe_uv.a 2024-11-01T16:40:06.5894347Z inflating: build/lib/libasmjit.a 2024-11-01T16:40:06.6321543Z inflating: build/lib/libprotobuf.a 2024-11-01T16:40:06.6430207Z inflating: build/lib/libgloo.a 2024-11-01T16:40:06.6449958Z inflating: build/lib/libfmt.a 2024-11-01T16:40:06.6451233Z inflating: build/lib/libtorch_global_deps.so 2024-11-01T16:40:06.6548844Z inflating: build/lib/libc10.so 2024-11-01T16:40:06.6551079Z inflating: build/lib/libnnpack_reference_layers.a 2024-11-01T16:40:06.6569733Z inflating: build/lib/libpytorch_qnnpack.a 2024-11-01T16:40:06.6586426Z inflating: build/lib/libgmock.a 2024-11-01T16:40:06.6587236Z inflating: build/lib/libgtest_main.a 2024-11-01T16:40:06.6588129Z inflating: build/lib/libbenchmark_main.a 2024-11-01T16:40:06.7066909Z inflating: build/lib/libprotoc.a 2024-11-01T16:40:06.7084144Z inflating: build/lib/libnnpack.a 2024-11-01T16:40:06.7611628Z inflating: build/lib/libtensorpipe.a 2024-11-01T16:40:06.7612461Z inflating: build/lib/libgmock_main.a 2024-11-01T16:40:06.8603080Z inflating: build/lib/libfbgemm.a 2024-11-01T16:40:07.8088811Z inflating: build/lib/libdnnl.a 2024-11-01T16:40:07.8365896Z inflating: build/lib/libkineto.a 2024-11-01T16:40:07.8405010Z inflating: build/lib/libonnx_proto.a 2024-11-01T16:40:07.8596945Z inflating: build/lib/libXNNPACK.a 2024-11-01T16:40:07.9294289Z inflating: build/lib/libonnx.a 2024-11-01T16:40:10.2460560Z inflating: build/lib/libtorch_cpu.so 2024-11-01T16:40:10.2461448Z inflating: build/lib/libtorch.so 2024-11-01T16:40:10.2465701Z inflating: build/lib/libunbox_lib.a 2024-11-01T16:40:10.2469424Z inflating: build/lib/libshm.so 2024-11-01T16:40:10.2487757Z inflating: build/lib/libjitbackend_test.so 2024-11-01T16:40:10.2554625Z inflating: build/lib/libtorchbind_test.so 2024-11-01T16:40:10.2577723Z inflating: build/lib/libaoti_custom_ops.so 2024-11-01T16:40:10.2601524Z inflating: build/lib/libbackend_with_compiler.so 2024-11-01T16:40:10.4458691Z inflating: build/lib/libtorch_python.so 2024-11-01T16:40:10.4491633Z inflating: build/lib/libnnapi_backend.so 2024-11-01T16:40:10.4492487Z creating: build/bin/ 2024-11-01T16:40:10.4493149Z creating: build/bin/CMakeFiles/ 2024-11-01T16:40:10.4494216Z inflating: build/bin/CMakeFiles/CMakeDirectoryInformation.cmake 2024-11-01T16:40:10.4495445Z extracting: build/bin/CMakeFiles/progress.marks 2024-11-01T16:40:10.4496249Z inflating: build/bin/Makefile 2024-11-01T16:40:10.4496718Z inflating: build/bin/cmake_install.cmake 2024-11-01T16:40:10.4497217Z inflating: build/bin/CTestTestfile.cmake 2024-11-01T16:40:10.4546427Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-11-01T16:40:10.4599110Z inflating: build/bin/c10_DeviceGuard_test 2024-11-01T16:40:10.4652206Z inflating: build/bin/c10_Device_test 2024-11-01T16:40:10.4713121Z inflating: build/bin/c10_DispatchKeySet_test 2024-11-01T16:40:10.4767847Z inflating: build/bin/c10_Scalar_test 2024-11-01T16:40:10.4817999Z inflating: build/bin/c10_StreamGuard_test 2024-11-01T16:40:10.4870280Z inflating: build/bin/c10_SymInt_test 2024-11-01T16:40:10.4925811Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-11-01T16:40:10.4982490Z inflating: build/bin/c10_InlineStreamGuard_test 2024-11-01T16:40:10.5039317Z inflating: build/bin/c10_SizesAndStrides_test 2024-11-01T16:40:10.5111197Z inflating: build/bin/c10_cow_test 2024-11-01T16:40:10.5165682Z inflating: build/bin/c10_Bitset_test 2024-11-01T16:40:10.5215590Z inflating: build/bin/c10_ConstexprCrc_test 2024-11-01T16:40:10.5266725Z inflating: build/bin/c10_DeadlockDetection_test 2024-11-01T16:40:10.5323363Z inflating: build/bin/c10_Metaprogramming_test 2024-11-01T16:40:10.5375730Z inflating: build/bin/c10_Half_test 2024-11-01T16:40:10.5432541Z inflating: build/bin/c10_LeftRight_test 2024-11-01T16:40:10.5487303Z inflating: build/bin/c10_NetworkFlow_test 2024-11-01T16:40:10.5538389Z inflating: build/bin/c10_Synchronized_test 2024-11-01T16:40:10.5594855Z inflating: build/bin/c10_ThreadLocal_test 2024-11-01T16:40:10.5648092Z inflating: build/bin/c10_TypeIndex_test 2024-11-01T16:40:10.5700635Z inflating: build/bin/c10_TypeList_test 2024-11-01T16:40:10.5751007Z inflating: build/bin/c10_TypeTraits_test 2024-11-01T16:40:10.5804360Z inflating: build/bin/c10_accumulate_test 2024-11-01T16:40:10.5860661Z inflating: build/bin/c10_bfloat16_test 2024-11-01T16:40:10.5912117Z inflating: build/bin/c10_bit_cast_test 2024-11-01T16:40:10.5969846Z inflating: build/bin/c10_complex_math_test 2024-11-01T16:40:10.6025910Z inflating: build/bin/c10_complex_test 2024-11-01T16:40:10.6079826Z inflating: build/bin/c10_exception_test 2024-11-01T16:40:10.6131327Z inflating: build/bin/c10_flags_test 2024-11-01T16:40:10.6182857Z inflating: build/bin/c10_generic_math_test 2024-11-01T16:40:10.6343682Z inflating: build/bin/c10_intrusive_ptr_test 2024-11-01T16:40:10.6395870Z inflating: build/bin/c10_irange_test 2024-11-01T16:40:10.6450477Z inflating: build/bin/c10_lazy_test 2024-11-01T16:40:10.6508951Z inflating: build/bin/c10_logging_test 2024-11-01T16:40:10.6584806Z inflating: build/bin/c10_optional_test 2024-11-01T16:40:10.6639460Z inflating: build/bin/c10_registry_test 2024-11-01T16:40:10.6702739Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-11-01T16:40:10.6852588Z inflating: build/bin/c10_small_vector_test 2024-11-01T16:40:10.6907049Z inflating: build/bin/c10_string_util_test 2024-11-01T16:40:10.6960259Z inflating: build/bin/c10_ssize_test 2024-11-01T16:40:10.7012612Z inflating: build/bin/c10_tempfile_test 2024-11-01T16:40:10.7073049Z inflating: build/bin/c10_string_view_test 2024-11-01T16:40:10.7130418Z inflating: build/bin/c10_typeid_test 2024-11-01T16:40:10.7178678Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-11-01T16:40:10.7628124Z inflating: build/bin/protoc-3.13.0.0 2024-11-01T16:40:10.8077611Z inflating: build/bin/protoc 2024-11-01T16:40:10.8448502Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-11-01T16:40:10.8834504Z inflating: build/bin/vec_test_all_types_AVX512 2024-11-01T16:40:10.9231732Z inflating: build/bin/vec_test_all_types_AVX2 2024-11-01T16:40:10.9285994Z inflating: build/bin/FileStoreTest 2024-11-01T16:40:10.9339757Z inflating: build/bin/BackoffTest 2024-11-01T16:40:10.9397391Z inflating: build/bin/TCPStoreTest 2024-11-01T16:40:10.9453815Z inflating: build/bin/HashStoreTest 2024-11-01T16:40:10.9455861Z inflating: build/bin/example_allreduce 2024-11-01T16:40:10.9522669Z inflating: build/bin/ProcessGroupGlooTest 2024-11-01T16:40:10.9579027Z inflating: build/bin/test_dist_autograd 2024-11-01T16:40:10.9648766Z inflating: build/bin/test_cpp_rpc 2024-11-01T16:40:10.9651005Z inflating: build/bin/parallel_benchmark 2024-11-01T16:40:10.9659889Z inflating: build/bin/aot_model_compiler_test 2024-11-01T16:40:10.9727214Z inflating: build/bin/test_mobile_nnc 2024-11-01T16:40:11.0062644Z inflating: build/bin/test_lazy 2024-11-01T16:40:11.0121236Z inflating: build/bin/static_runtime_bench 2024-11-01T16:40:11.0175216Z inflating: build/bin/Dimname_test 2024-11-01T16:40:11.0250150Z inflating: build/bin/Dict_test 2024-11-01T16:40:11.0316575Z inflating: build/bin/MaybeOwned_test 2024-11-01T16:40:11.0564519Z inflating: build/bin/static_runtime_test 2024-11-01T16:40:11.0622501Z inflating: build/bin/NamedTensor_test 2024-11-01T16:40:11.0682725Z inflating: build/bin/apply_utils_test 2024-11-01T16:40:11.0747264Z inflating: build/bin/basic 2024-11-01T16:40:11.0807447Z inflating: build/bin/atest 2024-11-01T16:40:11.0863736Z inflating: build/bin/broadcast_test 2024-11-01T16:40:11.0915993Z inflating: build/bin/cpu_allocator_test 2024-11-01T16:40:11.0975850Z inflating: build/bin/cpu_generator_test 2024-11-01T16:40:11.1030400Z inflating: build/bin/cpu_profiling_allocator_test 2024-11-01T16:40:11.1081955Z inflating: build/bin/dispatch_key_set_test 2024-11-01T16:40:11.1174830Z inflating: build/bin/cpu_rng_test 2024-11-01T16:40:11.1226510Z inflating: build/bin/dlconvertor_test 2024-11-01T16:40:11.2332777Z inflating: build/bin/test_api 2024-11-01T16:40:11.2392152Z inflating: build/bin/extension_backend_test 2024-11-01T16:40:11.2448418Z inflating: build/bin/half_test 2024-11-01T16:40:11.2499626Z inflating: build/bin/lazy_tensor_test 2024-11-01T16:40:11.2594965Z inflating: build/bin/ivalue_test 2024-11-01T16:40:11.2649501Z inflating: build/bin/math_kernel_test 2024-11-01T16:40:11.2704482Z inflating: build/bin/memory_format_test 2024-11-01T16:40:11.2758830Z inflating: build/bin/memory_overlapping_test 2024-11-01T16:40:11.2810832Z inflating: build/bin/operator_name_test 2024-11-01T16:40:11.2868281Z inflating: build/bin/native_test 2024-11-01T16:40:11.2922613Z inflating: build/bin/mobile_memory_cleanup 2024-11-01T16:40:11.2976747Z inflating: build/bin/packedtensoraccessor_test 2024-11-01T16:40:11.3028666Z inflating: build/bin/operators_test 2024-11-01T16:40:11.3096322Z inflating: build/bin/pow_test 2024-11-01T16:40:11.3154755Z inflating: build/bin/quantized_test 2024-11-01T16:40:11.3206035Z inflating: build/bin/reduce_ops_test 2024-11-01T16:40:11.3258779Z inflating: build/bin/reportMemoryUsage_test 2024-11-01T16:40:11.3316882Z inflating: build/bin/scalar_tensor_test 2024-11-01T16:40:11.3369695Z inflating: build/bin/StorageUtils_test 2024-11-01T16:40:11.3428907Z inflating: build/bin/scalar_test 2024-11-01T16:40:11.3482769Z inflating: build/bin/stride_properties_test 2024-11-01T16:40:11.3539006Z inflating: build/bin/type_ptr_test 2024-11-01T16:40:11.3618384Z inflating: build/bin/tensor_iterator_test 2024-11-01T16:40:11.3621055Z inflating: build/bin/thread_init_test 2024-11-01T16:40:11.3677002Z inflating: build/bin/test_parallel 2024-11-01T16:40:11.3737354Z inflating: build/bin/type_test 2024-11-01T16:40:11.3738808Z inflating: build/bin/verify_api_visibility 2024-11-01T16:40:11.3792931Z inflating: build/bin/undefined_tensor_test 2024-11-01T16:40:11.3863196Z inflating: build/bin/legacy_vmap_test 2024-11-01T16:40:11.3915770Z inflating: build/bin/weakref_test 2024-11-01T16:40:11.3969001Z inflating: build/bin/wrapdim_test 2024-11-01T16:40:11.4029467Z inflating: build/bin/IListRef_test 2024-11-01T16:40:11.4083164Z inflating: build/bin/xla_tensor_test 2024-11-01T16:40:11.4188235Z inflating: build/bin/List_test 2024-11-01T16:40:11.4309015Z inflating: build/bin/kernel_function_legacy_test 2024-11-01T16:40:11.4404945Z inflating: build/bin/kernel_function_test 2024-11-01T16:40:11.4471999Z inflating: build/bin/KernelFunction_test 2024-11-01T16:40:11.4598166Z inflating: build/bin/kernel_lambda_legacy_test 2024-11-01T16:40:11.4699825Z inflating: build/bin/kernel_lambda_test 2024-11-01T16:40:11.4761662Z inflating: build/bin/kernel_stackbased_test 2024-11-01T16:40:11.4857065Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-11-01T16:40:11.4909397Z inflating: build/bin/CppSignature_test 2024-11-01T16:40:11.4959677Z inflating: build/bin/op_allowlist_test 2024-11-01T16:40:11.5265266Z inflating: build/bin/op_registration_test 2024-11-01T16:40:11.5321659Z inflating: build/bin/backend_fallback_test 2024-11-01T16:40:11.5386582Z inflating: build/bin/inline_container_test 2024-11-01T16:40:11.5441065Z inflating: build/bin/test_edge_op_registration 2024-11-01T16:40:11.5445153Z inflating: build/bin/torch_shm_manager 2024-11-01T16:40:11.5457890Z inflating: build/bin/tutorial_tensorexpr 2024-11-01T16:40:11.6235041Z inflating: build/bin/test_tensorexpr 2024-11-01T16:40:11.6799819Z inflating: build/bin/test_jit 2024-11-01T16:40:11.6800552Z creating: .additional_ci_files/ 2024-11-01T16:40:11.6875500Z inflating: .additional_ci_files/test-times.json 2024-11-01T16:40:11.7175566Z inflating: .additional_ci_files/test-class-times.json 2024-11-01T16:40:11.7219581Z ##[group]Run rm artifacts.zip 2024-11-01T16:40:11.7219969Z rm artifacts.zip 2024-11-01T16:40:11.7226302Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:40:11.7226811Z env: 2024-11-01T16:40:11.7227080Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:40:11.7227435Z ##[endgroup] 2024-11-01T16:40:12.0107970Z ##[group]Run df -H 2024-11-01T16:40:12.0108285Z df -H 2024-11-01T16:40:12.0114387Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:40:12.0114889Z env: 2024-11-01T16:40:12.0115141Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:40:12.0115481Z ##[endgroup] 2024-11-01T16:40:12.0169085Z Filesystem Size Used Avail Use% Mounted on 2024-11-01T16:40:12.0169933Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-11-01T16:40:12.0170420Z tmpfs 8.2G 0 8.2G 0% /dev/shm 2024-11-01T16:40:12.0170873Z tmpfs 3.3G 488k 3.3G 1% /run 2024-11-01T16:40:12.0171466Z /dev/nvme0n1p1 161G 33G 129G 21% / 2024-11-01T16:40:12.0172005Z tmpfs 8.2G 4.1k 8.2G 1% /tmp 2024-11-01T16:40:12.0172487Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-11-01T16:40:12.0222872Z Prepare all required actions 2024-11-01T16:40:12.0223302Z Getting action download info 2024-11-01T16:40:12.2053864Z ##[group]Run ./.github/actions/download-td-artifacts 2024-11-01T16:40:12.2054687Z with: 2024-11-01T16:40:12.2054917Z env: 2024-11-01T16:40:12.2055180Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:40:12.2055522Z ##[endgroup] 2024-11-01T16:40:12.2222889Z ##[group]Run seemethere/download-artifact-s3@v4 2024-11-01T16:40:12.2223345Z with: 2024-11-01T16:40:12.2223606Z name: td_results 2024-11-01T16:40:12.2223918Z s3-bucket: gha-artifacts 2024-11-01T16:40:12.2224250Z region: us-east-1 2024-11-01T16:40:12.2224537Z env: 2024-11-01T16:40:12.2224798Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:40:12.2225135Z ##[endgroup] 2024-11-01T16:40:12.7364366Z (node:39816) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-11-01T16:40:12.7365122Z 2024-11-01T16:40:12.7365462Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-11-01T16:40:12.7366212Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-11-01T16:40:12.7367098Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-11-01T16:40:12.8055425Z Found 1 objects with prefix pytorch/pytorch/11632514903/td_results/ 2024-11-01T16:40:12.8056662Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-11-01T16:40:12.8653157Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-11-01T16:40:12.8659638Z Artifact download has finished successfully 2024-11-01T16:40:12.8808477Z ##[group]Run mkdir -p .additional_ci_files 2024-11-01T16:40:12.8808962Z mkdir -p .additional_ci_files 2024-11-01T16:40:12.8809537Z mv td_results.json .additional_ci_files/td_results.json 2024-11-01T16:40:12.8816299Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:40:12.8816808Z env: 2024-11-01T16:40:12.8817094Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:40:12.8817424Z ##[endgroup] 2024-11-01T16:40:12.8918368Z ##[group]Run .github/scripts/parse_ref.py 2024-11-01T16:40:12.8918980Z .github/scripts/parse_ref.py 2024-11-01T16:40:12.8924398Z shell: /usr/bin/bash -e {0} 2024-11-01T16:40:12.8924745Z env: 2024-11-01T16:40:12.8925001Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:40:12.8925340Z ##[endgroup] 2024-11-01T16:40:12.9221686Z Prepare all required actions 2024-11-01T16:40:12.9264285Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-11-01T16:40:12.9264747Z with: 2024-11-01T16:40:12.9265322Z github-token: *** 2024-11-01T16:40:12.9265732Z env: 2024-11-01T16:40:12.9265996Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:40:12.9266323Z ##[endgroup] 2024-11-01T16:40:12.9284741Z ##[group]Run set -eux 2024-11-01T16:40:12.9285064Z set -eux 2024-11-01T16:40:12.9285666Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-11-01T16:40:12.9291934Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:40:12.9292443Z env: 2024-11-01T16:40:12.9292709Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:40:12.9293237Z GITHUB_TOKEN: *** 2024-11-01T16:40:12.9293541Z ##[endgroup] 2024-11-01T16:40:12.9316588Z + python3 .github/scripts/get_workflow_job_id.py 11632514903 i-07c6a72643395af71 2024-11-01T16:40:14.8299367Z setting job-id=32396543718 2024-11-01T16:40:14.8300275Z setting job-name=linux-jammy-py3.9-gcc11 / test (distributed, 1, 2, linux.2xlarge) 2024-11-01T16:40:14.8556053Z Prepare all required actions 2024-11-01T16:40:14.8556602Z Getting action download info 2024-11-01T16:40:14.9974793Z ##[group]Run ./.github/actions/filter-test-configs 2024-11-01T16:40:14.9975261Z with: 2024-11-01T16:40:14.9975721Z github-token: *** 2024-11-01T16:40:14.9979461Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "default", "shard": 2, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "default", "shard": 3, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "default", "shard": 4, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "default", "shard": 5, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "docs_test", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "jit_legacy", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "backwards_compat", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge"}]} 2024-11-01T16:40:14.9983381Z job-name: linux-jammy-py3.9-gcc11 / test (distributed, 1, 2, linux.2xlarge) 2024-11-01T16:40:14.9983996Z env: 2024-11-01T16:40:14.9984264Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:40:14.9984587Z ##[endgroup] 2024-11-01T16:40:15.0050811Z ##[group]Run nick-fields/retry@v3.0.0 2024-11-01T16:40:15.0051211Z with: 2024-11-01T16:40:15.0051471Z shell: bash 2024-11-01T16:40:15.0051760Z timeout_minutes: 10 2024-11-01T16:40:15.0052081Z max_attempts: 5 2024-11-01T16:40:15.0052392Z retry_wait_seconds: 30 2024-11-01T16:40:15.0053583Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-11-01T16:40:15.0054818Z polling_interval_seconds: 1 2024-11-01T16:40:15.0055184Z warning_on_retry: true 2024-11-01T16:40:15.0055533Z continue_on_error: false 2024-11-01T16:40:15.0055862Z env: 2024-11-01T16:40:15.0056121Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:40:15.0056626Z GITHUB_TOKEN: *** 2024-11-01T16:40:15.0056919Z ##[endgroup] 2024-11-01T16:40:16.5695061Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-11-01T16:40:16.8254993Z Defaulting to user installation because normal site-packages is not writeable 2024-11-01T16:40:17.0226459Z Collecting requests==2.27.1 2024-11-01T16:40:17.0585148Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-11-01T16:40:17.3098789Z Collecting pyyaml==6.0.1 2024-11-01T16:40:17.3142374Z Downloading PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB) 2024-11-01T16:40:17.3941178Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-11-01T16:40:17.7603326Z Collecting charset-normalizer~=2.0.0 2024-11-01T16:40:17.7646187Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-11-01T16:40:17.8710196Z Collecting certifi>=2017.4.17 2024-11-01T16:40:17.8750616Z Downloading certifi-2024.8.30-py3-none-any.whl (167 kB) 2024-11-01T16:40:17.9099431Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-11-01T16:40:17.9982706Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2024-11-01T16:40:18.2889050Z Successfully installed certifi-2024.8.30 charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 2024-11-01T16:40:18.4725525Z Command completed after 1 attempt(s). 2024-11-01T16:40:18.4917770Z ##[group]Run set -x 2024-11-01T16:40:18.4918090Z set -x 2024-11-01T16:40:18.4918393Z  2024-11-01T16:40:18.4918944Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-11-01T16:40:18.4919639Z # in runner workspace 2024-11-01T16:40:18.4920172Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-11-01T16:40:18.4926970Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:40:18.4927464Z env: 2024-11-01T16:40:18.4927737Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:40:18.4928078Z ##[endgroup] 2024-11-01T16:40:18.4952196Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-11-01T16:40:18.5243519Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-11-01T16:40:18.5244059Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-11-01T16:40:18.5244527Z echo "Job name: ${JOB_NAME}" 2024-11-01T16:40:18.5244922Z  2024-11-01T16:40:18.5245479Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-11-01T16:40:18.5246193Z # in runner workspace 2024-11-01T16:40:18.5246779Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-11-01T16:40:18.5247425Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-11-01T16:40:18.5247897Z  --job-name "${JOB_NAME}" \ 2024-11-01T16:40:18.5251635Z  --test-matrix "{"include": [{"config": "default", "shard": 1, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "default", "shard": 2, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "default", "shard": 3, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "default", "shard": 4, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "default", "shard": 5, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "docs_test", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "jit_legacy", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "backwards_compat", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge"}]}" \ 2024-11-01T16:40:18.5255615Z  --selected-test-configs "" \ 2024-11-01T16:40:18.5256088Z  --pr-number "${PR_NUMBER}" \ 2024-11-01T16:40:18.5256493Z  --tag "${TAG}" \ 2024-11-01T16:40:18.5256877Z  --event-name "${EVENT_NAME}" \ 2024-11-01T16:40:18.5257321Z  --schedule "${SCHEDULE}" \ 2024-11-01T16:40:18.5257744Z  --branch "${HEAD_BRANCH}" 2024-11-01T16:40:18.5263690Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:40:18.5264198Z env: 2024-11-01T16:40:18.5264450Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:40:18.5264985Z GITHUB_TOKEN: *** 2024-11-01T16:40:18.5265519Z JOB_NAME: linux-jammy-py3.9-gcc11 / test (distributed, 1, 2, linux.2xlarge) 2024-11-01T16:40:18.5266147Z PR_NUMBER: 138766 2024-11-01T16:40:18.5266574Z TAG: 2024-11-01T16:40:18.5266841Z EVENT_NAME: pull_request 2024-11-01T16:40:18.5267167Z SCHEDULE: 2024-11-01T16:40:18.5267442Z HEAD_BRANCH: 2024-11-01T16:40:18.5267730Z ##[endgroup] 2024-11-01T16:40:18.5291066Z Workflow: pull 2024-11-01T16:40:18.5292075Z Job name: linux-jammy-py3.9-gcc11 / test (distributed, 1, 2, linux.2xlarge) 2024-11-01T16:40:18.7555288Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-11-01T16:40:19.5413173Z ##[group]Run echo "Filtered matrix:" 2024-11-01T16:40:19.5413634Z echo "Filtered matrix:" 2024-11-01T16:40:19.5417497Z echo "{"include": [{"config": "default", "shard": 1, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "default", "shard": 2, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "default", "shard": 3, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "default", "shard": 4, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "default", "shard": 5, "num_shards": 5, "runner": "linux.2xlarge"}, {"config": "docs_test", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "jit_legacy", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "backwards_compat", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge"}]}" 2024-11-01T16:40:19.5421199Z  2024-11-01T16:40:19.5421463Z echo 2024-11-01T16:40:19.5421835Z echo "Is the current job unstable? False" 2024-11-01T16:40:19.5422301Z  2024-11-01T16:40:19.5422563Z echo 2024-11-01T16:40:19.5422903Z echo "Is keep-going label set? False" 2024-11-01T16:40:19.5423341Z  2024-11-01T16:40:19.5423591Z echo 2024-11-01T16:40:19.5423894Z echo "Renabled issues? " 2024-11-01T16:40:19.5429994Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:40:19.5430506Z env: 2024-11-01T16:40:19.5430756Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:40:19.5431096Z ##[endgroup] 2024-11-01T16:40:19.5457384Z Filtered matrix: 2024-11-01T16:40:19.5464219Z {include: [{config: default, shard: 1, num_shards: 5, runner: linux.2xlarge}, {config: default, shard: 2, num_shards: 5, runner: linux.2xlarge}, {config: default, shard: 3, num_shards: 5, runner: linux.2xlarge}, {config: default, shard: 4, num_shards: 5, runner: linux.2xlarge}, {config: default, shard: 5, num_shards: 5, runner: linux.2xlarge}, {config: docs_test, shard: 1, num_shards: 1, runner: linux.2xlarge}, {config: jit_legacy, shard: 1, num_shards: 1, runner: linux.2xlarge}, {config: backwards_compat, shard: 1, num_shards: 1, runner: linux.2xlarge}, {config: distributed, shard: 1, num_shards: 2, runner: linux.2xlarge}, {config: distributed, shard: 2, num_shards: 2, runner: linux.2xlarge}]} 2024-11-01T16:40:19.5470665Z 2024-11-01T16:40:19.5470936Z Is the current job unstable? False 2024-11-01T16:40:19.5471410Z 2024-11-01T16:40:19.5471884Z Is keep-going label set? False 2024-11-01T16:40:19.5472320Z 2024-11-01T16:40:19.5472522Z Renabled issues? 2024-11-01T16:40:19.5549656Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-11-01T16:40:19.5550388Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-11-01T16:40:19.5556287Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:40:19.5556792Z env: 2024-11-01T16:40:19.5557060Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:40:19.5557393Z JOB_TIMEOUT: 240 2024-11-01T16:40:19.5557689Z ##[endgroup] 2024-11-01T16:40:19.5921825Z ##[group]Run # Fetch aws credential from IMDs 2024-11-01T16:40:19.5922409Z # Fetch aws credential from IMDs 2024-11-01T16:40:19.5923014Z eval "$(python3 .github/scripts/get_aws_session_tokens.py)" 2024-11-01T16:40:19.5923569Z set -x 2024-11-01T16:40:19.5923839Z  2024-11-01T16:40:19.5924178Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-11-01T16:40:19.5924855Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-11-01T16:40:19.5925422Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-11-01T16:40:19.5925936Z  TEST_COMMAND=.ci/onnx/test.sh 2024-11-01T16:40:19.5926353Z else 2024-11-01T16:40:19.5926671Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-11-01T16:40:19.5927093Z fi 2024-11-01T16:40:19.5927353Z  2024-11-01T16:40:19.5927821Z # detached container should get cleaned up by teardown_ec2_linux 2024-11-01T16:40:19.5928650Z # TODO: Stop building test binaries as part of the build phase 2024-11-01T16:40:19.5929350Z # Used for GPU_FLAG since that doesn't play nice 2024-11-01T16:40:19.5929905Z # shellcheck disable=SC2086,SC2090 2024-11-01T16:40:19.5930357Z container_name=$(docker run \ 2024-11-01T16:40:19.5930782Z  ${GPU_FLAG:-} \ 2024-11-01T16:40:19.5931200Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2024-11-01T16:40:19.5931686Z  -e BUILD_ENVIRONMENT \ 2024-11-01T16:40:19.5932083Z  -e PR_NUMBER \ 2024-11-01T16:40:19.5932432Z  -e GITHUB_ACTIONS \ 2024-11-01T16:40:19.5932822Z  -e GITHUB_REPOSITORY \ 2024-11-01T16:40:19.5933230Z  -e GITHUB_WORKFLOW \ 2024-11-01T16:40:19.5933613Z  -e GITHUB_JOB \ 2024-11-01T16:40:19.5933974Z  -e GITHUB_RUN_ID \ 2024-11-01T16:40:19.5934354Z  -e GITHUB_RUN_NUMBER \ 2024-11-01T16:40:19.5934746Z  -e GITHUB_RUN_ATTEMPT \ 2024-11-01T16:40:19.5935148Z  -e JOB_ID \ 2024-11-01T16:40:19.5935480Z  -e JOB_NAME \ 2024-11-01T16:40:19.5935822Z  -e BASE_SHA \ 2024-11-01T16:40:19.5936159Z  -e BRANCH \ 2024-11-01T16:40:19.5936473Z  -e SHA1 \ 2024-11-01T16:40:19.5936808Z  -e AWS_DEFAULT_REGION \ 2024-11-01T16:40:19.5937213Z  -e AWS_ACCESS_KEY_ID \ 2024-11-01T16:40:19.5937622Z  -e AWS_SECRET_ACCESS_KEY \ 2024-11-01T16:40:19.5938041Z  -e AWS_SESSION_TOKEN \ 2024-11-01T16:40:19.5938420Z  -e IN_WHEEL_TEST \ 2024-11-01T16:40:19.5938791Z  -e SHARD_NUMBER \ 2024-11-01T16:40:19.5939153Z  -e TEST_CONFIG \ 2024-11-01T16:40:19.5939515Z  -e NUM_TEST_SHARDS \ 2024-11-01T16:40:19.5939900Z  -e REENABLED_ISSUES \ 2024-11-01T16:40:19.5940306Z  -e CONTINUE_THROUGH_ERROR \ 2024-11-01T16:40:19.5940715Z  -e VERBOSE_TEST_LOGS \ 2024-11-01T16:40:19.5941108Z  -e TEST_SHOWLOCALS \ 2024-11-01T16:40:19.5941498Z  -e NO_TEST_TIMEOUT \ 2024-11-01T16:40:19.5941866Z  -e NO_TD \ 2024-11-01T16:40:19.5942199Z  -e TD_DISTRIBUTED \ 2024-11-01T16:40:19.5942562Z  -e PR_LABELS \ 2024-11-01T16:40:19.5942962Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-11-01T16:40:19.5943416Z  -e SCCACHE_BUCKET \ 2024-11-01T16:40:19.5943792Z  -e SCCACHE_REGION \ 2024-11-01T16:40:19.5944188Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-11-01T16:40:19.5944593Z  -e XLA_CUDA \ 2024-11-01T16:40:19.5944977Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-11-01T16:40:19.5945489Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-11-01T16:40:19.5946015Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-11-01T16:40:19.5946535Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-11-01T16:40:19.5947007Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-11-01T16:40:19.5947460Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-11-01T16:40:19.5948001Z  -e DASHBOARD_TAG \ 2024-11-01T16:40:19.5948388Z  -e IS_A100_RUNNER \ 2024-11-01T16:40:19.5948783Z  -e ARTIFACTS_FILE_SUFFIX \ 2024-11-01T16:40:19.5949300Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-11-01T16:40:19.5949867Z  --security-opt seccomp=unconfined \ 2024-11-01T16:40:19.5950324Z  --cap-add=SYS_PTRACE \ 2024-11-01T16:40:19.5950781Z  --ipc=host \ 2024-11-01T16:40:19.5951129Z  --shm-size="${SHM_SIZE}" \ 2024-11-01T16:40:19.5951520Z  --tty \ 2024-11-01T16:40:19.5951826Z  --detach \ 2024-11-01T16:40:19.5952184Z  --name="${container_name}" \ 2024-11-01T16:40:19.5952591Z  --user jenkins \ 2024-11-01T16:40:19.5953181Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-11-01T16:40:19.5953757Z  -w /var/lib/jenkins/workspace \ 2024-11-01T16:40:19.5954198Z  "${DOCKER_IMAGE}" 2024-11-01T16:40:19.5954549Z ) 2024-11-01T16:40:19.5954946Z # Propagate download.pytorch.org IP to container 2024-11-01T16:40:19.5955912Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-11-01T16:40:19.5956931Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-11-01T16:40:19.5957890Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-11-01T16:40:19.5964047Z shell: /usr/bin/bash -e {0} 2024-11-01T16:40:19.5964402Z env: 2024-11-01T16:40:19.5964669Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:40:19.5965079Z BUILD_ENVIRONMENT: linux-jammy-py3.9-gcc11 2024-11-01T16:40:19.5965523Z PR_NUMBER: 138766 2024-11-01T16:40:19.5965855Z GITHUB_REPOSITORY: pytorch/pytorch 2024-11-01T16:40:19.5966252Z GITHUB_WORKFLOW: pull 2024-11-01T16:40:19.5966578Z GITHUB_JOB: test 2024-11-01T16:40:19.5966881Z GITHUB_RUN_ID: 11632514903 2024-11-01T16:40:19.5967243Z GITHUB_RUN_NUMBER: 261361 2024-11-01T16:40:19.5967591Z GITHUB_RUN_ATTEMPT: 1 2024-11-01T16:40:19.5967898Z JOB_ID: 32396543718 2024-11-01T16:40:19.5968430Z JOB_NAME: linux-jammy-py3.9-gcc11 / test (distributed, 1, 2, linux.2xlarge) 2024-11-01T16:40:19.5969049Z BRANCH: pull/138766 2024-11-01T16:40:19.5969430Z SHA1: d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:40:19.5969975Z BASE_SHA: 2055d3c8dc8b93c52eda6b9c07d716448d9e28c9 2024-11-01T16:40:19.5970449Z TEST_CONFIG: distributed 2024-11-01T16:40:19.5970777Z SHARD_NUMBER: 1 2024-11-01T16:40:19.5971071Z NUM_TEST_SHARDS: 2 2024-11-01T16:40:19.5971384Z REENABLED_ISSUES: 2024-11-01T16:40:19.5971710Z CONTINUE_THROUGH_ERROR: False 2024-11-01T16:40:19.5972088Z VERBOSE_TEST_LOGS: False 2024-11-01T16:40:19.5972425Z TEST_SHOWLOCALS: False 2024-11-01T16:40:19.5972759Z NO_TEST_TIMEOUT: False 2024-11-01T16:40:19.5973081Z NO_TD: False 2024-11-01T16:40:19.5973367Z TD_DISTRIBUTED: False 2024-11-01T16:40:19.5973791Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-11-01T16:40:19.5974264Z SCCACHE_REGION: us-east-1 2024-11-01T16:40:19.5974623Z SCCACHE_S3_KEY_PREFIX: pull 2024-11-01T16:40:19.5974974Z SHM_SIZE: 1g 2024-11-01T16:40:19.5975876Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:40:19.5976881Z XLA_CUDA: 2024-11-01T16:40:19.5977369Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-11-01T16:40:19.5977996Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-11-01T16:40:19.5978434Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-11-01T16:40:19.5978843Z DASHBOARD_TAG: 2024-11-01T16:40:19.5979359Z HUGGING_FACE_HUB_TOKEN: *** 2024-11-01T16:40:19.5979917Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2024-11-01T16:40:19.5980304Z IS_A100_RUNNER: 0 2024-11-01T16:40:19.5980803Z ARTIFACTS_FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_32396543718 2024-11-01T16:40:19.5981375Z ##[endgroup] 2024-11-01T16:40:19.6197906Z Traceback (most recent call last): 2024-11-01T16:40:19.6216516Z File "/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/scripts/get_aws_session_tokens.py", line 2, in 2024-11-01T16:40:19.6217543Z import boto3 # type: ignore[import] 2024-11-01T16:40:19.6218154Z ModuleNotFoundError: No module named 'boto3' 2024-11-01T16:40:19.6225162Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-11-01T16:40:19.6226144Z + [[ linux-jammy-py3.9-gcc11 == *onnx* ]] 2024-11-01T16:40:19.6226597Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-11-01T16:40:19.6233205Z +++ nproc --ignore=2 2024-11-01T16:40:19.6356002Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=6 -e SCCACHE_BUCKET -e SCCACHE_REGION -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e IS_A100_RUNNER -e ARTIFACTS_FILE_SUFFIX --env-file=/tmp/github_env_11632514903 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:40:24.6301403Z + container_name=ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e 2024-11-01T16:40:24.6305537Z + grep download.pytorch.org /etc/hosts 2024-11-01T16:40:24.6307537Z + docker exec -i ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e sudo bash -c '/bin/cat >> /etc/hosts' 2024-11-01T16:40:24.7996428Z + echo DOCKER_CONTAINER_ID=ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e 2024-11-01T16:40:24.7999821Z ++ echo dist/torch-2.6.0a0+gitd1aa4ef-cp39-cp39-linux_x86_64.whl 2024-11-01T16:40:24.8002006Z + docker exec -t ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e sh -c 'pip install dist/torch-2.6.0a0+gitd1aa4ef-cp39-cp39-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-11-01T16:40:25.7460212Z Processing ./dist/torch-2.6.0a0+gitd1aa4ef-cp39-cp39-linux_x86_64.whl (from torch==2.6.0a0+gitd1aa4ef) 2024-11-01T16:40:26.2134577Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (3.16.1) 2024-11-01T16:40:26.2138809Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (4.12.2) 2024-11-01T16:40:26.2142400Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (2.8.8) 2024-11-01T16:40:26.2146063Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (3.1.4) 2024-11-01T16:40:26.2149959Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (2024.10.0) 2024-11-01T16:40:26.2156402Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (1.13.1) 2024-11-01T16:40:26.2182167Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from sympy==1.13.1->torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (1.3.0) 2024-11-01T16:40:26.2197922Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (3.3.0) 2024-11-01T16:40:26.2225825Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from opt-einsum>=3.3->torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (1.22.4) 2024-11-01T16:40:26.2689196Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from jinja2->torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (3.0.2) 2024-11-01T16:40:26.8969788Z Installing collected packages: torch 2024-11-01T16:40:35.1334978Z Successfully installed torch-2.6.0a0+gitd1aa4ef 2024-11-01T16:40:35.2094708Z + export TERM=vt100 2024-11-01T16:40:35.2095302Z + TERM=vt100 2024-11-01T16:40:35.2096569Z ++ dirname .ci/pytorch/test.sh 2024-11-01T16:40:35.2116529Z + source .ci/pytorch/common.sh 2024-11-01T16:40:35.2124051Z +++ dirname .ci/pytorch/common.sh 2024-11-01T16:40:35.2130050Z ++ source .ci/pytorch/common_utils.sh 2024-11-01T16:40:35.2136651Z +++ declare -f -t trap_add 2024-11-01T16:40:35.2140344Z ++ set -ex 2024-11-01T16:40:35.2140934Z ++ [[ linux-jammy-py3.9-gcc11 == *rocm* ]] 2024-11-01T16:40:35.2141545Z ++ BUILD_TEST_LIBTORCH=0 2024-11-01T16:40:35.2142002Z + [[ linux-jammy-py3.9-gcc11 != *rocm* ]] 2024-11-01T16:40:35.2144313Z ++ stat -c %u /var/lib/jenkins/workspace 2024-11-01T16:40:35.2168733Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-11-01T16:40:35.2169186Z + trap_add cleanup_workspace EXIT 2024-11-01T16:40:35.2169603Z + trap_add_cmd=cleanup_workspace 2024-11-01T16:40:35.2169970Z + shift 2024-11-01T16:40:35.2170227Z + for trap_add_name in "$@" 2024-11-01T16:40:35.2175158Z +++ trap -p EXIT 2024-11-01T16:40:35.2178331Z ++ eval 'extract_trap_cmd ' 2024-11-01T16:40:35.2179036Z +++ extract_trap_cmd 2024-11-01T16:40:35.2179411Z +++ printf '%s\n' '' 2024-11-01T16:40:35.2180014Z ++ printf '%s\n' cleanup_workspace 2024-11-01T16:40:35.2180603Z + trap -- ' 2024-11-01T16:40:35.2180928Z cleanup_workspace' EXIT 2024-11-01T16:40:35.2181402Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-11-01T16:40:35.6876351Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-11-01T16:40:35.7155919Z + echo 'Environment variables:' 2024-11-01T16:40:35.7156530Z Environment variables: 2024-11-01T16:40:35.7156852Z + env 2024-11-01T16:40:35.7174372Z INSTALLED_DB=yes 2024-11-01T16:40:35.7175733Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:40:35.7176530Z CONTINUE_THROUGH_ERROR=False 2024-11-01T16:40:35.7177260Z BUILD_ENVIRONMENT=linux-jammy-py3.9-gcc11 2024-11-01T16:40:35.7177949Z HOSTNAME=ec832529b912 2024-11-01T16:40:35.7179543Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_834aad36-2342-47c5-9899-f047a511c2a5 2024-11-01T16:40:35.7180836Z GITHUB_ACTION=__self 2024-11-01T16:40:35.7181415Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-11-01T16:40:35.7182159Z GITHUB_RUN_NUMBER=261361 2024-11-01T16:40:35.7182509Z TEST_CONFIG=distributed 2024-11-01T16:40:35.7182874Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-11-01T16:40:35.7183708Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-11-01T16:40:35.7184478Z IS_A100_RUNNER=0 2024-11-01T16:40:35.7185262Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2024-11-01T16:40:35.7185675Z GITHUB_TRIGGERING_ACTOR=c00w 2024-11-01T16:40:35.7186027Z GITHUB_REF_TYPE=branch 2024-11-01T16:40:35.7186386Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-11-01T16:40:35.7186851Z BASE_SHA=2055d3c8dc8b93c52eda6b9c07d716448d9e28c9 2024-11-01T16:40:35.7187423Z XLA_CUDA= 2024-11-01T16:40:35.7188031Z HUGGING_FACE_HUB_TOKEN=*** 2024-11-01T16:40:35.7190418Z *** 2024-11-01T16:40:35.7190913Z GITHUB_REPOSITORY_ID=65600975 2024-11-01T16:40:35.7191703Z GITHUB_ACTIONS=true 2024-11-01T16:40:35.7192529Z SHA1=d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:40:35.7193826Z GITHUB_SHA=368da7510f5d755034681777a4e5ae6d33c07b38 2024-11-01T16:40:35.7196500Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/138766/merge 2024-11-01T16:40:35.7197268Z UCC_HOME=/usr 2024-11-01T16:40:35.7197666Z VERBOSE_TEST_LOGS=False 2024-11-01T16:40:35.7198001Z GITHUB_REF=refs/pull/138766/merge 2024-11-01T16:40:35.7198435Z SHARD_NUMBER=1 2024-11-01T16:40:35.7198736Z GITHUB_REF_PROTECTED=false 2024-11-01T16:40:35.7199270Z HOME=/var/lib/jenkins 2024-11-01T16:40:35.7199636Z GITHUB_API_URL=https://api.github.com 2024-11-01T16:40:35.7200125Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-11-01T16:40:35.7200572Z UCX_COMMIT= 2024-11-01T16:40:35.7200859Z SCCACHE_S3_KEY_PREFIX=pull 2024-11-01T16:40:35.7201218Z NUM_TEST_SHARDS=2 2024-11-01T16:40:35.7201544Z UCX_HOME=/usr 2024-11-01T16:40:35.7202518Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_834aad36-2342-47c5-9899-f047a511c2a5 2024-11-01T16:40:35.7203798Z JOB_NAME=linux-jammy-py3.9-gcc11 / test (distributed, 1, 2, linux.2xlarge) 2024-11-01T16:40:35.7205101Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_834aad36-2342-47c5-9899-f047a511c2a5 2024-11-01T16:40:35.7206369Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-11-01T16:40:35.7207129Z GITHUB_EVENT_NAME=pull_request 2024-11-01T16:40:35.7207553Z DASHBOARD_TAG= 2024-11-01T16:40:35.7207842Z GITHUB_RUN_ID=11632514903 2024-11-01T16:40:35.7208938Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_834aad36-2342-47c5-9899-f047a511c2a5 2024-11-01T16:40:35.7209907Z GITHUB_ACTOR=c00w 2024-11-01T16:40:35.7210206Z PR_NUMBER=138766 2024-11-01T16:40:35.7210556Z DESIRED_CUDA= 2024-11-01T16:40:35.7210845Z GITHUB_RUN_ATTEMPT=1 2024-11-01T16:40:35.7211216Z ANACONDA_PYTHON_VERSION=3.9 2024-11-01T16:40:35.7211662Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-11-01T16:40:35.7212184Z TERM=vt100 2024-11-01T16:40:35.7212454Z INSTALLED_VISION=yes 2024-11-01T16:40:35.7212824Z BRANCH=pull/138766 2024-11-01T16:40:35.7213176Z SCCACHE_REGION=us-east-1 2024-11-01T16:40:35.7213586Z OPENSSL_ROOT_DIR=/opt/openssl 2024-11-01T16:40:35.7213956Z CUDA_PATH=/usr/local/cuda 2024-11-01T16:40:35.7214815Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-11-01T16:40:35.7215682Z GITHUB_SERVER_URL=https://github.com 2024-11-01T16:40:35.7216112Z UCC_COMMIT= 2024-11-01T16:40:35.7216383Z REENABLED_ISSUES= 2024-11-01T16:40:35.7216724Z DOCS=yes 2024-11-01T16:40:35.7216993Z INSTALLED_ANDROID= 2024-11-01T16:40:35.7217289Z SHLVL=1 2024-11-01T16:40:35.7217526Z MAX_JOBS=6 2024-11-01T16:40:35.7217799Z GITHUB_ACTOR_ID=486199 2024-11-01T16:40:35.7218296Z GITHUB_WORKFLOW_SHA=368da7510f5d755034681777a4e5ae6d33c07b38 2024-11-01T16:40:35.7218818Z GITHUB_REF_NAME=138766/merge 2024-11-01T16:40:35.7219449Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-11-01T16:40:35.7220034Z GITHUB_JOB=test 2024-11-01T16:40:35.7220391Z NO_TEST_TIMEOUT=False 2024-11-01T16:40:35.7220726Z TD_DISTRIBUTED=False 2024-11-01T16:40:35.7221068Z GITHUB_REPOSITORY=pytorch/pytorch 2024-11-01T16:40:35.7221462Z GITHUB_RETENTION_DAYS=90 2024-11-01T16:40:35.7221789Z OPENSSL_DIR=/opt/openssl 2024-11-01T16:40:35.7222136Z GITHUB_ACTION_REPOSITORY= 2024-11-01T16:40:35.7223230Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:40:35.7224449Z GITHUB_BASE_REF=gh/c00w/2/base 2024-11-01T16:40:35.7224819Z INSTALLED_ACL= 2024-11-01T16:40:35.7225378Z ARTIFACTS_FILE_SUFFIX=test-distributed-1-2-linux.2xlarge_32396543718 2024-11-01T16:40:35.7225939Z CI=true 2024-11-01T16:40:35.7226220Z GITHUB_REPOSITORY_OWNER=pytorch 2024-11-01T16:40:35.7226589Z JOB_ID=32396543718 2024-11-01T16:40:35.7226892Z INSTALLED_PROTOBUF=yes 2024-11-01T16:40:35.7227230Z GITHUB_HEAD_REF=gh/c00w/2/head 2024-11-01T16:40:35.7227579Z GITHUB_ACTION_REF= 2024-11-01T16:40:35.7228146Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-11-01T16:40:35.7228626Z TEST_SHOWLOCALS=False 2024-11-01T16:40:35.7228950Z GITHUB_WORKFLOW=pull 2024-11-01T16:40:35.7229284Z DEBIAN_FRONTEND=noninteractive 2024-11-01T16:40:35.7230241Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_834aad36-2342-47c5-9899-f047a511c2a5 2024-11-01T16:40:35.7231164Z NO_TD=False 2024-11-01T16:40:35.7231537Z SKIP_SCCACHE_INITIALIZATION=1 2024-11-01T16:40:35.7231899Z _=/usr/bin/env 2024-11-01T16:40:35.7232396Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-11-01T16:40:35.7439904Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch 2024-11-01T16:40:35.7441289Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/bin 2024-11-01T16:40:35.7442298Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/lib 2024-11-01T16:40:35.7443229Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/test 2024-11-01T16:40:35.7443951Z + BUILD_DIR=build 2024-11-01T16:40:35.7444353Z + BUILD_RENAMED_DIR=build_renamed 2024-11-01T16:40:35.7444745Z + BUILD_BIN_DIR=build/bin 2024-11-01T16:40:35.7445068Z + SHARD_NUMBER=1 2024-11-01T16:40:35.7445366Z + NUM_TEST_SHARDS=2 2024-11-01T16:40:35.7445678Z + export VALGRIND=ON 2024-11-01T16:40:35.7445988Z + VALGRIND=ON 2024-11-01T16:40:35.7446377Z + [[ linux-jammy-py3.9-gcc11 == *clang9* ]] 2024-11-01T16:40:35.7446806Z + [[ 0 == \1 ]] 2024-11-01T16:40:35.7447097Z + [[ False == \1 ]] 2024-11-01T16:40:35.7447493Z + [[ linux-jammy-py3.9-gcc11 != *bazel* ]] 2024-11-01T16:40:35.7447960Z ++ realpath build/custom_test_artifacts 2024-11-01T16:40:35.7466629Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-11-01T16:40:35.7467908Z + [[ -n '' ]] 2024-11-01T16:40:35.7468368Z + echo 'Environment variables' 2024-11-01T16:40:35.7468753Z Environment variables 2024-11-01T16:40:35.7469080Z + env 2024-11-01T16:40:35.7474845Z INSTALLED_DB=yes 2024-11-01T16:40:35.7475950Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:40:35.7477013Z CONTINUE_THROUGH_ERROR=False 2024-11-01T16:40:35.7477617Z BUILD_ENVIRONMENT=linux-jammy-py3.9-gcc11 2024-11-01T16:40:35.7478420Z HOSTNAME=ec832529b912 2024-11-01T16:40:35.7479354Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_834aad36-2342-47c5-9899-f047a511c2a5 2024-11-01T16:40:35.7480218Z GITHUB_ACTION=__self 2024-11-01T16:40:35.7480563Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-11-01T16:40:35.7480951Z GITHUB_RUN_NUMBER=261361 2024-11-01T16:40:35.7481295Z TEST_CONFIG=distributed 2024-11-01T16:40:35.7481653Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-11-01T16:40:35.7482132Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-11-01T16:40:35.7482544Z IS_A100_RUNNER=0 2024-11-01T16:40:35.7483157Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2024-11-01T16:40:35.7483535Z GITHUB_TRIGGERING_ACTOR=c00w 2024-11-01T16:40:35.7483895Z GITHUB_REF_TYPE=branch 2024-11-01T16:40:35.7484230Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-11-01T16:40:35.7484665Z BASE_SHA=2055d3c8dc8b93c52eda6b9c07d716448d9e28c9 2024-11-01T16:40:35.7485119Z XLA_CUDA= 2024-11-01T16:40:35.7485570Z HUGGING_FACE_HUB_TOKEN=*** 2024-11-01T16:40:35.7485949Z *** 2024-11-01T16:40:35.7486215Z GITHUB_REPOSITORY_ID=65600975 2024-11-01T16:40:35.7486583Z GITHUB_ACTIONS=true 2024-11-01T16:40:35.7486953Z SHA1=d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:40:35.7487482Z GITHUB_SHA=368da7510f5d755034681777a4e5ae6d33c07b38 2024-11-01T16:40:35.7488200Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/138766/merge 2024-11-01T16:40:35.7488859Z UCC_HOME=/usr 2024-11-01T16:40:35.7489146Z VERBOSE_TEST_LOGS=False 2024-11-01T16:40:35.7489489Z GITHUB_REF=refs/pull/138766/merge 2024-11-01T16:40:35.7489861Z SHARD_NUMBER=1 2024-11-01T16:40:35.7490148Z GITHUB_REF_PROTECTED=false 2024-11-01T16:40:35.7490496Z HOME=/var/lib/jenkins 2024-11-01T16:40:35.7490858Z GITHUB_API_URL=https://api.github.com 2024-11-01T16:40:35.7491297Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-11-01T16:40:35.7491899Z UCX_COMMIT= 2024-11-01T16:40:35.7492190Z SCCACHE_S3_KEY_PREFIX=pull 2024-11-01T16:40:35.7492522Z NUM_TEST_SHARDS=2 2024-11-01T16:40:35.7492816Z UCX_HOME=/usr 2024-11-01T16:40:35.7493707Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_834aad36-2342-47c5-9899-f047a511c2a5 2024-11-01T16:40:35.7494868Z JOB_NAME=linux-jammy-py3.9-gcc11 / test (distributed, 1, 2, linux.2xlarge) 2024-11-01T16:40:35.7496157Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_834aad36-2342-47c5-9899-f047a511c2a5 2024-11-01T16:40:35.7497355Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-11-01T16:40:35.7498032Z GITHUB_EVENT_NAME=pull_request 2024-11-01T16:40:35.7498403Z DASHBOARD_TAG= 2024-11-01T16:40:35.7498701Z GITHUB_RUN_ID=11632514903 2024-11-01T16:40:35.7499696Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_834aad36-2342-47c5-9899-f047a511c2a5 2024-11-01T16:40:35.7500605Z GITHUB_ACTOR=c00w 2024-11-01T16:40:35.7500893Z PR_NUMBER=138766 2024-11-01T16:40:35.7501187Z DESIRED_CUDA= 2024-11-01T16:40:35.7501477Z GITHUB_RUN_ATTEMPT=1 2024-11-01T16:40:35.7501784Z VALGRIND=ON 2024-11-01T16:40:35.7502096Z ANACONDA_PYTHON_VERSION=3.9 2024-11-01T16:40:35.7502536Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-11-01T16:40:35.7502996Z TERM=vt100 2024-11-01T16:40:35.7503270Z INSTALLED_VISION=yes 2024-11-01T16:40:35.7503585Z BRANCH=pull/138766 2024-11-01T16:40:35.7503946Z SCCACHE_REGION=us-east-1 2024-11-01T16:40:35.7504297Z OPENSSL_ROOT_DIR=/opt/openssl 2024-11-01T16:40:35.7504652Z CUDA_PATH=/usr/local/cuda 2024-11-01T16:40:35.7505445Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-11-01T16:40:35.7506236Z GITHUB_SERVER_URL=https://github.com 2024-11-01T16:40:35.7506639Z UCC_COMMIT= 2024-11-01T16:40:35.7506913Z REENABLED_ISSUES= 2024-11-01T16:40:35.7507192Z DOCS=yes 2024-11-01T16:40:35.7507461Z INSTALLED_ANDROID= 2024-11-01T16:40:35.7507753Z SHLVL=1 2024-11-01T16:40:35.7508005Z MAX_JOBS=6 2024-11-01T16:40:35.7508280Z GITHUB_ACTOR_ID=486199 2024-11-01T16:40:35.7508713Z GITHUB_WORKFLOW_SHA=368da7510f5d755034681777a4e5ae6d33c07b38 2024-11-01T16:40:35.7509236Z GITHUB_REF_NAME=138766/merge 2024-11-01T16:40:35.7509869Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-11-01T16:40:35.7510470Z GITHUB_JOB=test 2024-11-01T16:40:35.7510767Z NO_TEST_TIMEOUT=False 2024-11-01T16:40:35.7511080Z TD_DISTRIBUTED=False 2024-11-01T16:40:35.7511425Z GITHUB_REPOSITORY=pytorch/pytorch 2024-11-01T16:40:35.7511819Z GITHUB_RETENTION_DAYS=90 2024-11-01T16:40:35.7512164Z OPENSSL_DIR=/opt/openssl 2024-11-01T16:40:35.7512514Z GITHUB_ACTION_REPOSITORY= 2024-11-01T16:40:35.7513698Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:40:35.7514854Z GITHUB_BASE_REF=gh/c00w/2/base 2024-11-01T16:40:35.7515229Z INSTALLED_ACL= 2024-11-01T16:40:35.7515788Z ARTIFACTS_FILE_SUFFIX=test-distributed-1-2-linux.2xlarge_32396543718 2024-11-01T16:40:35.7516356Z CI=true 2024-11-01T16:40:35.7516643Z GITHUB_REPOSITORY_OWNER=pytorch 2024-11-01T16:40:35.7517001Z JOB_ID=32396543718 2024-11-01T16:40:35.7517308Z INSTALLED_PROTOBUF=yes 2024-11-01T16:40:35.7517647Z GITHUB_HEAD_REF=gh/c00w/2/head 2024-11-01T16:40:35.7518018Z GITHUB_ACTION_REF= 2024-11-01T16:40:35.7518465Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-11-01T16:40:35.7518933Z TEST_SHOWLOCALS=False 2024-11-01T16:40:35.7519257Z GITHUB_WORKFLOW=pull 2024-11-01T16:40:35.7519596Z DEBIAN_FRONTEND=noninteractive 2024-11-01T16:40:35.7520572Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_834aad36-2342-47c5-9899-f047a511c2a5 2024-11-01T16:40:35.7521438Z NO_TD=False 2024-11-01T16:40:35.7521719Z SKIP_SCCACHE_INITIALIZATION=1 2024-11-01T16:40:35.7522085Z _=/usr/bin/env 2024-11-01T16:40:35.7522550Z + echo 'Testing pytorch' 2024-11-01T16:40:35.7522897Z Testing pytorch 2024-11-01T16:40:35.7523215Z + export LANG=C.UTF-8 2024-11-01T16:40:35.7523532Z + LANG=C.UTF-8 2024-11-01T16:40:35.7552688Z + PR_NUMBER=138766 2024-11-01T16:40:35.7553324Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-11-01T16:40:35.7554099Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-11-01T16:40:35.7555052Z + [[ linux-jammy-py3.9-gcc11 == *rocm* ]] 2024-11-01T16:40:35.7555894Z + [[ distributed == \s\l\o\w ]] 2024-11-01T16:40:35.7556412Z + [[ linux-jammy-py3.9-gcc11 == *slow-gradcheck* ]] 2024-11-01T16:40:35.7556974Z + [[ linux-jammy-py3.9-gcc11 == *cuda* ]] 2024-11-01T16:40:35.7557485Z + [[ linux-jammy-py3.9-gcc11 == *rocm* ]] 2024-11-01T16:40:35.7557988Z + [[ linux-jammy-py3.9-gcc11 == *xpu* ]] 2024-11-01T16:40:35.7558426Z + [[ distributed == *crossref* ]] 2024-11-01T16:40:35.7559045Z + [[ linux-jammy-py3.9-gcc11 == *rocm* ]] 2024-11-01T16:40:35.7559550Z + [[ linux-jammy-py3.9-gcc11 == *xpu* ]] 2024-11-01T16:40:35.7560094Z + [[ linux-jammy-py3.9-gcc11 != *-bazel-* ]] 2024-11-01T16:40:35.7560599Z + pip_install --user ninja==1.10.2 2024-11-01T16:40:35.7561144Z + pip install --progress-bar off --user ninja==1.10.2 2024-11-01T16:40:36.2124007Z Collecting ninja==1.10.2 2024-11-01T16:40:36.2248349Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-11-01T16:40:36.2363272Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-11-01T16:40:36.7411459Z Installing collected packages: ninja 2024-11-01T16:40:36.7503036Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-11-01T16:40:36.7504419Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-11-01T16:40:36.8145662Z Successfully installed ninja-1.10.2 2024-11-01T16:40:36.8840511Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:40:36.8843413Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:40:36.8844982Z + [[ linux-jammy-py3.9-gcc11 == *aarch64* ]] 2024-11-01T16:40:36.8845464Z + install_tlparse 2024-11-01T16:40:36.8845848Z + pip_install --user tlparse==0.3.25 2024-11-01T16:40:36.8846418Z + pip install --progress-bar off --user tlparse==0.3.25 2024-11-01T16:40:37.3411107Z Collecting tlparse==0.3.25 2024-11-01T16:40:37.3508702Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-11-01T16:40:37.4036732Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-11-01T16:40:38.0078023Z Installing collected packages: tlparse 2024-11-01T16:40:38.0858532Z Successfully installed tlparse-0.3.25 2024-11-01T16:40:38.1538962Z ++ python -m site --user-base 2024-11-01T16:40:38.1954343Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:40:38.1956381Z + [[ linux-jammy-py3.9-gcc11 == *asan* ]] 2024-11-01T16:40:38.1957082Z + [[ linux-jammy-py3.9-gcc11 == *-debug* ]] 2024-11-01T16:40:38.1957629Z + [[ linux-jammy-py3.9-gcc11 != *-bazel-* ]] 2024-11-01T16:40:38.1958522Z + echo 'We are not in debug mode: linux-jammy-py3.9-gcc11. Expect the assertion to pass' 2024-11-01T16:40:38.1959773Z We are not in debug mode: linux-jammy-py3.9-gcc11. Expect the assertion to pass 2024-11-01T16:40:38.1960473Z + cd test 2024-11-01T16:40:38.1961050Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-11-01T16:40:40.4303154Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-11-01T16:40:40.4303989Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-11-01T16:40:40.4306601Z + DYNAMO_BENCHMARK_FLAGS=() 2024-11-01T16:40:40.4307432Z + [[ distributed == *pr_time_benchmarks* ]] 2024-11-01T16:40:40.4308294Z + [[ distributed == *dynamo_eager* ]] 2024-11-01T16:40:40.4308750Z + [[ distributed == *aot_eager* ]] 2024-11-01T16:40:40.4309165Z + [[ distributed == *aot_inductor* ]] 2024-11-01T16:40:40.4309867Z + [[ distributed == *inductor* ]] 2024-11-01T16:40:40.4310273Z + [[ distributed == *dynamic* ]] 2024-11-01T16:40:40.4310654Z + [[ distributed == *cpu* ]] 2024-11-01T16:40:40.4311257Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-11-01T16:40:40.4329095Z + [[ linux-jammy-py3.9-gcc11 == *libtorch* ]] 2024-11-01T16:40:40.4329713Z + [[ linux-jammy-py3.9-gcc11 == *-bazel-* ]] 2024-11-01T16:40:40.4332165Z + cd test 2024-11-01T16:40:40.4333154Z + python -c 'import torch; print(torch.__config__.show())' 2024-11-01T16:40:41.6808584Z PyTorch built with: 2024-11-01T16:40:41.6809342Z - GCC 11.4 2024-11-01T16:40:41.6809717Z - C++ Version: 201703 2024-11-01T16:40:41.6810798Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-11-01T16:40:41.6812019Z - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-11-01T16:40:41.6812788Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-11-01T16:40:41.6813375Z - LAPACK is enabled (usually provided by MKL) 2024-11-01T16:40:41.6813933Z - NNPACK is enabled 2024-11-01T16:40:41.6814323Z - CPU capability usage: AVX512 2024-11-01T16:40:41.6822791Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-11-01T16:40:41.6830575Z 2024-11-01T16:40:41.8918907Z + cd test 2024-11-01T16:40:41.8919683Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-11-01T16:40:43.1505206Z ATen/Parallel: 2024-11-01T16:40:43.1505791Z at::get_num_threads() : 4 2024-11-01T16:40:43.1506393Z at::get_num_interop_threads() : 4 2024-11-01T16:40:43.1507064Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-11-01T16:40:43.1507688Z omp_get_max_threads() : 4 2024-11-01T16:40:43.1509318Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-11-01T16:40:43.1510646Z mkl_get_max_threads() : 4 2024-11-01T16:40:43.1511649Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-11-01T16:40:43.1512657Z std::thread::hardware_concurrency() : 8 2024-11-01T16:40:43.1513444Z Environment variables: 2024-11-01T16:40:43.1513975Z OMP_NUM_THREADS : [not set] 2024-11-01T16:40:43.1514548Z MKL_NUM_THREADS : [not set] 2024-11-01T16:40:43.1515116Z ATen parallel backend: OpenMP 2024-11-01T16:40:43.1515524Z 2024-11-01T16:40:43.3625367Z + [[ linux-jammy-py3.9-gcc11 == *aarch64* ]] 2024-11-01T16:40:43.3625893Z + [[ distributed == *backward* ]] 2024-11-01T16:40:43.3626281Z + [[ distributed == *xla* ]] 2024-11-01T16:40:43.3626994Z + [[ distributed == *executorch* ]] 2024-11-01T16:40:43.3627434Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-11-01T16:40:43.3627990Z + [[ linux-jammy-py3.9-gcc11 == *libtorch* ]] 2024-11-01T16:40:43.3628456Z + [[ distributed == distributed ]] 2024-11-01T16:40:43.3628835Z + test_distributed 2024-11-01T16:40:43.3629229Z + echo 'Testing distributed python tests' 2024-11-01T16:40:43.3629792Z Testing distributed python tests 2024-11-01T16:40:43.3630420Z + python test/run_test.py --distributed-tests --shard 1 2 --verbose 2024-11-01T16:40:43.4879222Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-11-01T16:40:43.4880492Z import pkg_resources 2024-11-01T16:40:49.0367339Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-11-01T16:40:49.0923959Z Ignoring disabled issues: [''] 2024-11-01T16:40:49.1058733Z Found test times from artifacts 2024-11-01T16:40:49.1579134Z Found test times from artifacts 2024-11-01T16:40:49.1600865Z Running all tests 2024-11-01T16:40:49.1757384Z Running parallel tests on 3 processes 2024-11-01T16:40:49.1766081Z Name: tests to run (est. time: 64.97min) 2024-11-01T16:40:49.1766768Z Serial tests (129): 2024-11-01T16:40:49.1767476Z distributed/_tensor/test_dtensor_compile 1/1 2024-11-01T16:40:49.1768364Z distributed/test_inductor_collectives 1/1 2024-11-01T16:40:49.1769242Z distributed/_tensor/debug/test_op_coverage 1/1 2024-11-01T16:40:49.1770217Z distributed/flight_recorder/test_fr_analysis 1/1 2024-11-01T16:40:49.1770745Z distributed/_tools/test_mod_tracker 1/1 2024-11-01T16:40:49.1771288Z distributed/checkpoint/test_nested_dict 1/1 2024-11-01T16:40:49.1772134Z distributed/_shard/sharded_tensor/test_logger 1/1 2024-11-01T16:40:49.1773333Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 2024-11-01T16:40:49.1774798Z distributed/pipelining/test_schedule_multiproc 1/1 2024-11-01T16:40:49.1775983Z distributed/_tools/test_sac_ilp 1/1 2024-11-01T16:40:49.1777084Z distributed/pipelining/test_backward 1/1 2024-11-01T16:40:49.1778206Z distributed/pipelining/test_transformer 1/1 2024-11-01T16:40:49.1778989Z distributed/_tools/test_sac_estimator 1/1 2024-11-01T16:40:49.1779498Z distributed/checkpoint/test_dedup_tensors 1/1 2024-11-01T16:40:49.1780004Z distributed/pipelining/test_pipe 1/1 2024-11-01T16:40:49.1780460Z distributed/_tensor/test_common_rules 1/1 2024-11-01T16:40:49.1780937Z distributed/_tensor/test_attention 1/1 2024-11-01T16:40:49.1781435Z distributed/checkpoint/test_save_load_api 1/1 2024-11-01T16:40:49.1781959Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 2024-11-01T16:40:49.1782468Z distributed/test_c10d_functional_native 1/1 2024-11-01T16:40:49.1782968Z distributed/_tools/test_runtime_estimator 1/1 2024-11-01T16:40:49.1783447Z distributed/_tensor/test_api 1/1 2024-11-01T16:40:49.1783918Z distributed/pipelining/test_microbatch 1/1 2024-11-01T16:40:49.1784436Z distributed/_tensor/test_experimental_ops 1/1 2024-11-01T16:40:49.1784933Z distributed/_tensor/test_tensor_ops 1/1 2024-11-01T16:40:49.1785503Z distributed/_tensor/experimental/test_register_sharding 1/1 2024-11-01T16:40:49.1786048Z distributed/_tensor/test_init 1/1 2024-11-01T16:40:49.1786513Z distributed/_tools/test_memory_tracker 1/1 2024-11-01T16:40:49.1787079Z distributed/_tensor/test_xla_integration 1/1 2024-11-01T16:40:49.1787646Z distributed/_composable/fsdp/test_fully_shard_state 1/1 2024-11-01T16:40:49.1788196Z distributed/pipelining/test_stage 1/1 2024-11-01T16:40:49.1788676Z distributed/pipelining/test_unflatten 1/1 2024-11-01T16:40:49.1789182Z distributed/_composable/test_checkpoint 1/1 2024-11-01T16:40:49.1789660Z distributed/checkpoint/test_fsspec 1/1 2024-11-01T16:40:49.1790232Z distributed/_composable/fully_shard/test_fully_shard_util 1/1 2024-11-01T16:40:49.1791055Z distributed/checkpoint/test_traverse 1/1 2024-11-01T16:40:49.1791566Z distributed/test_compute_comm_reordering 1/1 2024-11-01T16:40:49.1792081Z distributed/_tensor/test_convolution_ops 1/1 2024-11-01T16:40:49.1792567Z distributed/_tensor/test_math_ops 1/1 2024-11-01T16:40:49.1793178Z distributed/checkpoint/test_dtensor_resharding 1/1 2024-11-01T16:40:49.1793859Z distributed/checkpoint/test_format_utils 1/1 2024-11-01T16:40:49.1794390Z distributed/checkpoint/test_tp_checkpoint 1/1 2024-11-01T16:40:49.1794922Z distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 2024-11-01T16:40:49.1795439Z distributed/_tools/test_fsdp2_mem_tracker 1/1 2024-11-01T16:40:49.1795929Z distributed/_tensor/test_redistribute 1/1 2024-11-01T16:40:49.1796430Z distributed/elastic/test_control_plane 1/1 2024-11-01T16:40:49.1796945Z distributed/tensor/parallel/test_tp_style 1/1 2024-11-01T16:40:49.1797434Z distributed/test_collective_utils 1/1 2024-11-01T16:40:49.1797985Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 2024-11-01T16:40:49.1798526Z distributed/test_device_mesh 1/1 2024-11-01T16:40:49.1798952Z distributed/_tensor/test_matrix_ops 1/1 2024-11-01T16:40:49.1799590Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 1/1 2024-11-01T16:40:49.1800221Z distributed/test_functional_api 1/1 2024-11-01T16:40:49.1800731Z distributed/checkpoint/test_fsdp_model_state 1/1 2024-11-01T16:40:49.1801250Z distributed/_tensor/test_optimizers 1/1 2024-11-01T16:40:49.1801777Z distributed/tensor/parallel/test_parallelize_api 1/1 2024-11-01T16:40:49.1802293Z distributed/test_c10d_logger 1/1 2024-11-01T16:40:49.1802733Z distributed/checkpoint/test_utils 1/1 2024-11-01T16:40:49.1803275Z distributed/_composable/fsdp/test_fully_shard_compile 1/1 2024-11-01T16:40:49.1803850Z distributed/_tensor/debug/test_comm_mode 1/1 2024-11-01T16:40:49.1804471Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 2024-11-01T16:40:49.1805187Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 2024-11-01T16:40:49.1805799Z distributed/_composable/test_compose 1/1 2024-11-01T16:40:49.1806291Z distributed/fsdp/test_fsdp_pure_fp16 1/1 2024-11-01T16:40:49.1806845Z distributed/optim/test_apply_optimizer_in_backward 1/1 2024-11-01T16:40:49.1807388Z distributed/fsdp/test_fsdp_overlap 1/1 2024-11-01T16:40:49.1808020Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 1/1 2024-11-01T16:40:49.1808741Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 2024-11-01T16:40:49.1809282Z distributed/_tensor/test_view_ops 1/1 2024-11-01T16:40:49.1809718Z distributed/test_fake_pg 1/1 2024-11-01T16:40:49.1810207Z distributed/checkpoint/test_file_system_checkpoint 1/1 2024-11-01T16:40:49.1810750Z distributed/rpc/test_share_memory 1/1 2024-11-01T16:40:49.1811292Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 2024-11-01T16:40:49.1811884Z distributed/fsdp/test_fsdp_state_dict 1/1 2024-11-01T16:40:49.1812358Z distributed/fsdp/test_fsdp_apply 1/1 2024-11-01T16:40:49.1812825Z distributed/fsdp/test_fsdp_traversal 1/1 2024-11-01T16:40:49.1813410Z distributed/_composable/fully_shard/test_fully_shard_init 1/1 2024-11-01T16:40:49.1814004Z distributed/fsdp/test_fsdp_ignored_modules 1/1 2024-11-01T16:40:49.1814517Z distributed/fsdp/test_fsdp_grad_acc 1/1 2024-11-01T16:40:49.1815017Z distributed/fsdp/test_checkpoint_wrapper 1/1 2024-11-01T16:40:49.1815507Z distributed/fsdp/test_fsdp_memory 1/1 2024-11-01T16:40:49.1815992Z distributed/fsdp/test_fsdp_flatten_params 1/1 2024-11-01T16:40:49.1816572Z distributed/_composable/test_replicate_with_compiler 1/1 2024-11-01T16:40:49.1817148Z distributed/fsdp/test_fsdp_freezing_weights 1/1 2024-11-01T16:40:49.1817705Z distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 2024-11-01T16:40:49.1818218Z distributed/fsdp/test_fsdp_uneven 1/1 2024-11-01T16:40:49.1818871Z distributed/_composable/fsdp/test_fully_shard_extensions 1/1 2024-11-01T16:40:49.1819435Z distributed/fsdp/test_wrap 1/1 2024-11-01T16:40:49.1819829Z distributed/test_c10d_pypg 1/1 2024-11-01T16:40:49.1820273Z distributed/checkpoint/test_state_dict 1/1 2024-11-01T16:40:49.1820845Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 2024-11-01T16:40:49.1821419Z distributed/fsdp/test_fsdp_checkpoint 1/1 2024-11-01T16:40:49.1821969Z distributed/algorithms/test_join 1/1 2024-11-01T16:40:49.1822453Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 2024-11-01T16:40:49.1822930Z distributed/fsdp/test_fsdp_misc 1/1 2024-11-01T16:40:49.1823402Z distributed/fsdp/test_fsdp_hybrid_shard 1/1 2024-11-01T16:40:49.1823937Z distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 2024-11-01T16:40:49.1824521Z distributed/optim/test_zero_redundancy_optimizer 1/1 2024-11-01T16:40:49.1825044Z distributed/_shard/test_sharder 1/1 2024-11-01T16:40:49.1825568Z distributed/_shard/sharding_plan/test_sharding_plan 1/1 2024-11-01T16:40:49.1826145Z distributed/fsdp/test_fsdp_use_orig_params 1/1 2024-11-01T16:40:49.1826686Z distributed/fsdp/test_fsdp_multiple_forward 1/1 2024-11-01T16:40:49.1827368Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 1/1 2024-11-01T16:40:49.1828008Z distributed/rpc/test_faulty_agent 1/1 2024-11-01T16:40:49.1828462Z distributed/fsdp/test_fsdp_comm 1/1 2024-11-01T16:40:49.1828976Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 2024-11-01T16:40:49.1829605Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 2024-11-01T16:40:49.1830176Z distributed/rpc/test_tensorpipe_agent 1/4 2024-11-01T16:40:49.1830672Z distributed/rpc/test_tensorpipe_agent 3/4 2024-11-01T16:40:49.1831236Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 2024-11-01T16:40:49.1831840Z distributed/_shard/sharded_tensor/ops/test_init 1/1 2024-11-01T16:40:49.1832385Z distributed/fsdp/test_fsdp_mixed_precision 1/1 2024-11-01T16:40:49.1832959Z distributed/test_launcher 1/1 2024-11-01T16:40:49.1833468Z distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 2024-11-01T16:40:49.1834013Z distributed/test_c10d_spawn_gloo 1/1 2024-11-01T16:40:49.1834444Z distributed/test_store 1/1 2024-11-01T16:40:49.1834823Z distributed/test_c10d_spawn_ucc 1/1 2024-11-01T16:40:49.1835257Z distributed/test_c10d_nccl 1/1 2024-11-01T16:40:49.1835678Z distributed/test_c10d_spawn_nccl 1/1 2024-11-01T16:40:49.1836117Z distributed/test_c10d_ops_nccl 1/1 2024-11-01T16:40:49.1836561Z distributed/elastic/events/lib_test 1/1 2024-11-01T16:40:49.1837044Z distributed/elastic/metrics/api_test 1/1 2024-11-01T16:40:49.1837565Z distributed/elastic/multiprocessing/api_test 1/1 2024-11-01T16:40:49.1838085Z distributed/elastic/timer/api_test 1/1 2024-11-01T16:40:49.1838604Z distributed/elastic/timer/local_timer_example 1/1 2024-11-01T16:40:49.1839153Z distributed/elastic/timer/local_timer_test 1/1 2024-11-01T16:40:49.1839694Z distributed/elastic/utils/distributed_test 1/1 2024-11-01T16:40:49.1840208Z distributed/elastic/utils/logging_test 1/1 2024-11-01T16:40:49.1840694Z distributed/elastic/utils/util_test 1/1 2024-11-01T16:40:49.1841134Z Parallel tests (0): 2024-11-01T16:40:49.1841477Z Name: excluded (est. time: 0.0min) 2024-11-01T16:40:49.1841866Z Serial tests (0): 2024-11-01T16:40:49.1842170Z Parallel tests (0): 2024-11-01T16:40:49.2032888Z Running distributed/_tensor/test_dtensor_compile 1/1 ... [2024-11-01 16:40:49.202548] 2024-11-01T16:40:49.2034168Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:40:49.2040963Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tensor/test_dtensor_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:40:49.203401] 2024-11-01T16:41:29.1978966Z 2024-11-01T16:41:29.1983641Z distributed/_tensor/test_dtensor_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_compile_1.1_a469e7b066894dae_.log 2024-11-01T16:41:29.2019336Z Running 28 items in this shard: test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_dynamo_disable, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_graph_break, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_contiguous_dtensor_noncontiguous_local_as_tangent, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_different_gradient_placement, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dont_recompile_on_same_placement_devicemesh, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamo_device_mesh_attrs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_noncontiguous_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_graph_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_redistribute_unbalanced_correct_strides, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_dynamic_shapes, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_recompile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs_forward_hook, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_graph_input_is_async, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_unwrap_async_collective_tensor_tangent, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph 2024-11-01T16:41:29.2052028Z 2024-11-01T16:41:29.2053003Z Running distributed/test_inductor_collectives 1/1 ... [2024-11-01 16:41:29.199448] 2024-11-01T16:41:29.2054267Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:41:29.2057767Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/test_inductor_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:41:29.200309] 2024-11-01T16:41:34.0843645Z 2024-11-01T16:41:34.0845406Z distributed/test_inductor_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_inductor_collectives_1.1_4353786c487e4573_.log 2024-11-01T16:41:34.0877718Z Running 48 items in this shard: test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_contiguous_input, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_into_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_output_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor_cudagraph_trees, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_input_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_broadcast_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_c10d_functional_tagged_pt2_compliant, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_async_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_inductor_allreduce_eager_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_permute_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_reduce_scatter_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_backwards, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_GroupMember_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source__get_default_group, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_group_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_graphbreaks_unsupported_async_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_pg_var, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_args_match, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_list, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_to_all_single, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_unspecified, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op0, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op1, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op2, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op3, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op4, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_reduce_scatter, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_support_collective_op_with_async_op_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_pg, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allgather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allreduce, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_reduce_scatter_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_all_gather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_doesnt_mutate_shared, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_reduce_scatter_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_single_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_steal_buffer, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_meta 2024-11-01T16:41:34.0907563Z 2024-11-01T16:41:34.0908102Z Running distributed/_tensor/debug/test_op_coverage 1/1 ... [2024-11-01 16:41:34.085361] 2024-11-01T16:41:34.0908807Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:41:34.0910673Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tensor/debug/test_op_coverage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:41:34.085989] 2024-11-01T16:41:38.4674966Z 2024-11-01T16:41:38.4676920Z distributed/_tensor/debug/test_op_coverage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_ee90816efebb1da2_.log 2024-11-01T16:41:38.4678880Z Running 1 items in this shard: test/distributed/_tensor/debug/test_op_coverage.py::TestOpCoverage::test_trace_with_inductor_decomp 2024-11-01T16:41:38.4679709Z 2024-11-01T16:41:38.4684747Z Running distributed/flight_recorder/test_fr_analysis 1/1 ... [2024-11-01 16:41:38.468232] 2024-11-01T16:41:38.4685478Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:41:38.4690679Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/flight_recorder/test_fr_analysis.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:41:38.468739] 2024-11-01T16:41:42.4453088Z 2024-11-01T16:41:42.4455295Z distributed/flight_recorder/test_fr_analysis 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.flight_recorder.test_fr_analysis_1.1_c81edbc1986bbb29_.log 2024-11-01T16:41:42.4457228Z Running 1 items in this shard: test/distributed/flight_recorder/test_fr_analysis.py::FlightRecorderEventTest::test_match_one_event 2024-11-01T16:41:42.4458062Z 2024-11-01T16:41:42.4462288Z Running distributed/_tools/test_mod_tracker 1/1 ... [2024-11-01 16:41:42.445970] 2024-11-01T16:41:42.4462948Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:41:42.4467865Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tools/test_mod_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:41:42.446450] 2024-11-01T16:41:46.5232987Z 2024-11-01T16:41:46.5234995Z distributed/_tools/test_mod_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_mod_tracker_1.1_16ce026496f28e59_.log 2024-11-01T16:41:46.5238203Z Running 4 items in this shard: test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_ac, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_bw_detection, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_module_hierarchy, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_user_hooks 2024-11-01T16:41:46.5240291Z 2024-11-01T16:41:46.5242721Z Running distributed/checkpoint/test_nested_dict 1/1 ... [2024-11-01 16:41:46.523924] 2024-11-01T16:41:46.5243585Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:41:46.5247574Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/checkpoint/test_nested_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:41:46.524424] 2024-11-01T16:41:50.6013217Z 2024-11-01T16:41:50.6015060Z distributed/checkpoint/test_nested_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_nested_dict_1.1_e22ef8621120214b_.log 2024-11-01T16:41:50.6017327Z Running 2 items in this shard: test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_flattening_round_trip, test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_mapping 2024-11-01T16:41:50.6018546Z 2024-11-01T16:41:50.6024913Z Running distributed/_shard/sharded_tensor/test_logger 1/1 ... [2024-11-01 16:41:50.602250] 2024-11-01T16:41:50.6025643Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:41:50.6032103Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:41:50.602902] 2024-11-01T16:41:54.5799619Z 2024-11-01T16:41:54.5801450Z distributed/_shard/sharded_tensor/test_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_00614d8de49ee238_.log 2024-11-01T16:41:54.5803391Z Running 1 items in this shard: test/distributed/_shard/sharded_tensor/test_logger.py::ShardingSpecLoggerTest::test_get_or_create_logger 2024-11-01T16:41:54.5804234Z 2024-11-01T16:41:54.5808210Z Running distributed/tensor/parallel/test_micro_pipeline_tp 1/1 ... [2024-11-01 16:41:54.580601] 2024-11-01T16:41:54.5808968Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:41:54.5813970Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/tensor/parallel/test_micro_pipeline_tp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:41:54.581099] 2024-11-01T16:41:59.1588347Z 2024-11-01T16:41:59.1591026Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_a1aaaf109cded443_.log 2024-11-01T16:41:59.1617479Z Running 29 items in this shard: test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_all_gather_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_reduce_scatter_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_get_unexposed_collectives 2024-11-01T16:41:59.1640794Z 2024-11-01T16:41:59.1641461Z Running distributed/pipelining/test_schedule_multiproc 1/1 ... [2024-11-01 16:41:59.159676] 2024-11-01T16:41:59.1642193Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:41:59.1644134Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/pipelining/test_schedule_multiproc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:41:59.160277] 2024-11-01T16:42:02.9992021Z 2024-11-01T16:42:02.9994292Z distributed/pipelining/test_schedule_multiproc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_schedule_multiproc_1.1_d7d9026c87f2cd40_.log 2024-11-01T16:42:02.9995559Z 2024-11-01T16:42:03.0001067Z Running distributed/_tools/test_sac_ilp 1/1 ... [2024-11-01 16:42:02.999828] 2024-11-01T16:42:03.0001714Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:42:03.0006781Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tools/test_sac_ilp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:42:03.000349] 2024-11-01T16:42:07.0271052Z 2024-11-01T16:42:07.0273158Z distributed/_tools/test_sac_ilp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_sac_ilp_1.1_f81764fda347aea3_.log 2024-11-01T16:42:07.0276518Z Running 4 items in this shard: test/distributed/_tools/test_sac_ilp.py::TestSACILP::test_sac_ilp_case1, test/distributed/_tools/test_sac_ilp.py::TestSACILP::test_sac_ilp_case2, test/distributed/_tools/test_sac_ilp.py::TestSACILP::test_sac_ilp_case3, test/distributed/_tools/test_sac_ilp.py::TestOptimalCheckpointingPolicy::test_get_optimial_checkpointing_policy_per_module 2024-11-01T16:42:07.0278888Z 2024-11-01T16:42:07.0279720Z Running distributed/pipelining/test_backward 1/1 ... [2024-11-01 16:42:07.027723] 2024-11-01T16:42:07.0280392Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:42:07.0285288Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/pipelining/test_backward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:42:07.028205] 2024-11-01T16:42:11.1065528Z 2024-11-01T16:42:11.1067875Z distributed/pipelining/test_backward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_backward_1.1_8ce8d971dc553868_.log 2024-11-01T16:42:11.1071356Z Running 4 items in this shard: test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward, test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward_input, test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward_weight, test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward_weight_multiple_iters 2024-11-01T16:42:11.1073921Z 2024-11-01T16:42:11.1074893Z Running distributed/pipelining/test_transformer 1/1 ... [2024-11-01 16:42:11.107216] 2024-11-01T16:42:11.1075587Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:42:11.1080009Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/pipelining/test_transformer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:42:11.107701] 2024-11-01T16:42:15.6372177Z 2024-11-01T16:42:15.6374232Z distributed/pipelining/test_transformer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_transformer_1.1_a0aac115c39999a5_.log 2024-11-01T16:42:15.6375976Z Running 1 items in this shard: test/distributed/pipelining/test_transformer.py::TransformerTests::test_ir 2024-11-01T16:42:15.6376665Z 2024-11-01T16:42:15.6381127Z Running distributed/_tools/test_sac_estimator 1/1 ... [2024-11-01 16:42:15.637862] 2024-11-01T16:42:15.6381845Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:42:15.6386607Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tools/test_sac_estimator.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:42:15.638356] 2024-11-01T16:42:19.7149807Z 2024-11-01T16:42:19.7151925Z distributed/_tools/test_sac_estimator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_sac_estimator_1.1_2886b7459565973d_.log 2024-11-01T16:42:19.7154330Z Running 2 items in this shard: test/distributed/_tools/test_sac_estimator.py::TestSACEstimator::test_simple_model_sac_estimation, test/distributed/_tools/test_sac_estimator.py::TestSACEstimator::test_transformer_sac_estimation 2024-11-01T16:42:19.7155649Z 2024-11-01T16:42:19.7158411Z Running distributed/checkpoint/test_dedup_tensors 1/1 ... [2024-11-01 16:42:19.715630] 2024-11-01T16:42:19.7159561Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:42:19.7164268Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/checkpoint/test_dedup_tensors.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:42:19.716125] 2024-11-01T16:42:23.7931136Z 2024-11-01T16:42:23.7934093Z distributed/checkpoint/test_dedup_tensors 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_60c0aaa5133f4847_.log 2024-11-01T16:42:23.7937366Z Running 1 items in this shard: test/distributed/checkpoint/test_dedup_tensors.py::TestDedupTensor::test_dedup_shards 2024-11-01T16:42:23.7938604Z 2024-11-01T16:42:23.7939895Z Running distributed/pipelining/test_pipe 1/1 ... [2024-11-01 16:42:23.793745] 2024-11-01T16:42:23.7940944Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:42:23.7947974Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/pipelining/test_pipe.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:42:23.794344] 2024-11-01T16:42:28.4719526Z 2024-11-01T16:42:28.4721914Z distributed/pipelining/test_pipe 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_pipe_1.1_ac97b86dfe40f9f2_.log 2024-11-01T16:42:28.4724672Z Running 3 items in this shard: test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass0, test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass1, test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass2 2024-11-01T16:42:28.4726300Z 2024-11-01T16:42:28.4728865Z Running distributed/_tensor/test_common_rules 1/1 ... [2024-11-01 16:42:28.472618] 2024-11-01T16:42:28.4729538Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:42:28.4734985Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tensor/test_common_rules.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:42:28.473120] 2024-11-01T16:43:12.8371284Z 2024-11-01T16:43:12.8374224Z distributed/_tensor/test_common_rules 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_common_rules_1.1_df1f450b2118e537_.log 2024-11-01T16:43:12.8385694Z Running 10 items in this shard: test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_basic_propagation, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_errors, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_linearity, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_merge_sharding, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_pointwise_propagation, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_enforce_sharding_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_broadcasting, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_suggestion 2024-11-01T16:43:12.8394939Z 2024-11-01T16:43:12.8395660Z Running distributed/_tensor/test_attention 1/1 ... [2024-11-01 16:43:12.837914] 2024-11-01T16:43:12.8396559Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:43:12.8399624Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tensor/test_attention.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:43:12.838498] 2024-11-01T16:43:16.9158962Z 2024-11-01T16:43:16.9160760Z distributed/_tensor/test_attention 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_attention_1.1_a3ea0e487f04bccc_.log 2024-11-01T16:43:16.9164047Z Running 4 items in this shard: test/distributed/_tensor/test_attention.py::RingAttentionTest::test_is_causal_behavior, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_custom_transformer, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_native_transformer, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa 2024-11-01T16:43:16.9166901Z 2024-11-01T16:43:16.9167418Z Running distributed/checkpoint/test_save_load_api 1/1 ... [2024-11-01 16:43:16.916488] 2024-11-01T16:43:16.9168117Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:43:16.9172670Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/checkpoint/test_save_load_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:43:16.916967] 2024-11-01T16:43:24.9007642Z 2024-11-01T16:43:24.9010059Z distributed/checkpoint/test_save_load_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_save_load_api_1.1_85c83615d47cc128_.log 2024-11-01T16:43:24.9012175Z Running 1 items in this shard: test/distributed/checkpoint/test_save_load_api.py::TestSaveAndLoadAPI::test_auto_detect 2024-11-01T16:43:24.9012942Z 2024-11-01T16:43:24.9016357Z Running distributed/checkpoint/e2e/test_fsdp_ep 1/1 ... [2024-11-01 16:43:24.901423] 2024-11-01T16:43:24.9017054Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:43:24.9022339Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/checkpoint/e2e/test_fsdp_ep.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:43:24.901937] 2024-11-01T16:43:28.8785523Z 2024-11-01T16:43:28.8787563Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_46404614e0971eb0_.log 2024-11-01T16:43:28.8789321Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_fsdp_ep.py::TestFSDPWithEP::test_e2e 2024-11-01T16:43:28.8790015Z 2024-11-01T16:43:28.8793773Z Running distributed/test_c10d_functional_native 1/1 ... [2024-11-01 16:43:28.879136] 2024-11-01T16:43:28.8794456Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:43:28.8799619Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/test_c10d_functional_native.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:43:28.879676] 2024-11-01T16:43:32.7561899Z 2024-11-01T16:43:32.7582136Z distributed/test_c10d_functional_native 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_functional_native_1.1_a60bc979e6a4c6b6_.log 2024-11-01T16:43:32.7605956Z Running 28 items in this shard: test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_to_all_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_broadcast, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_inductor_dtypeview_memory_leak, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_py_work, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_threading, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_unwaited, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_wait_tensor, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_non_contig_input, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_to_all_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_broadcast, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_inplace_op_on_view, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reuse_buffer_after_inplace_collective, test/distributed/test_c10d_functional_native.py::CompileTest::test_ranks_and_tag, test/distributed/test_c10d_functional_native.py::CompileTest::test_wait_tensor 2024-11-01T16:43:32.7625554Z 2024-11-01T16:43:32.7626106Z Running distributed/_tools/test_runtime_estimator 1/1 ... [2024-11-01 16:43:32.756778] 2024-11-01T16:43:32.7626772Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:43:32.7628630Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tools/test_runtime_estimator.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:43:32.757256] 2024-11-01T16:43:36.8857475Z 2024-11-01T16:43:36.8859591Z distributed/_tools/test_runtime_estimator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_runtime_estimator_1.1_0a7283d274c3c6b3_.log 2024-11-01T16:43:36.8861960Z Running 2 items in this shard: test/distributed/_tools/test_runtime_estimator.py::TestRuntimeEstimator::test_conv_model_runtime, test/distributed/_tools/test_runtime_estimator.py::TestRuntimeEstimator::test_transformer_runtime 2024-11-01T16:43:36.8863307Z 2024-11-01T16:43:36.8866209Z Running distributed/_tensor/test_api 1/1 ... [2024-11-01 16:43:36.886379] 2024-11-01T16:43:36.8866837Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:43:36.8871486Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tensor/test_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:43:36.886856] 2024-11-01T16:44:13.4339135Z 2024-11-01T16:44:13.4340829Z distributed/_tensor/test_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_api_1.1_08e79625c7df2a42_.log 2024-11-01T16:44:13.4346375Z Running 8 items in this shard: test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_casting, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn_warning, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_meta, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor_errors, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor_uneven_sharding 2024-11-01T16:44:13.4350834Z 2024-11-01T16:44:13.4351490Z Running distributed/pipelining/test_microbatch 1/1 ... [2024-11-01 16:44:13.434591] 2024-11-01T16:44:13.4352228Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:44:13.4354353Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/pipelining/test_microbatch.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:44:13.435060] 2024-11-01T16:44:17.7121197Z 2024-11-01T16:44:17.7123021Z distributed/pipelining/test_microbatch 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_microbatch_1.1_23b407a2a50b255b_.log 2024-11-01T16:44:17.7125306Z Running 2 items in this shard: test/distributed/pipelining/test_microbatch.py::MicrobatchTests::test_chunk_spec, test/distributed/pipelining/test_microbatch.py::MicrobatchTests::test_split_and_merge 2024-11-01T16:44:17.7126544Z 2024-11-01T16:44:17.7129336Z Running distributed/_tensor/test_experimental_ops 1/1 ... [2024-11-01 16:44:17.712723] 2024-11-01T16:44:17.7130027Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:44:17.7135010Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tensor/test_experimental_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:44:17.713210] 2024-11-01T16:44:33.3061976Z 2024-11-01T16:44:33.3064974Z distributed/_tensor/test_experimental_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_experimental_ops_1.1_c2a35f2aaeac9ea2_.log 2024-11-01T16:44:33.3069807Z Running 3 items in this shard: test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_bernoulli, test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_nll, test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_slice 2024-11-01T16:44:33.3072731Z 2024-11-01T16:44:33.3073528Z Running distributed/_tensor/test_tensor_ops 1/1 ... [2024-11-01 16:44:33.306984] 2024-11-01T16:44:33.3074635Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:44:33.3080661Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tensor/test_tensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:44:33.307620] 2024-11-01T16:46:19.4965548Z 2024-11-01T16:46:19.4968206Z distributed/_tensor/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_tensor_ops_1.1_530eebccad0742f6_.log 2024-11-01T16:46:19.4989077Z Running 25 items in this shard: test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_aten_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_clone, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_detach, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_dtensor_dtype_conversion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_empty_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_equal, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_full_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_gather, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_index, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_inplace_op, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_empty_strided, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_full, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_op_out_variant, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_scatter, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_slice, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_stack, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_where_type_promotion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zero_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like_partial_sum 2024-11-01T16:46:19.5007025Z 2024-11-01T16:46:19.5008121Z Running distributed/_tensor/experimental/test_register_sharding 1/1 ... [2024-11-01 16:46:19.496973] 2024-11-01T16:46:19.5009334Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:46:19.5012206Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tensor/experimental/test_register_sharding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:46:19.497574] 2024-11-01T16:46:31.6942589Z 2024-11-01T16:46:31.6944910Z distributed/_tensor/experimental/test_register_sharding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.experimental.test_register_sharding_1.1_1ceedf54c093af65_.log 2024-11-01T16:46:31.6947547Z Running 2 items in this shard: test/distributed/_tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_argmax, test/distributed/_tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_softmax_fwd 2024-11-01T16:46:31.6948969Z 2024-11-01T16:46:31.6951240Z Running distributed/_tensor/test_init 1/1 ... [2024-11-01 16:46:31.694877] 2024-11-01T16:46:31.6951925Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:46:31.6957247Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tensor/test_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:46:31.695383] 2024-11-01T16:47:04.1304338Z 2024-11-01T16:47:04.1306939Z distributed/_tensor/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_init_1.1_ab1369296e26e546_.log 2024-11-01T16:47:04.1314220Z Running 7 items in this shard: test/distributed/_tensor/test_init.py::DTensorInitOpsTest::test_init_ops, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_empty, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_full, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_ones, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros_full_mesh, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros_submesh 2024-11-01T16:47:04.1320411Z 2024-11-01T16:47:04.1321361Z Running distributed/_tools/test_memory_tracker 1/1 ... [2024-11-01 16:47:04.131030] 2024-11-01T16:47:04.1322561Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:47:04.1326163Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tools/test_memory_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:47:04.131580] 2024-11-01T16:47:08.0579663Z 2024-11-01T16:47:08.0581849Z distributed/_tools/test_memory_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_memory_tracker_1.1_7b995ee20278e400_.log 2024-11-01T16:47:08.0583883Z Running 1 items in this shard: test/distributed/_tools/test_memory_tracker.py::TestMemoryTracker::test_local_model 2024-11-01T16:47:08.0584609Z 2024-11-01T16:47:08.0588208Z Running distributed/_tensor/test_xla_integration 1/1 ... [2024-11-01 16:47:08.058586] 2024-11-01T16:47:08.0588899Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:47:08.0593655Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tensor/test_xla_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:47:08.059058] 2024-11-01T16:47:12.1857825Z 2024-11-01T16:47:12.1859703Z distributed/_tensor/test_xla_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_xla_integration_1.1_5d84593e13fedf1b_.log 2024-11-01T16:47:12.1862853Z Running 3 items in this shard: test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_replicate, test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_shard, test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_2d 2024-11-01T16:47:12.1865006Z 2024-11-01T16:47:12.1867350Z Running distributed/_composable/fsdp/test_fully_shard_state 1/1 ... [2024-11-01 16:47:12.186460] 2024-11-01T16:47:12.1868101Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:47:12.1872997Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:47:12.186955] 2024-11-01T16:47:16.1133448Z 2024-11-01T16:47:16.1135380Z distributed/_composable/fsdp/test_fully_shard_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_f6769fe9317d31b5_.log 2024-11-01T16:47:16.1139776Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_cls, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_deepcopy, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_reapply, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_state, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_unsupported_module_cls 2024-11-01T16:47:16.1143130Z 2024-11-01T16:47:16.1143572Z Running distributed/pipelining/test_stage 1/1 ... [2024-11-01 16:47:16.113954] 2024-11-01T16:47:16.1144225Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:47:16.1147359Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/pipelining/test_stage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:47:16.114438] 2024-11-01T16:47:19.9234834Z 2024-11-01T16:47:19.9236942Z distributed/pipelining/test_stage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_stage_1.1_8e1167c086be5b0c_.log 2024-11-01T16:47:19.9238073Z 2024-11-01T16:47:19.9243760Z Running distributed/pipelining/test_unflatten 1/1 ... [2024-11-01 16:47:19.924103] 2024-11-01T16:47:19.9244446Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:47:19.9249190Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/pipelining/test_unflatten.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:47:19.924607] 2024-11-01T16:47:24.4018163Z 2024-11-01T16:47:24.4020268Z distributed/pipelining/test_unflatten 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_unflatten_1.1_fd702e8802ac731f_.log 2024-11-01T16:47:24.4022110Z Running 1 items in this shard: test/distributed/pipelining/test_unflatten.py::UnflattenTests::test_unflatten 2024-11-01T16:47:24.4022874Z 2024-11-01T16:47:24.4026657Z Running distributed/_composable/test_checkpoint 1/1 ... [2024-11-01 16:47:24.402462] 2024-11-01T16:47:24.4027339Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:47:24.4032370Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_composable/test_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:47:24.402936] 2024-11-01T16:47:28.4294450Z 2024-11-01T16:47:28.4296304Z distributed/_composable/test_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_checkpoint_1.1_4b3cfe1805dfcb23_.log 2024-11-01T16:47:28.4300460Z Running 6 items in this shard: test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_checkpoint_kwargs, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_clears_state_on_error_in_forward, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_multi_args, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_random_cpu, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_tensor_only_cpu, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_tensor_only_gpu 2024-11-01T16:47:28.4303928Z 2024-11-01T16:47:28.4304396Z Running distributed/checkpoint/test_fsspec 1/1 ... [2024-11-01 16:47:28.430081] 2024-11-01T16:47:28.4305062Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:47:28.4308706Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/checkpoint/test_fsspec.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:47:28.430567] 2024-11-01T16:47:32.2567865Z 2024-11-01T16:47:32.2569923Z distributed/checkpoint/test_fsspec 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsspec_1.1_41962c754bb3eaf7_.log 2024-11-01T16:47:32.2572000Z Running 2 items in this shard: test/distributed/checkpoint/test_fsspec.py::TestFSSpec::test_fsspec, test/distributed/checkpoint/test_fsspec.py::TestFSSpec::test_overwrite 2024-11-01T16:47:32.2573051Z 2024-11-01T16:47:32.2577039Z Running distributed/_composable/fully_shard/test_fully_shard_util 1/1 ... [2024-11-01 16:47:32.257434] 2024-11-01T16:47:32.2577821Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:47:32.2582478Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_util.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:47:32.257922] 2024-11-01T16:47:40.2400405Z 2024-11-01T16:47:40.2402701Z distributed/_composable/fully_shard/test_fully_shard_util 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_util_1.1_be6eac3a6366af29_.log 2024-11-01T16:47:40.2406147Z Running 2 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_util.py::TestUtils::test_get_sharded_module_tree_with_module_name_to_fqns, test/distributed/_composable/fully_shard/test_fully_shard_util.py::TestUtilsSingleDevice::test_foreach_copy_float8 2024-11-01T16:47:40.2407742Z 2024-11-01T16:47:40.2411780Z Running distributed/checkpoint/test_traverse 1/1 ... [2024-11-01 16:47:40.240924] 2024-11-01T16:47:40.2412477Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:47:40.2419325Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/checkpoint/test_traverse.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:47:40.241619] 2024-11-01T16:47:44.3197020Z 2024-11-01T16:47:44.3199058Z distributed/checkpoint/test_traverse 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_traverse_1.1_895540db10557f2a_.log 2024-11-01T16:47:44.3203781Z Running 7 items in this shard: test/distributed/checkpoint/test_traverse.py::TestTraverse::test_get_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_set_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_doesnt_ignore_intermediate_collections, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_dict, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_list, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_shallow, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_with_ordered_dict 2024-11-01T16:47:44.3208018Z 2024-11-01T16:47:44.3208513Z Running distributed/test_compute_comm_reordering 1/1 ... [2024-11-01 16:47:44.320347] 2024-11-01T16:47:44.3209198Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:47:44.3211958Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/test_compute_comm_reordering.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:47:44.320838] 2024-11-01T16:47:48.2475779Z 2024-11-01T16:47:48.2478506Z distributed/test_compute_comm_reordering 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_compute_comm_reordering_1.1_2148f20a2fdefc8e_.log 2024-11-01T16:47:48.2489365Z Running 7 items in this shard: test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_grouped_scheduler_node, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_nccl_heuristics, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_raise_comms, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap_custom_runtime_estimation, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits_raise_comms 2024-11-01T16:47:48.2498242Z 2024-11-01T16:47:48.2499223Z Running distributed/_tensor/test_convolution_ops 1/1 ... [2024-11-01 16:47:48.248204] 2024-11-01T16:47:48.2500453Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:47:48.2503777Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tensor/test_convolution_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:47:48.248725] 2024-11-01T16:48:00.1407953Z 2024-11-01T16:48:00.1410241Z distributed/_tensor/test_convolution_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_convolution_ops_1.1_8e9b172c3a17f6c4_.log 2024-11-01T16:48:00.1413145Z Running 2 items in this shard: test/distributed/_tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_depthwise_convolution, test/distributed/_tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_downsampling_convolution 2024-11-01T16:48:00.1414653Z 2024-11-01T16:48:00.1416431Z Running distributed/_tensor/test_math_ops 1/1 ... [2024-11-01 16:48:00.141440] 2024-11-01T16:48:00.1417259Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:48:00.1422315Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tensor/test_math_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:48:00.141944] 2024-11-01T16:49:00.9364890Z 2024-11-01T16:49:00.9367010Z distributed/_tensor/test_math_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_math_ops_1.1_b4ef298050f823a8_.log 2024-11-01T16:49:00.9374443Z Running 14 items in this shard: test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_foreach_norm, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_bwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_bwd_req_grad, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_fwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_linalg_eigh, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_linear_op_reductions, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_mean, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_nll_loss_and_cross_entropy, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_shard0_svd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_shard_math_ops, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_softmax_fwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_softmax_with_bwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_topk, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_upsampling 2024-11-01T16:49:00.9381095Z 2024-11-01T16:49:00.9381640Z Running distributed/checkpoint/test_dtensor_resharding 1/1 ... [2024-11-01 16:49:00.937146] 2024-11-01T16:49:00.9382367Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:49:00.9384271Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/checkpoint/test_dtensor_resharding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:49:00.937649] 2024-11-01T16:49:25.1054037Z 2024-11-01T16:49:25.1056876Z distributed/checkpoint/test_dtensor_resharding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dtensor_resharding_1.1_57063fd90a312c7c_.log 2024-11-01T16:49:25.1065445Z Running 5 items in this shard: test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_1d_to_1d_reshard_placement_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_2d_to_2d_reshard_placement_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_1d_to_2d_reshard_mesh_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_2d_to_1d_reshard_mesh_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_dtensor_checkpoint_resharding_with_empty_shard 2024-11-01T16:49:25.1069552Z 2024-11-01T16:49:25.1070113Z Running distributed/checkpoint/test_format_utils 1/1 ... [2024-11-01 16:49:25.106086] 2024-11-01T16:49:25.1070806Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:49:25.1073490Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/checkpoint/test_format_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:49:25.106620] 2024-11-01T16:49:41.2080139Z 2024-11-01T16:49:41.2081914Z distributed/checkpoint/test_format_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_format_utils_1.1_1ddf99ddcc148d79_.log 2024-11-01T16:49:41.2085499Z Running 3 items in this shard: test/distributed/checkpoint/test_format_utils.py::TestFormatUtils::test_dcp_to_torch_save, test/distributed/checkpoint/test_format_utils.py::TestFormatUtils::test_online_torch_save_to_dcp, test/distributed/checkpoint/test_format_utils.py::TestFormatUtils::test_torch_save_to_dcp 2024-11-01T16:49:41.2087417Z 2024-11-01T16:49:41.2089062Z Running distributed/checkpoint/test_tp_checkpoint 1/1 ... [2024-11-01 16:49:41.208686] 2024-11-01T16:49:41.2089820Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:49:41.2095062Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/checkpoint/test_tp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:49:41.209164] 2024-11-01T16:49:53.3028590Z 2024-11-01T16:49:53.3030540Z distributed/checkpoint/test_tp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_tp_checkpoint_1.1_9ff78521a680d030_.log 2024-11-01T16:49:53.3033106Z Running 2 items in this shard: test/distributed/checkpoint/test_tp_checkpoint.py::TestTpCheckpoint::test_tp_checkpoint, test/distributed/checkpoint/test_tp_checkpoint.py::TestTpCheckpoint::test_tp_checkpoint_load_on_meta_device 2024-11-01T16:49:53.3034496Z 2024-11-01T16:49:53.3036854Z Running distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 ... [2024-11-01 16:49:53.303490] 2024-11-01T16:49:53.3037548Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:49:53.3042780Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/checkpoint/fsdp/test_fsdp_dsd.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:49:53.303996] 2024-11-01T16:49:57.3305834Z 2024-11-01T16:49:57.3307921Z distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.fsdp.test_fsdp_dsd_1.1_444aedad98422d4a_.log 2024-11-01T16:49:57.3314541Z Running 6 items in this shard: test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_1d_fsdp_cpu_offload_full_model_state_dict, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_1d_fsdp_get_model_state_dict, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp1_and_load_with_fsdp2, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp1_and_load_with_fsdp2_tp, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp2_tp_and_load_with_tp, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_tp_and_load_with_fsdp2_tp 2024-11-01T16:49:57.3319602Z 2024-11-01T16:49:57.3320156Z Running distributed/_tools/test_fsdp2_mem_tracker 1/1 ... [2024-11-01 16:49:57.331240] 2024-11-01T16:49:57.3320897Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:49:57.3322857Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tools/test_fsdp2_mem_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:49:57.331766] 2024-11-01T16:50:01.4585892Z 2024-11-01T16:50:01.4588116Z distributed/_tools/test_fsdp2_mem_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_fsdp2_mem_tracker_1.1_8423f211690d712e_.log 2024-11-01T16:50:01.4591554Z Running 3 items in this shard: test/distributed/_tools/test_fsdp2_mem_tracker.py::TestTrackerFullyShard1DTrainingCore::test_tracker_multi_group_eager, test/distributed/_tools/test_fsdp2_mem_tracker.py::TestTrackerFullyShard1DTrainingCore::test_tracker_non_root_forward_backward, test/distributed/_tools/test_fsdp2_mem_tracker.py::TestTrackerFullyShard1DTrainingCompose::test_tracker_with_activation_checkpointing 2024-11-01T16:50:01.4594603Z 2024-11-01T16:50:01.4595113Z Running distributed/_tensor/test_redistribute 1/1 ... [2024-11-01 16:50:01.459208] 2024-11-01T16:50:01.4595786Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:50:01.4600371Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tensor/test_redistribute.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:50:01.459722] 2024-11-01T16:51:02.8017323Z 2024-11-01T16:51:02.8019316Z distributed/_tensor/test_redistribute 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_redistribute_1.1_e1f15058288238cc_.log 2024-11-01T16:51:02.8028450Z Running 13 items in this shard: test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_partial_to_replicate_forward_backward, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_partial_to_shard, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_redistribute_negative_shard_dim, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_redistribute_shard_dim_change, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_redistribute_uneven_sharding, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_local_partial_grad, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_partial, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_replicate_forward_backward, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_shard_forward_backward, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_shard_dim_alltoall, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_shard_to_replicate_forward_backward, test/distributed/_tensor/test_redistribute.py::MultiDimRedistributeTest::test_multi_dim_mesh, test/distributed/_tensor/test_redistribute.py::MultiDimRedistributeTest::test_redistribute_shard_dim_multi_dim_mesh 2024-11-01T16:51:02.8036794Z 2024-11-01T16:51:02.8037291Z Running distributed/elastic/test_control_plane 1/1 ... [2024-11-01 16:51:02.802388] 2024-11-01T16:51:02.8037975Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:51:02.8039820Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/elastic/test_control_plane.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:51:02.802880] 2024-11-01T16:51:09.8341787Z 2024-11-01T16:51:09.8344029Z distributed/elastic/test_control_plane 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.test_control_plane_1.1_9e6d4dcd4bae34fd_.log 2024-11-01T16:51:09.8350695Z Running 9 items in this shard: test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_dump_nccl_trace_pickle, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_dump_nccl_trace_pickle_with_json, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_dump_nccl_trace_pickle_with_params, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_dump_traceback, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_get_handler_names, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_get_handler_nonexistant, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_run_handler, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_tcp, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_worker_server 2024-11-01T16:51:09.8356025Z 2024-11-01T16:51:09.8356563Z Running distributed/tensor/parallel/test_tp_style 1/1 ... [2024-11-01 16:51:09.834821] 2024-11-01T16:51:09.8357439Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:51:09.8359474Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_style.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:51:09.835321] 2024-11-01T16:51:50.4394628Z 2024-11-01T16:51:50.4397209Z distributed/tensor/parallel/test_tp_style 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_style_1.1_e6d8409a9aa7ab1d_.log 2024-11-01T16:51:50.4404992Z Running 9 items in this shard: test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input_multiple_inputs, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_kwargs_input, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_output, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_sequence_parallel_style 2024-11-01T16:51:50.4410928Z 2024-11-01T16:51:50.4411411Z Running distributed/test_collective_utils 1/1 ... [2024-11-01 16:51:50.440188] 2024-11-01T16:51:50.4412043Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:51:50.4413854Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/test_collective_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:51:50.440708] 2024-11-01T16:51:54.1010419Z 2024-11-01T16:51:54.1012664Z distributed/test_collective_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_collective_utils_1.1_a5ffcda6ee014335_.log 2024-11-01T16:51:54.1013766Z 2024-11-01T16:51:54.1018972Z Running distributed/_composable/fsdp/test_fully_shard_overlap 1/1 ... [2024-11-01 16:51:54.101633] 2024-11-01T16:51:54.1019733Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:51:54.1024678Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_overlap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:51:54.102133] 2024-11-01T16:51:58.0807916Z 2024-11-01T16:51:58.0810280Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_overlap_1.1_ea4ba97d6bd6aef9_.log 2024-11-01T16:51:58.0813332Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_post_optim_event_overlap, test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_training_overlap 2024-11-01T16:51:58.0815202Z 2024-11-01T16:51:58.0816320Z Running distributed/test_device_mesh 1/1 ... [2024-11-01 16:51:58.081415] 2024-11-01T16:51:58.0816947Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:51:58.0821951Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/test_device_mesh.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:51:58.081910] 2024-11-01T16:56:14.6057240Z 2024-11-01T16:56:14.6059140Z distributed/test_device_mesh 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_device_mesh_1.1_84784ae416633333_.log 2024-11-01T16:56:14.6085782Z Running 46 items in this shard: test/distributed/test_device_mesh.py::DeviceMeshTestGlooBackend::test_device_mesh_reuse_default_group, test/distributed/test_device_mesh.py::DeviceMeshTest::test_2d_mesh_eager_init_subgroup, test/distributed/test_device_mesh.py::DeviceMeshTest::test_2d_mesh_non_eager_init_subgroup, test/distributed/test_device_mesh.py::DeviceMeshTest::test_assert_invalid_mesh_tensor, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_2d, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_init_backend, test/distributed/test_device_mesh.py::DeviceMeshTest::test_fake_pg_device_mesh, test/distributed/test_device_mesh.py::DeviceMeshTest::test_from_group_with_global_pg, test/distributed/test_device_mesh.py::DeviceMeshTest::test_from_group_with_invalid_mesh, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_group_and_get_all_groups, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank_raises_exception, test/distributed/test_device_mesh.py::DeviceMeshTest::test_init_process_group, test/distributed/test_device_mesh.py::DeviceMeshTest::test_raises_invalid_device_type, test/distributed/test_device_mesh.py::DeviceMeshTest::test_set_mesh_dim_group_options, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_nd, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_parent_child_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_from_group_with_mesh_shape, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_get_local_rank_3d, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_init_device_mesh, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_duplicate_mesh_dim_names, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_mesh_shape_mesh_dim_names_mismatch, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_cache_and_reuse_submesh_slice_result, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_flatten_mesh_3d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_flatten_mesh_4d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_1d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_2d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_3d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_3d_noncontiguous_slicing, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_invalid_mesh_dim_name, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_no_mesh_dim_found, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_reconstruct_mesh_with_flatten_dim, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_all_submeshes, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_mesh_dim_by_name, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_root_mesh, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_root_mesh_dim_exist, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_root_mesh_dim_not_exist, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_gather_uneven, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_contiguous, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_uneven, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_uneven 2024-11-01T16:56:14.6108045Z 2024-11-01T16:56:14.6108647Z Running distributed/_tensor/test_matrix_ops 1/1 ... [2024-11-01 16:56:14.606444] 2024-11-01T16:56:14.6109302Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:56:14.6111136Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tensor/test_matrix_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:56:14.606960] 2024-11-01T16:56:59.5173344Z 2024-11-01T16:56:59.5175115Z distributed/_tensor/test_matrix_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_matrix_ops_1.1_a2be7291d7988348_.log 2024-11-01T16:56:59.5183167Z Running 10 items in this shard: test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm_auto_redistribute, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm_empty_operand, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_baddbmm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_bmm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_dtensor_mm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_mm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_scaled_dot_product_attention, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_t, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_t_partial 2024-11-01T16:56:59.5188146Z 2024-11-01T16:56:59.5188919Z Running distributed/_composable/fully_shard/test_fully_shard_mixed_precision 1/1 ... [2024-11-01 16:56:59.517959] 2024-11-01T16:56:59.5189756Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:56:59.5191782Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:56:59.518463] 2024-11-01T16:57:07.3023426Z 2024-11-01T16:57:07.3026319Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_mixed_precision_1.1_cf278296d88e0f60_.log 2024-11-01T16:57:07.3028870Z Running 1 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_mixed_precision.py::TestMixedPrecision::test_float16_cast_forward 2024-11-01T16:57:07.3031377Z 2024-11-01T16:57:07.3032413Z Running distributed/test_functional_api 1/1 ... [2024-11-01 16:57:07.302918] 2024-11-01T16:57:07.3033624Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:57:07.3039515Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/test_functional_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:57:07.303513] 2024-11-01T16:57:52.3052178Z 2024-11-01T16:57:52.3054028Z distributed/test_functional_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_functional_api_1.1_a2ea1506a3fcde53_.log 2024-11-01T16:57:52.3085245Z Running 50 items in this shard: test/distributed/test_functional_api.py::TestExpand::test_expand_1d_rank_list, test/distributed/test_functional_api.py::TestExpand::test_expand_2d_rank_list, test/distributed/test_functional_api.py::TestExpand::test_expand_device_mesh, test/distributed/test_functional_api.py::TestExpand::test_expand_device_mesh_tuple, test/distributed/test_functional_api.py::TestExpand::test_expand_process_group, test/distributed/test_functional_api.py::TestPgTag::test_find_or_create_pg, test/distributed/test_functional_api.py::TestPgTag::test_find_root_pg, test/distributed/test_functional_api.py::TestPgTag::test_pg_creation_with_tag, test/distributed/test_functional_api.py::TestPgTag::test_pg_lookup_roundtrip, test/distributed/test_functional_api.py::TestPgTag::test_pg_lookup_with_tag, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_into_tensor_coalesced_device_cpu, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_into_tensor_coalesced_device_cuda, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_tensor_device_cpu, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_gather_tensor_device_cuda, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_coalesced_eager_device_cpu, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_coalesced_eager_device_cuda, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_eager_device_cpu, test/distributed/test_functional_api.py::TestTraceableCollectives::test_all_reduce_eager_device_cuda, test/distributed/test_functional_api.py::TestTraceableCollectives::test_broadcast_device_cpu, test/distributed/test_functional_api.py::TestTraceableCollectives::test_broadcast_device_cuda, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_into_tensor_coalesced_device_cpu, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_into_tensor_coalesced_device_cuda, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_tensor_device_cpu, test/distributed/test_functional_api.py::TestTraceableCollectives::test_reduce_scatter_tensor_device_cuda, test/distributed/test_functional_api.py::TestMetaCollectives::test_all_reduce, test/distributed/test_functional_api.py::TestGradCollectives::test_all_reduce, test/distributed/test_functional_api.py::TestMakeFx::test_all_reduce_tracing, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_gather_into_tensor_coalesced, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_to_all_single, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_to_all_single_1d_input, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_all_to_all_single_split_sizes_none, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_tracing, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_tracing_with_dce_code, test/distributed/test_functional_api.py::TestCollectivesWithNCCL::test_tracing_with_fakepg, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_gather_into_tensor_coalesced, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_to_all_single, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_to_all_single_1d_input, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_all_to_all_single_split_sizes_none, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_permute_tensor_with_sub_group, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_tracing, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_tracing_with_dce_code, test/distributed/test_functional_api.py::TestNCCLCollectivesWithWorldSize4::test_tracing_with_fakepg, test/distributed/test_functional_api.py::TestFunctionalAutograd::test_all_gather_tensor_compile_False, test/distributed/test_functional_api.py::TestFunctionalAutograd::test_all_gather_tensor_compile_True, test/distributed/test_functional_api.py::TestFunctionalAutograd::test_all_to_all_single_compile_False, test/distributed/test_functional_api.py::TestFunctionalAutograd::test_all_to_all_single_compile_True, test/distributed/test_functional_api.py::TestFunctionalAutograd::test_all_to_all_single_inductor, test/distributed/test_functional_api.py::TestFunctionalAutograd::test_reduce_scatter_tensor_compile_False, test/distributed/test_functional_api.py::TestFunctionalAutograd::test_reduce_scatter_tensor_compile_True, test/distributed/test_functional_api.py::TestFunctionalAutogradWithNCCL::test_all_to_all_single 2024-11-01T16:57:52.3112556Z 2024-11-01T16:57:52.3113262Z Running distributed/checkpoint/test_fsdp_model_state 1/1 ... [2024-11-01 16:57:52.305992] 2024-11-01T16:57:52.3113987Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:57:52.3115880Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_model_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:57:52.306490] 2024-11-01T16:58:04.5002697Z 2024-11-01T16:58:04.5004854Z distributed/checkpoint/test_fsdp_model_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_model_state_1.1_62329a94e97347cd_.log 2024-11-01T16:58:04.5008319Z Running 2 items in this shard: test/distributed/checkpoint/test_fsdp_model_state.py::FsdpModelStateCheckpoint::test_fsdp_model_state_no_resharding, test/distributed/checkpoint/test_fsdp_model_state.py::FsdpModelStateCheckpoint::test_fsdp_model_state_with_resharding 2024-11-01T16:58:04.5010505Z 2024-11-01T16:58:04.5011014Z Running distributed/_tensor/test_optimizers 1/1 ... [2024-11-01 16:58:04.500736] 2024-11-01T16:58:04.5011672Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:58:04.5015966Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tensor/test_optimizers.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:58:04.501287] 2024-11-01T16:58:53.3727189Z 2024-11-01T16:58:53.3728831Z distributed/_tensor/test_optimizers 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_optimizers_1.1_19d7c0109d621187_.log 2024-11-01T16:58:53.3736855Z Running 11 items in this shard: test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_RMSprop_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adadelta_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adagrad_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adam_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adamax_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adamw_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_asgd_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_nadam_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_optimizer_foreach_supported_types_include_DTensor, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_radam_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_sgd_1d_sharding 2024-11-01T16:58:53.3743393Z 2024-11-01T16:58:53.3744112Z Running distributed/tensor/parallel/test_parallelize_api 1/1 ... [2024-11-01 16:58:53.373379] 2024-11-01T16:58:53.3744919Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:58:53.3747249Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/tensor/parallel/test_parallelize_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:58:53.373876] 2024-11-01T16:59:46.2006637Z 2024-11-01T16:59:46.2008605Z distributed/tensor/parallel/test_parallelize_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_parallelize_api_1.1_1922db24e879738e_.log 2024-11-01T16:59:46.2019198Z Running 12 items in this shard: test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_empty_plan, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_linear_col_wise_parallel, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_linear_row_wise_parallel, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_mlp_with_module_api, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_mlp_with_module_api_nested, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_module_multi_wildcard, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_module_with_digit, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_module_with_question, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_module_with_star, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_prepare_module_input, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_prepare_module_output, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_under_devicemesh_context 2024-11-01T16:59:46.2028242Z 2024-11-01T16:59:46.2028760Z Running distributed/test_c10d_logger 1/1 ... [2024-11-01 16:59:46.201299] 2024-11-01T16:59:46.2029439Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:59:46.2031328Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/test_c10d_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:59:46.201793] 2024-11-01T17:00:01.7451434Z 2024-11-01T17:00:01.7453385Z distributed/test_c10d_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_logger_1.1_4746d7f07d998299_.log 2024-11-01T17:00:01.7455878Z Running 3 items in this shard: test/distributed/test_c10d_logger.py::C10dErrorLoggerTest::test_exception_logger, test/distributed/test_c10d_logger.py::C10dErrorLoggerTest::test_get_or_create_logger, test/distributed/test_c10d_logger.py::C10dErrorLoggerTest::test_time_logger 2024-11-01T17:00:01.7457456Z 2024-11-01T17:00:01.7460021Z Running distributed/checkpoint/test_utils 1/1 ... [2024-11-01 17:00:01.745731] 2024-11-01T17:00:01.7460799Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:00:01.7465980Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/checkpoint/test_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:00:01.746284] 2024-11-01T17:00:05.8232237Z 2024-11-01T17:00:05.8234237Z distributed/checkpoint/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_utils_1.1_68898c9e76bd447a_.log 2024-11-01T17:00:05.8238713Z Running 6 items in this shard: test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_dcp_logger, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_flat_data, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_index_hint_ignored_on_equals, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_index_hint_ignored_on_hash, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_init_convert_offset, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_sharded_tensor_lookup 2024-11-01T17:00:05.8242197Z 2024-11-01T17:00:05.8243126Z Running distributed/_composable/fsdp/test_fully_shard_compile 1/1 ... [2024-11-01 17:00:05.824064] 2024-11-01T17:00:05.8243890Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:00:05.8249327Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:00:05.824609] 2024-11-01T17:00:09.8012525Z 2024-11-01T17:00:09.8015125Z distributed/_composable/fsdp/test_fully_shard_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_compile_1.1_896262ff9593c0e4_.log 2024-11-01T17:00:09.8028548Z Running 15 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompileCompute::test_disable_compiling_hooks, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_compiled_autograd_ctx, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_dynamo_trace_use_training_state, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_inductor_fullgraph_False, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_inductor_fullgraph_True, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_inductor, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_trace_fsdp_copy_, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor_fullgraph_False, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor_fullgraph_True 2024-11-01T17:00:09.8039788Z 2024-11-01T17:00:09.8040385Z Running distributed/_tensor/debug/test_comm_mode 1/1 ... [2024-11-01 17:00:09.801887] 2024-11-01T17:00:09.8041067Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:00:09.8042930Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tensor/debug/test_comm_mode.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:00:09.802363] 2024-11-01T17:00:13.8802331Z 2024-11-01T17:00:13.8804530Z distributed/_tensor/debug/test_comm_mode 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_comm_mode_1.1_a767bc3608f14b93_.log 2024-11-01T17:00:13.8808251Z Running 4 items in this shard: test/distributed/_tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode, test/distributed/_tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode_coalesced, test/distributed/_tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode_with_c10d, test/distributed/_tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode_with_dtensor 2024-11-01T17:00:13.8810873Z 2024-11-01T17:00:13.8811548Z Running distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 ... [2024-11-01 17:00:13.880873] 2024-11-01T17:00:13.8812402Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:00:13.8816926Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:00:13.881373] 2024-11-01T17:00:17.8579840Z 2024-11-01T17:00:17.8582225Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_clip_grad_norm__1.1_f763df747841611d_.log 2024-11-01T17:00:17.8585207Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py::TestClipGradNormWorldSize2::test_clip_grad_norm_1d, test/distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py::TestClipGradNormWorldSize4::test_clip_grad_norm_2d 2024-11-01T17:00:17.8587537Z 2024-11-01T17:00:17.8588227Z Running distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 ... [2024-11-01 17:00:17.858510] 2024-11-01T17:00:17.8589043Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:00:17.8593632Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:00:17.859011] 2024-11-01T17:00:21.8355014Z 2024-11-01T17:00:21.8357465Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_0b9303d06f4f6a53_.log 2024-11-01T17:00:21.8364903Z Running 7 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_compute_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_grad_acc_with_reduce_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_reduce_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_float16_on_one_submodule, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_norm_modules_bf16, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_norm_modules_fp16, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_submodules_with_external_inputs 2024-11-01T17:00:21.8370452Z 2024-11-01T17:00:21.8370935Z Running distributed/_composable/test_compose 1/1 ... [2024-11-01 17:00:21.836160] 2024-11-01T17:00:21.8371598Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:00:21.8373432Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_composable/test_compose.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:00:21.836654] 2024-11-01T17:01:00.2290310Z 2024-11-01T17:01:00.2292211Z distributed/_composable/test_compose 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_compose_1.1_365449705f3e36d6_.log 2024-11-01T17:01:00.2300420Z Running 9 items in this shard: test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_checkpoint_fsdp_submodules_non_reentrant, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_checkpoint_fsdp_submodules_with_param, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_checkpoint_fsdp_submodules_with_param_no_shard, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_checkpoint_fully_shard_cast_forward_inputs, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_composable_fsdp_replicate, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_fully_shard_replicate_composability, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_fully_shard_replicate_correct_replicate_params, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_state_dict_fsdp_submodules, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_wrap_same_submodule 2024-11-01T17:01:00.2306591Z 2024-11-01T17:01:00.2307103Z Running distributed/fsdp/test_fsdp_pure_fp16 1/1 ... [2024-11-01 17:01:00.229693] 2024-11-01T17:01:00.2307774Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:01:00.2309597Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_pure_fp16.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:01:00.230183] 2024-11-01T17:01:11.6167537Z 2024-11-01T17:01:11.6169275Z distributed/fsdp/test_fsdp_pure_fp16 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_pure_fp16_1.1_b0ffd1b18b5b1cb4_.log 2024-11-01T17:01:11.6171424Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_pure_fp16.py::TestPureFP16::test_fp16_dtypes, test/distributed/fsdp/test_fsdp_pure_fp16.py::TestPureFP16::test_pure_fp16_training 2024-11-01T17:01:11.6172575Z 2024-11-01T17:01:11.6176054Z Running distributed/optim/test_apply_optimizer_in_backward 1/1 ... [2024-11-01 17:01:11.617367] 2024-11-01T17:01:11.6176912Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:01:11.6181507Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/optim/test_apply_optimizer_in_backward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:01:11.617849] 2024-11-01T17:01:14.3598115Z 2024-11-01T17:01:14.3600250Z distributed/optim/test_apply_optimizer_in_backward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_apply_optimizer_in_backward_1.1_d620c8cdf22b1603_.log 2024-11-01T17:01:14.3601654Z 2024-11-01T17:01:14.3606465Z Running distributed/fsdp/test_fsdp_overlap 1/1 ... [2024-11-01 17:01:14.360442] 2024-11-01T17:01:14.3607177Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:01:14.3613009Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:01:14.360947] 2024-11-01T17:01:25.9484834Z 2024-11-01T17:01:25.9486849Z distributed/fsdp/test_fsdp_overlap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_overlap_1.1_45bc8e5c6ed91616_.log 2024-11-01T17:01:25.9489252Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_overlap.py::TestForwardOverlapWorldSizeOne::test_forward_overlap, test/distributed/fsdp/test_fsdp_overlap.py::TestForwardOverlapWorldSizeTwo::test_forward_overlap 2024-11-01T17:01:25.9490890Z 2024-11-01T17:01:25.9491891Z Running distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 1/1 ... [2024-11-01 17:01:25.948923] 2024-11-01T17:01:25.9492728Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:01:25.9497757Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_model_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:01:25.949436] 2024-11-01T17:01:45.2984127Z 2024-11-01T17:01:45.2987821Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_model_checkpoint_1.1_55767205334ffa1e_.log 2024-11-01T17:01:45.2996785Z Running 4 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_model_checkpoint.py::TestModelCheckpointing::test_full_state_dict_save_load_mixed_sharding, test/distributed/_composable/fully_shard/test_fully_shard_model_checkpoint.py::TestModelCheckpointing::test_state_dict_save_load_flow, test/distributed/_composable/fully_shard/test_fully_shard_model_checkpoint.py::TestModelCheckpointing::test_state_dict_save_load_root_fully_shard, test/distributed/_composable/fully_shard/test_fully_shard_model_checkpoint.py::TestModelCheckpointing::test_state_dict_save_load_submodule_fully_shard 2024-11-01T17:01:45.3002617Z 2024-11-01T17:01:45.3003536Z Running distributed/_composable/fsdp/test_fully_shard_frozen 1/1 ... [2024-11-01 17:01:45.299255] 2024-11-01T17:01:45.3004602Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:01:45.3007995Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_frozen.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:01:45.299870] 2024-11-01T17:01:49.2766152Z 2024-11-01T17:01:49.2768318Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_b26fa0a37fd21fd0_.log 2024-11-01T17:01:49.2771810Z Running 3 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_multi_forward_mixed_requires_grad, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_across_groups, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_per_group 2024-11-01T17:01:49.2774237Z 2024-11-01T17:01:49.2774718Z Running distributed/_tensor/test_view_ops 1/1 ... [2024-11-01 17:01:49.277261] 2024-11-01T17:01:49.2775357Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:01:49.2780365Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_tensor/test_view_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:01:49.277752] 2024-11-01T17:02:15.4111895Z 2024-11-01T17:02:15.4114654Z distributed/_tensor/test_view_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_view_ops_1.1_ed58dcdfe9e67fa8_.log 2024-11-01T17:02:15.4120004Z Running 4 items in this shard: test/distributed/_tensor/test_view_ops.py::TestViewOps::test_complex_view_ops, test/distributed/_tensor/test_view_ops.py::TestViewOps::test_dtensor_view_op_uneven, test/distributed/_tensor/test_view_ops.py::TestViewOps::test_view_groups, test/distributed/_tensor/test_view_ops.py::TestViewOps::test_view_ops 2024-11-01T17:02:15.4123389Z 2024-11-01T17:02:15.4124093Z Running distributed/test_fake_pg 1/1 ... [2024-11-01 17:02:15.412133] 2024-11-01T17:02:15.4125062Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:02:15.4131570Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/test_fake_pg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:02:15.412733] 2024-11-01T17:02:19.5395473Z 2024-11-01T17:02:19.5398195Z distributed/test_fake_pg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_fake_pg_1.1_231d4a36c18fa7d6_.log 2024-11-01T17:02:19.5408173Z Running 13 items in this shard: test/distributed/test_fake_pg.py::TestFakePG::test_all_reduce, test/distributed/test_fake_pg.py::TestFakePG::test_allgather, test/distributed/test_fake_pg.py::TestFakePG::test_alltoall, test/distributed/test_fake_pg.py::TestFakePG::test_alltoall_base, test/distributed/test_fake_pg.py::TestFakePG::test_broadcast, test/distributed/test_fake_pg.py::TestFakePG::test_construct_fsdp, test/distributed/test_fake_pg.py::TestFakePG::test_fake_pg_tracing, test/distributed/test_fake_pg.py::TestFakePG::test_fsdp_fake_e2e, test/distributed/test_fake_pg.py::TestFakePG::test_fsdp_tp_fake_e2e, test/distributed/test_fake_pg.py::TestFakePG::test_recv, test/distributed/test_fake_pg.py::TestFakePG::test_reduce_scatter, test/distributed/test_fake_pg.py::TestFakePG::test_scatter, test/distributed/test_fake_pg.py::TestFakePG::test_send 2024-11-01T17:02:19.5416001Z 2024-11-01T17:02:19.5417093Z Running distributed/checkpoint/test_file_system_checkpoint 1/1 ... [2024-11-01 17:02:19.540348] 2024-11-01T17:02:19.5418426Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:02:19.5421202Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:02:19.540946] 2024-11-01T17:02:23.6679949Z 2024-11-01T17:02:23.6682331Z distributed/checkpoint/test_file_system_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_1.1_7f7e45671e56d868_.log 2024-11-01T17:02:23.6688631Z Running 6 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_save_load_bytes, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor 2024-11-01T17:02:23.6693843Z 2024-11-01T17:02:23.6694320Z Running distributed/rpc/test_share_memory 1/1 ... [2024-11-01 17:02:23.668638] 2024-11-01T17:02:23.6694968Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:02:23.6696869Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/rpc/test_share_memory.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:02:23.669145] 2024-11-01T17:02:35.6155816Z 2024-11-01T17:02:35.6157540Z distributed/rpc/test_share_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.test_share_memory_1.1_7b1a234f874ad3d2_.log 2024-11-01T17:02:35.6160015Z Running 1 items in this shard: test/distributed/rpc/test_share_memory.py::TestRPCPickler::test_case 2024-11-01T17:02:35.6161138Z Running 1 items in this shard: test/distributed/rpc/test_share_memory.py::TestRPCPickler::test_case 2024-11-01T17:02:35.6161781Z 2024-11-01T17:02:35.6164538Z Running distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 ... [2024-11-01 17:02:35.616172] 2024-11-01T17:02:35.6165312Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:02:35.6170520Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:02:35.616668] 2024-11-01T17:02:39.6431896Z 2024-11-01T17:02:39.6435124Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_dict_1.1_177838ce76b8a1de_.log 2024-11-01T17:02:39.6446109Z Running 6 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_2d_state_dict_correctness, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_dp_state_dict_cpu_offload, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_dp_state_dict_save_load, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_dp_tp_state_dict_save_load, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_hsdp_tp_state_dict_save_load, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiThread::test_rank0_offload_full_state_dict 2024-11-01T17:02:39.6453900Z 2024-11-01T17:02:39.6454851Z Running distributed/fsdp/test_fsdp_state_dict 1/1 ... [2024-11-01 17:02:39.644037] 2024-11-01T17:02:39.6456048Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:02:39.6458838Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:02:39.644648] 2024-11-01T17:02:44.3225651Z 2024-11-01T17:02:44.3228143Z distributed/fsdp/test_fsdp_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_state_dict_1.1_fd44cf44517c0946_.log 2024-11-01T17:02:44.3414446Z Running 179 items in this shard: test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_full_state_dict_missing_unexpected_keys_cleaned, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_local_state_dict_with_empty_ranks, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_sharded_load_multi_backend_pg, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_shared_module_and_shared_parameter, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_rank0_offload_save_load_flow_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_rank0_offload_save_load_flow_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_type, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_torch_save_load, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_world_size_one, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_wrong_state_dict_config, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict4GPUs::test_local_state_dict_reshard 2024-11-01T17:02:44.3595276Z 2024-11-01T17:02:44.3595860Z Running distributed/fsdp/test_fsdp_apply 1/1 ... [2024-11-01 17:02:44.323927] 2024-11-01T17:02:44.3596512Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:02:44.3598329Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:02:44.324451] 2024-11-01T17:02:59.7206584Z 2024-11-01T17:02:59.7208729Z distributed/fsdp/test_fsdp_apply 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_apply_1.1_1c9c85b75525a841_.log 2024-11-01T17:02:59.7211526Z Running 3 items in this shard: test/distributed/fsdp/test_fsdp_apply.py::TestApply::test_apply_in_summon_raises_error, test/distributed/fsdp/test_fsdp_apply.py::TestApply::test_nested_module_apply, test/distributed/fsdp/test_fsdp_apply.py::TestApply::test_transformer_module_apply 2024-11-01T17:02:59.7213162Z 2024-11-01T17:02:59.7215560Z Running distributed/fsdp/test_fsdp_traversal 1/1 ... [2024-11-01 17:02:59.721286] 2024-11-01T17:02:59.7216238Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:02:59.7221127Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:02:59.721794] 2024-11-01T17:03:07.5542497Z 2024-11-01T17:03:07.5544538Z distributed/fsdp/test_fsdp_traversal 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_traversal_1.1_8036c0671fba494a_.log 2024-11-01T17:03:07.5546246Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_traversal.py::TestTraversal::test_fsdp_modules 2024-11-01T17:03:07.5546949Z 2024-11-01T17:03:07.5550980Z Running distributed/_composable/fully_shard/test_fully_shard_init 1/1 ... [2024-11-01 17:03:07.554846] 2024-11-01T17:03:07.5551753Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:03:07.5574007Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:03:07.555371] 2024-11-01T17:03:38.3761544Z 2024-11-01T17:03:38.3763917Z distributed/_composable/fully_shard/test_fully_shard_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_init_1.1_d4d8a8330349fcab_.log 2024-11-01T17:03:38.3771829Z Running 7 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_device_id, test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_manual_fully_shard, test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_materialize_meta_module, test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_nested_fully_shard_shared_state, test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_policy, test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_raise_scalar_parameter, test/distributed/_composable/fully_shard/test_fully_shard_init.py::TestInitialization::test_sync_module_states 2024-11-01T17:03:38.3777028Z 2024-11-01T17:03:38.3777596Z Running distributed/fsdp/test_fsdp_ignored_modules 1/1 ... [2024-11-01 17:03:38.376848] 2024-11-01T17:03:38.3778299Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:03:38.3780172Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_ignored_modules.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:03:38.377365] 2024-11-01T17:03:42.3540543Z 2024-11-01T17:03:42.3542355Z distributed/fsdp/test_fsdp_ignored_modules 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_ignored_modules_1.1_f6ee96b5d7120393_.log 2024-11-01T17:03:42.3553157Z Running 13 items in this shard: test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_diff_ignored_modules_across_ranks, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_invalid_composable_False, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_invalid_composable_True, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_nested, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_nested_composable, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_False_composable_False, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_False_composable_True, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_True_composable_False, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_True_composable_True, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_transformer, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_transformer_composable, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_states_auto_wrap, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_states_check 2024-11-01T17:03:42.3562839Z 2024-11-01T17:03:42.3563316Z Running distributed/fsdp/test_fsdp_grad_acc 1/1 ... [2024-11-01 17:03:42.354560] 2024-11-01T17:03:42.3563984Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:03:42.3565807Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:03:42.355060] 2024-11-01T17:04:09.2673709Z 2024-11-01T17:04:09.2676922Z distributed/fsdp/test_fsdp_grad_acc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_grad_acc_1.1_6f5726a54111d402_.log 2024-11-01T17:04:09.2681870Z Running 6 items in this shard: test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs0_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs0_use_orig_params_True, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs1_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs1_use_orig_params_True, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_cpu_offload_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_cpu_offload_use_orig_params_True 2024-11-01T17:04:09.2685491Z 2024-11-01T17:04:09.2686005Z Running distributed/fsdp/test_checkpoint_wrapper 1/1 ... [2024-11-01 17:04:09.268014] 2024-11-01T17:04:09.2687054Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:04:09.2688990Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:04:09.268554] 2024-11-01T17:04:13.2952017Z 2024-11-01T17:04:13.2954056Z distributed/fsdp/test_checkpoint_wrapper 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_checkpoint_wrapper_1.1_0ac5f03ab832e366_.log 2024-11-01T17:04:13.2961184Z Running 8 items in this shard: test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_apply_activation_checkpointing, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_args_kwargs, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_cpu_offload, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_kwarg_support, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_parity, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_forward_missing_attributes, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_fqn, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_load_activation_checkpointed_module 2024-11-01T17:04:13.2966375Z 2024-11-01T17:04:13.2966859Z Running distributed/fsdp/test_fsdp_memory 1/1 ... [2024-11-01 17:04:13.295894] 2024-11-01T17:04:13.2967508Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:04:13.2969316Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:04:13.296411] 2024-11-01T17:04:24.8844945Z 2024-11-01T17:04:24.8847391Z distributed/fsdp/test_fsdp_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_memory_1.1_622fe548029732e7_.log 2024-11-01T17:04:24.8851533Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_memory.py::TestFSDPMemory::test_fsdp_memory_ckpt_ckpt, test/distributed/fsdp/test_fsdp_memory.py::TestFSDPMemory::test_fsdp_memory_ckpt_no_ckpt 2024-11-01T17:04:24.8853803Z 2024-11-01T17:04:24.8857742Z Running distributed/fsdp/test_fsdp_flatten_params 1/1 ... [2024-11-01 17:04:24.885458] 2024-11-01T17:04:24.8858477Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:04:24.8863457Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_flatten_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:04:24.886018] 2024-11-01T17:05:17.2811285Z 2024-11-01T17:05:17.2813593Z distributed/fsdp/test_fsdp_flatten_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_flatten_params_1.1_2e816716b0bb5106_.log 2024-11-01T17:05:17.2823356Z Running 13 items in this shard: test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_empty_module, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_aligned_full_precision, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_aligned_mixed_precision, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_unaligned, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_with_memory_format_memory_format0, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_with_memory_format_memory_format1, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flatten_nothing, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_numel_with_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_numel_without_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_output_with_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_output_without_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_partial_flattening, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_pnorm_after_step_with_shared_params 2024-11-01T17:05:17.2832255Z 2024-11-01T17:05:17.2832965Z Running distributed/_composable/test_replicate_with_compiler 1/1 ... [2024-11-01 17:05:17.281838] 2024-11-01T17:05:17.2833715Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:05:17.2835645Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_composable/test_replicate_with_compiler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:05:17.282364] 2024-11-01T17:05:21.3610401Z 2024-11-01T17:05:21.3612706Z distributed/_composable/test_replicate_with_compiler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_replicate_with_compiler_1.1_f079b1e9bf507609_.log 2024-11-01T17:05:21.3620174Z Running 10 items in this shard: test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_bucketing_coalesced_op, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_bucketing_concat_op, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_backward_only, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_bf16, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_cpu, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_cpu_no_sync, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_fp16, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_gpu, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_gpu_ac, test/distributed/_composable/test_replicate_with_compiler.py::DDP_TP_Test::test_ddp_tp 2024-11-01T17:05:21.3625995Z 2024-11-01T17:05:21.3626541Z Running distributed/fsdp/test_fsdp_freezing_weights 1/1 ... [2024-11-01 17:05:21.361663] 2024-11-01T17:05:21.3627248Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:05:21.3629113Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_freezing_weights.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:05:21.362167] 2024-11-01T17:07:24.3064798Z 2024-11-01T17:07:24.3066895Z distributed/fsdp/test_fsdp_freezing_weights 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_freezing_weights_1.1_546683fc4aa86702_.log 2024-11-01T17:07:24.3109847Z Running 32 items in this shard: test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True 2024-11-01T17:07:24.3150090Z 2024-11-01T17:07:24.3150712Z Running distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 ... [2024-11-01 17:07:24.306819] 2024-11-01T17:07:24.3151426Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:07:24.3153369Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_hsdp_dtensor_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:07:24.307377] 2024-11-01T17:07:48.6257269Z 2024-11-01T17:07:48.6259279Z distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_hsdp_dtensor_state_dict_1.1_0df314066c2d45f7_.log 2024-11-01T17:07:48.6265363Z Running 5 items in this shard: test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_hsdp_init_with_device_mesh, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_root_module_is_not_FSDP 2024-11-01T17:07:48.6269959Z 2024-11-01T17:07:48.6270607Z Running distributed/fsdp/test_fsdp_uneven 1/1 ... [2024-11-01 17:07:48.626441] 2024-11-01T17:07:48.6271629Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:07:48.6273540Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:07:48.626936] 2024-11-01T17:07:56.3089303Z 2024-11-01T17:07:56.3091305Z distributed/fsdp/test_fsdp_uneven 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_uneven_1.1_8dae878b6c58bc7f_.log 2024-11-01T17:07:56.3093044Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShard::test_one_iteration 2024-11-01T17:07:56.3093795Z 2024-11-01T17:07:56.3097771Z Running distributed/_composable/fsdp/test_fully_shard_extensions 1/1 ... [2024-11-01 17:07:56.309562] 2024-11-01T17:07:56.3098540Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:07:56.3103275Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_extensions.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:07:56.310028] 2024-11-01T17:08:04.2924613Z 2024-11-01T17:08:04.2926840Z distributed/_composable/fsdp/test_fully_shard_extensions 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_extensions_1.1_16cafc7859d812d8_.log 2024-11-01T17:08:04.2932620Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiProcess::test_all_gather_extensions_train_parity, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extension_hsdp_mesh, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extension_outer_size_stride, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extensions_end_to_end, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extensions_monkey_patch 2024-11-01T17:08:04.2937393Z 2024-11-01T17:08:04.2937802Z Running distributed/fsdp/test_wrap 1/1 ... [2024-11-01 17:08:04.293383] 2024-11-01T17:08:04.2938621Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:08:04.2943466Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:08:04.294052] 2024-11-01T17:09:38.5976707Z 2024-11-01T17:09:38.5979335Z distributed/fsdp/test_wrap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_wrap_1.1_4845fb334e39fd1b_.log 2024-11-01T17:09:38.6014085Z Running 51 items in this shard: test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_bn_always_wrapped_individually, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_wrap_batchnorm_individually_use_or_policy_False, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_wrap_batchnorm_individually_use_or_policy_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap_with_ignored_modules_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap_with_ignored_modules_wrap_method1, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_api, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_exclude_wrap, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_exclude_wrap_include_children, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_force_leaf, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_force_leaf_custom, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload0_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload0_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload1_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload1_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload0_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload0_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload1_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload1_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_with_ignored_modules_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_with_ignored_modules_wrap_method1, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_custom_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_frozen_params, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_module_wrap_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_module_wrap_policy_callable, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_transformer_auto_wrap_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_disabled_outside_context, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_override_defaults, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_wrap_method1, test/distributed/fsdp/test_wrap.py::TestWrapUtils::test_validate_frozen_params 2024-11-01T17:09:38.6045000Z 2024-11-01T17:09:38.6045521Z Running distributed/test_c10d_pypg 1/1 ... [2024-11-01 17:09:38.598394] 2024-11-01T17:09:38.6046119Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:09:38.6047893Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:09:38.598904] 2024-11-01T17:12:18.9004115Z 2024-11-01T17:12:18.9006557Z distributed/test_c10d_pypg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_pypg_1.1_df943223382cde97_.log 2024-11-01T17:12:18.9047275Z Running 42 items in this shard: test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_dataclass_output, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_dataclass_output_unused_param, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_invoke_work_object, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_with_pypg, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_with_pypg_with_grad_views, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_invalid_powerSGD_state, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_sync_batch_norm_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_dataclass_output, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_dataclass_output_unused_param, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_invoke_work_object, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_with_pypg, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_with_pypg_with_grad_views, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_invalid_powerSGD_state, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_sync_batch_norm_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_sync_batch_norm_only_empty_input 2024-11-01T17:12:18.9079278Z 2024-11-01T17:12:18.9079895Z Running distributed/checkpoint/test_state_dict 1/1 ... [2024-11-01 17:12:18.901045] 2024-11-01T17:12:18.9080591Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:12:18.9082459Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/checkpoint/test_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:12:18.901643] 2024-11-01T17:12:27.6391572Z 2024-11-01T17:12:27.6393514Z distributed/checkpoint/test_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_state_dict_1.1_bc93eb1775f48ba3_.log 2024-11-01T17:12:27.6408766Z Running 25 items in this shard: test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_activation_ckpt_fqns_ddp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_activation_ckpt_fqns_fsdp1, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_broadcast_from_rank0, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_broadcast_from_rank0_hsdp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_compiled_fsdp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_cpu_offload_full_state_dict, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_ddp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_deprecate_fsdp_api, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_deprecate_partial, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_extra_state, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_flattened_osd, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_frozen_parameters, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_fsdp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_fsdp2, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_fsdp_ddp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_fsdp_root_not_initialized, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_non_persistent_buffers, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_optim_state_dict_param_matching, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_optim_state_dict_tensor_matching, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_setting_meta_device_model, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_setting_meta_device_model_broadcasting, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_shared_weight, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_single_gpu, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_strict, test/distributed/checkpoint/test_state_dict.py::TestNoComm::test_no_dist 2024-11-01T17:12:27.6422427Z 2024-11-01T17:12:27.6423097Z Running distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 ... [2024-11-01 17:12:27.639860] 2024-11-01T17:12:27.6423901Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:12:27.6425979Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:12:27.640338] 2024-11-01T17:12:31.4667729Z 2024-11-01T17:12:31.4670386Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks_1.1_659d63b1accbdcb4_.log 2024-11-01T17:12:31.4677584Z Running 6 items in this shard: test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_allreduce_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_fp16compress_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_noop_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_quantize_per_channel_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_quantize_per_tensor_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_is_last_hook 2024-11-01T17:12:31.4682990Z 2024-11-01T17:12:31.4683505Z Running distributed/fsdp/test_fsdp_checkpoint 1/1 ... [2024-11-01 17:12:31.467381] 2024-11-01T17:12:31.4684175Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:12:31.4686029Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:12:31.467855] 2024-11-01T17:13:38.7823060Z 2024-11-01T17:13:38.7825130Z distributed/fsdp/test_fsdp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_checkpoint_1.1_f83e6bd475faa888_.log 2024-11-01T17:13:38.7843149Z Running 17 items in this shard: test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpointSubmodule::test_checkpoint_submodule_use_reentrant_False 2024-11-01T17:13:38.7857812Z 2024-11-01T17:13:38.7858337Z Running distributed/algorithms/test_join 1/1 ... [2024-11-01 17:13:38.783128] 2024-11-01T17:13:38.7859179Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:13:38.7860996Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/algorithms/test_join.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:13:38.783715] 2024-11-01T17:14:17.4640883Z 2024-11-01T17:14:17.4642790Z distributed/algorithms/test_join 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.test_join_1.1_a71032f07223b852_.log 2024-11-01T17:14:17.4649037Z Running 9 items in this shard: test/distributed/algorithms/test_join.py::TestJoin::test_join_kwargs, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinable_disable, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinables, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinables_throw, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_disable, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_main_hooks, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_post_hooks, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_throw 2024-11-01T17:14:17.4653892Z 2024-11-01T17:14:17.4654486Z Running distributed/fsdp/test_fsdp_clip_grad_norm 1/1 ... [2024-11-01 17:14:17.464765] 2024-11-01T17:14:17.4655213Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:14:17.4657246Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:14:17.465260] 2024-11-01T17:14:36.3120455Z 2024-11-01T17:14:36.3123284Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_clip_grad_norm_1.1_6fe1b389972e5d09_.log 2024-11-01T17:14:36.3129022Z Running 4 items in this shard: test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNorm::test_ddp_parity, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNorm::test_low_precision_grads, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNorm::test_no_gradients, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNorm::test_non_root 2024-11-01T17:14:36.3132704Z 2024-11-01T17:14:36.3133521Z Running distributed/fsdp/test_fsdp_misc 1/1 ... [2024-11-01 17:14:36.312577] 2024-11-01T17:14:36.3134724Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:14:36.3137396Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_misc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:14:36.313198] 2024-11-01T17:15:34.2725341Z 2024-11-01T17:15:34.2727336Z distributed/fsdp/test_fsdp_misc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_misc_1.1_57b1dd8d775c0295_.log 2024-11-01T17:15:34.2746611Z Running 28 items in this shard: test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_cpu_init_with_sync_module_states, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_cpu_init_stays_on_cpu, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_cpu_training, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_device_id_use_index_False, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_device_id_use_index_True, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_not_all_outputs_used_in_loss, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_optim_overlap_no_use_orig_params_error, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_optimizer_overlap, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_zero2_eval_with_prefetch, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_cpu_gpu_module, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_device_id_auto_wrap, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_device_id_cpu_offload, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_device_id_no_move_ignored_params_and_bufs, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_ignored_module_meta, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_namedtuple, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_same_model_across_ranks, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_unsupported_module_cls, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_homogeneous_attributes, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_module_device_mismatches_device_id, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_multigpu_module, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_no_params, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_training_device_mismatch_errors, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_unsafe_setattr, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_world_size_1_sharding_strategy_warning 2024-11-01T17:15:34.2765140Z 2024-11-01T17:15:34.2765696Z Running distributed/fsdp/test_fsdp_hybrid_shard 1/1 ... [2024-11-01 17:15:34.273369] 2024-11-01T17:15:34.2766438Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:15:34.2768304Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_hybrid_shard.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:15:34.273894] 2024-11-01T17:16:01.1855456Z 2024-11-01T17:16:01.1857740Z distributed/fsdp/test_fsdp_hybrid_shard 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_hybrid_shard_1.1_799418be87dd7274_.log 2024-11-01T17:16:01.1863595Z Running 6 items in this shard: test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_fsdp_hybrid_shard_basic_setup, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_fsdp_hybrid_shard_parity, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_hsdp_save_load_state_dict, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_hsdp_sync_module_state, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_invalid_pg_specification_raises, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_raises_manual_wrap_hybrid_shard_when_none_policy 2024-11-01T17:16:01.1867860Z 2024-11-01T17:16:01.1868478Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 ... [2024-11-01 17:16:01.186188] 2024-11-01T17:16:01.1869655Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:16:01.1871555Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:16:01.186713] 2024-11-01T17:17:08.3516624Z 2024-11-01T17:17:08.3520472Z distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_sharded_grad_scaler_1.1_52e9796290d64be8_.log 2024-11-01T17:17:08.3540720Z Running 20 items in this shard: test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_grad_scaling, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_inf_gradients_skip_optim_step, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_scaling_unscaling_sparse, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_sharded_grad_scaler_found_inf 2024-11-01T17:17:08.3559176Z 2024-11-01T17:17:08.3559799Z Running distributed/optim/test_zero_redundancy_optimizer 1/1 ... [2024-11-01 17:17:08.352108] 2024-11-01T17:17:08.3560530Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:17:08.3562431Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/optim/test_zero_redundancy_optimizer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:17:08.352642] 2024-11-01T17:18:53.3279401Z 2024-11-01T17:18:53.3282986Z distributed/optim/test_zero_redundancy_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_zero_redundancy_optimizer_1.1_ea5bd3325b6a8bf4_.log 2024-11-01T17:18:53.3348541Z Running 42 items in this shard: test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_constructor, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_lr_scheduler, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_same_dense_param_type, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_state_dict, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_with_extra_inner_key, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_with_kwargs, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_without_closure, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_zero_grad, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_add_param_group, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_collect_shards, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_lr_scheduler, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_multiple_param_groups, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_nondefault_process_group, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_sharding, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_step, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_step_with_closure, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_join_cpu, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_join_gpu, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_model_parallel_parameters_as_bucket_view_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_model_parallel_parameters_as_bucket_view_True 2024-11-01T17:18:53.3388676Z 2024-11-01T17:18:53.3389226Z Running distributed/_shard/test_sharder 1/1 ... [2024-11-01 17:18:53.329018] 2024-11-01T17:18:53.3389870Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:18:53.3391688Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:18:53.329617] 2024-11-01T17:18:57.1075387Z 2024-11-01T17:18:57.1077483Z distributed/_shard/test_sharder 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.test_sharder_1.1_64eff4f11e4145c6_.log 2024-11-01T17:18:57.1079647Z Running 2 items in this shard: test/distributed/_shard/test_sharder.py::TestCustomSharder::test_custom_sharder, test/distributed/_shard/test_sharder.py::TestCustomSharder::test_custom_sharder_errors 2024-11-01T17:18:57.1080827Z 2024-11-01T17:18:57.1084707Z Running distributed/_shard/sharding_plan/test_sharding_plan 1/1 ... [2024-11-01 17:18:57.108224] 2024-11-01T17:18:57.1085522Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:18:57.1090032Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:18:57.108705] 2024-11-01T17:19:00.8350142Z 2024-11-01T17:19:00.8352379Z distributed/_shard/sharding_plan/test_sharding_plan 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharding_plan.test_sharding_plan_1.1_86b7da3c28a21120_.log 2024-11-01T17:19:00.8355924Z Running 3 items in this shard: test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_custom_sharding_planner, test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_shard_module_sub_process_group, test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_sharding_plan_errors 2024-11-01T17:19:00.8358360Z 2024-11-01T17:19:00.8359243Z Running distributed/fsdp/test_fsdp_use_orig_params 1/1 ... [2024-11-01 17:19:00.835575] 2024-11-01T17:19:00.8359946Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:19:00.8363930Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_use_orig_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:19:00.836089] 2024-11-01T17:20:38.2037643Z 2024-11-01T17:20:38.2040301Z distributed/fsdp/test_fsdp_use_orig_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_use_orig_params_1.1_7b87f24658f71cd4_.log 2024-11-01T17:20:38.2070414Z Running 25 items in this shard: test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_full_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_no_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_shard_grad_op, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_full_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_no_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_shard_grad_op, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_trainability, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_fsdp_compile, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_multiple_optimizers, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_multiple_forward_offload_params_False, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_multiple_forward_offload_params_True, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_summon_between_two_forwards_offload_params_False, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_summon_between_two_forwards_offload_params_True, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsParamAccess::test_access_params_after_forward, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_grad_writeback, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_no_reshard_and_mixed_precision, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_param_writeback, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_writeback_between_fwd_and_bwd_for_no_reshard_raises, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_writeback_shape_mismatch, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsFQNs::test_named_parameters_in_forward, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsNoSync::test_no_sync_correctness, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsNoSync::test_no_sync_mixed_precision, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsInit::test_non_uniform_requires_grad, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestMultiTensorApply::test_multi_tensor_apply_size0_tensors_cpu, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestMultiTensorApply::test_multi_tensor_apply_size0_tensors_cuda 2024-11-01T17:20:38.2089845Z 2024-11-01T17:20:38.2090688Z Running distributed/fsdp/test_fsdp_multiple_forward 1/1 ... [2024-11-01 17:20:38.205381] 2024-11-01T17:20:38.2091401Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:20:38.2093276Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:20:38.205954] 2024-11-01T17:20:45.9406840Z 2024-11-01T17:20:45.9408797Z distributed/fsdp/test_fsdp_multiple_forward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_multiple_forward_1.1_0fe75bc31284f6e0_.log 2024-11-01T17:20:45.9410791Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_multiple_forward.py::TestMultiForward::test_multi_forward 2024-11-01T17:20:45.9411628Z 2024-11-01T17:20:45.9414877Z Running distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 1/1 ... [2024-11-01 17:20:45.941229] 2024-11-01T17:20:45.9415783Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:20:45.9420462Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:20:45.941711] 2024-11-01T17:20:57.5319734Z 2024-11-01T17:20:57.5322149Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_optim_checkpoint_1.1_df75bebe9677188f_.log 2024-11-01T17:20:57.5325354Z Running 2 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint.py::TestOptimStateCheckpointing::test_optim_state_dict_save_load, test/distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint.py::TestOptimStateCheckpointing::test_optim_state_dict_submodule_fully_shard 2024-11-01T17:20:57.5327422Z 2024-11-01T17:20:57.5328628Z Running distributed/rpc/test_faulty_agent 1/1 ... [2024-11-01 17:20:57.532633] 2024-11-01T17:20:57.5329315Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:20:57.5334814Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/rpc/test_faulty_agent.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:20:57.533139] 2024-11-01T17:27:49.2324191Z 2024-11-01T17:27:49.2326576Z distributed/rpc/test_faulty_agent 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.test_faulty_agent_1.1_f91759641ecc5142_.log 2024-11-01T17:27:49.2350793Z Running 28 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_builtin_remote_message_dropped_timeout, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_builtin_remote_message_dropped_timeout_to_self, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_check_failed_messages, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_custom_faulty_messages, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_custom_messages_to_delay, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_no_faulty_messages, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_builtin_delay_timeout, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_builtin_delay_timeout_to_self, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_dropped_pickle, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_dropped_pickle_to_self, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_script_delay_timeout, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_script_delay_timeout_to_self, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_rpc_builtin_timeout, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_rpc_script_timeout, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_rref_to_here_timeout, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_udf_remote_message_delay_timeout, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_udf_remote_message_delay_timeout_to_self, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_udf_remote_message_dropped_timeout, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_udf_remote_message_dropped_timeout_to_self, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_verify_backend_options, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentDistAutogradTest::test_context_cleanup_tensor_with_grad, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentDistAutogradTest::test_verify_backend_options, test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_remote_timeout_to_here_in_jit, test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_rref_timeout_pickle_in_jit, test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_rref_timeout_pickle_script_func, test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_rref_to_here_timeout_in_jit, test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_timeout_in_python, test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_timeout_in_torchscript_function 2024-11-01T17:27:49.2369275Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_builtin_remote_message_dropped_timeout 2024-11-01T17:27:49.2370948Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_builtin_remote_message_dropped_timeout_to_self 2024-11-01T17:27:49.2372515Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_check_failed_messages 2024-11-01T17:27:49.2373933Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_custom_faulty_messages 2024-11-01T17:27:49.2375376Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_custom_messages_to_delay 2024-11-01T17:27:49.2376798Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_no_faulty_messages 2024-11-01T17:27:49.2378259Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_builtin_delay_timeout 2024-11-01T17:27:49.2379880Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_builtin_delay_timeout_to_self 2024-11-01T17:27:49.2381478Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_dropped_pickle 2024-11-01T17:27:49.2383023Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_dropped_pickle_to_self 2024-11-01T17:27:49.2384749Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_script_delay_timeout 2024-11-01T17:27:49.2386370Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_script_delay_timeout_to_self 2024-11-01T17:27:49.2387902Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_rpc_builtin_timeout 2024-11-01T17:27:49.2389299Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_rpc_script_timeout 2024-11-01T17:27:49.2390781Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_rref_to_here_timeout 2024-11-01T17:27:49.2392241Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_udf_remote_message_delay_timeout 2024-11-01T17:27:49.2393913Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_udf_remote_message_delay_timeout_to_self 2024-11-01T17:27:49.2395521Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_udf_remote_message_dropped_timeout 2024-11-01T17:27:49.2397118Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_udf_remote_message_dropped_timeout_to_self 2024-11-01T17:27:49.2398670Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_verify_backend_options 2024-11-01T17:27:49.2400225Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentDistAutogradTest::test_context_cleanup_tensor_with_grad 2024-11-01T17:27:49.2401834Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentDistAutogradTest::test_verify_backend_options 2024-11-01T17:27:49.2403391Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_remote_timeout_to_here_in_jit 2024-11-01T17:27:49.2404934Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_rref_timeout_pickle_in_jit 2024-11-01T17:27:49.2406471Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_rref_timeout_pickle_script_func 2024-11-01T17:27:49.2408027Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_rref_to_here_timeout_in_jit 2024-11-01T17:27:49.2409508Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_timeout_in_python 2024-11-01T17:27:49.2411014Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_timeout_in_torchscript_function 2024-11-01T17:27:49.2411891Z 2024-11-01T17:27:49.2412426Z Running distributed/fsdp/test_fsdp_comm 1/1 ... [2024-11-01 17:27:49.234436] 2024-11-01T17:27:49.2413063Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:27:49.2414864Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:27:49.234940] 2024-11-01T17:28:23.1553415Z 2024-11-01T17:28:23.1555267Z distributed/fsdp/test_fsdp_comm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_comm_1.1_57cda1d7dabef5f9_.log 2024-11-01T17:28:23.1564992Z Running 10 items in this shard: test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_False_use_no_sync_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_False_use_no_sync_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_False_use_no_sync_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_False_use_no_sync_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_True_use_no_sync_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_True_use_no_sync_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_True_use_no_sync_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm.py::TestCommunication::test_communication_nested_model_True_use_no_sync_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshard::test_unshard_async_use_orig_params_False, test/distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshard::test_unshard_async_use_orig_params_True 2024-11-01T17:28:23.1573203Z 2024-11-01T17:28:23.1573833Z Running distributed/_shard/sharded_optim/test_sharded_optim 1/1 ... [2024-11-01 17:28:23.155958] 2024-11-01T17:28:23.1574588Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:28:23.1576513Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:28:23.156457] 2024-11-01T17:28:26.8826937Z 2024-11-01T17:28:26.8829001Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_optim.test_sharded_optim_1.1_5617a5a5f10fb30d_.log 2024-11-01T17:28:26.8832455Z Running 2 items in this shard: test/distributed/_shard/sharded_optim/test_sharded_optim.py::TestShardedOptimizer::test_named_params_with_sharded_tensor, test/distributed/_shard/sharded_optim/test_sharded_optim.py::TestShardedOptimizer::test_sharded_optim 2024-11-01T17:28:26.8834078Z 2024-11-01T17:28:26.8836827Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 ... [2024-11-01 17:28:26.883390] 2024-11-01T17:28:26.8837584Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:28:26.8842471Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:28:26.883916] 2024-11-01T17:28:30.6602842Z 2024-11-01T17:28:30.6604789Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_tensor_ops_1.1_e494b3f036d6a6be_.log 2024-11-01T17:28:30.6608815Z Running 5 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_clone, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_deep_copy, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_detach, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_inplace_copy, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_set_requires_grad 2024-11-01T17:28:30.6611995Z 2024-11-01T17:28:30.6612463Z Running distributed/rpc/test_tensorpipe_agent 1/4 ... [2024-11-01 17:28:30.660925] 2024-11-01T17:28:30.6613147Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:28:30.6617064Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/rpc/test_tensorpipe_agent.py', '--shard-id=1', '--num-shards=4', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:28:30.661415] 2024-11-01T17:42:17.6049661Z 2024-11-01T17:42:17.6054182Z distributed/rpc/test_tensorpipe_agent 1/4 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.test_tensorpipe_agent_1.4_a409cb129b840c80_.log 2024-11-01T17:42:17.6124661Z Running 85 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_add_done_callback, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_class_rref_proxy, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_class_rref_proxy_async, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_chained, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_chained_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_wrong_return_type, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_call_method_on_rref, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_chain, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_multi, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_with_ret, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_custom_exception_throw_during_reconstruction, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_debug_info, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_future_nested_callback, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_int_callee, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_local_rref_no_fork, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_multi_layer_nested_async_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_multi_py_udf_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_rpc_key_names, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_async_rpc_builtin, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_async_rpc_builtin_single_threaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_async_rpc_udf, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_remote_udf, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_remote_udf_single_threaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_remote_rpc_single_threaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_sync_rpc_builtin, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_class_static_method, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_register_rpc_backend_and_set_and_start_rpc_backend, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_remote_same_worker, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_barrier_partial_subset, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_return_rref, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_forward_chain, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_proxy_class, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_proxy_non_exist, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_with_error_non_blocking, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_self_py_udf_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_self_remote_rref_as_rpc_arg, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_stress_heavy_rpc_torchscript, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_exit_early_builtin, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_exit_early_script_function, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wrong_types, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_accumulate_grads, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_autograd_engine_error, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_different_tensor_dims, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_rref_nested, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_simple_python_udf, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_unused_send_function, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_without_context, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_clean_context_during_backward, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_context_cleanup_tensor_with_grad, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_error_in_context, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_grad_copy_sparse_indices_extra_ref, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_grad_only_on_return_value_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_py_nested_call, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_py_nested_remote_call, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_no_grad_copy, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_no_graph_with_tensors_not_require_grad, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_rpc_complex_args, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_thread_local_context_id, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_all_kwargs_are_populated_by_defaults, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_function_wrong_return_type, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_callback_with_exception, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_create_local_script_class_rref_in_py, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_create_local_script_module_rref_in_py, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_kwargs_not_passed, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_remote_script_module, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_return_local_script_class_rref_in_py_and_use_in_script, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_rref_python_annotation, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_torchscript_function, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_unexepected_kwarg_is_specified, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_user_rrefs_confirmed_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_forward_async, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeThreeWorkersRemoteModuleTest::test_create_remote_module_from_module_rref, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDdpComparisonTest::test_ddp_comparison, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_dynamic_rpc_existing_rank_can_communicate_with_new_rank, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_infer_backend_from_options, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_mismatched_type_for_options, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_multi_builtin_remote_ret_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_multi_rpc_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_self_remote_rref_as_remote_arg_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_self_remote_rref_as_self_rpc_arg_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_send_to_rank_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_stress_heavy_rpc_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_different_dtypes_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_graph_for_py_nested_call_itself_sparse 2024-11-01T17:42:17.6175823Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_add_done_callback 2024-11-01T17:42:17.6177218Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_class_rref_proxy 2024-11-01T17:42:17.6178632Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_class_rref_proxy_async 2024-11-01T17:42:17.6180053Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_chained 2024-11-01T17:42:17.6181482Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_chained_remote 2024-11-01T17:42:17.6182961Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_wrong_return_type 2024-11-01T17:42:17.6184391Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_call_method_on_rref 2024-11-01T17:42:17.6185721Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_chain 2024-11-01T17:42:17.6187025Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_multi 2024-11-01T17:42:17.6188341Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_with_ret 2024-11-01T17:42:17.6189812Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_custom_exception_throw_during_reconstruction 2024-11-01T17:42:17.6191261Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_debug_info 2024-11-01T17:42:17.6192577Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_future_nested_callback 2024-11-01T17:42:17.6193964Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_int_callee 2024-11-01T17:42:17.6195274Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_local_rref_no_fork 2024-11-01T17:42:17.6196679Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_multi_layer_nested_async_rpc 2024-11-01T17:42:17.6198085Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_multi_py_udf_remote 2024-11-01T17:42:17.6199460Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_rpc_key_names 2024-11-01T17:42:17.6200898Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_async_rpc_builtin 2024-11-01T17:42:17.6202467Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_async_rpc_builtin_single_threaded 2024-11-01T17:42:17.6204016Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_async_rpc_udf 2024-11-01T17:42:17.6205591Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_remote_udf 2024-11-01T17:42:17.6207094Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_remote_udf_single_threaded 2024-11-01T17:42:17.6208713Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_remote_rpc_single_threaded 2024-11-01T17:42:17.6210385Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_sync_rpc_builtin 2024-11-01T17:42:17.6211821Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_class_static_method 2024-11-01T17:42:17.6213351Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_register_rpc_backend_and_set_and_start_rpc_backend 2024-11-01T17:42:17.6214878Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_remote_same_worker 2024-11-01T17:42:17.6216260Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_barrier_partial_subset 2024-11-01T17:42:17.6217631Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_return_rref 2024-11-01T17:42:17.6218958Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_forward_chain 2024-11-01T17:42:17.6220287Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_proxy_class 2024-11-01T17:42:17.6221619Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_proxy_non_exist 2024-11-01T17:42:17.6223049Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_with_error_non_blocking 2024-11-01T17:42:17.6224458Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_self_py_udf_remote 2024-11-01T17:42:17.6225847Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_self_remote_rref_as_rpc_arg 2024-11-01T17:42:17.6227299Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_stress_heavy_rpc_torchscript 2024-11-01T17:42:17.6228748Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_exit_early_builtin 2024-11-01T17:42:17.6230225Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_exit_early_script_function 2024-11-01T17:42:17.6231623Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wrong_types 2024-11-01T17:42:17.6233113Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_accumulate_grads 2024-11-01T17:42:17.6234700Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_autograd_engine_error 2024-11-01T17:42:17.6236295Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_different_tensor_dims 2024-11-01T17:42:17.6237838Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_rref_nested 2024-11-01T17:42:17.6239353Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_simple_python_udf 2024-11-01T17:42:17.6240914Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_unused_send_function 2024-11-01T17:42:17.6242535Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_without_context 2024-11-01T17:42:17.6244092Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_clean_context_during_backward 2024-11-01T17:42:17.6245699Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_context_cleanup_tensor_with_grad 2024-11-01T17:42:17.6247284Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_error_in_context 2024-11-01T17:42:17.6248822Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_grad_copy_sparse_indices_extra_ref 2024-11-01T17:42:17.6250428Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_grad_only_on_return_value_remote 2024-11-01T17:42:17.6251996Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_py_nested_call 2024-11-01T17:42:17.6253560Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_py_nested_remote_call 2024-11-01T17:42:17.6255061Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_no_grad_copy 2024-11-01T17:42:17.6256603Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_no_graph_with_tensors_not_require_grad 2024-11-01T17:42:17.6258152Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_rpc_complex_args 2024-11-01T17:42:17.6259804Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_thread_local_context_id 2024-11-01T17:42:17.6261360Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_all_kwargs_are_populated_by_defaults 2024-11-01T17:42:17.6262917Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_function_wrong_return_type 2024-11-01T17:42:17.6264389Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_callback_with_exception 2024-11-01T17:42:17.6265893Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_create_local_script_class_rref_in_py 2024-11-01T17:42:17.6267479Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_create_local_script_module_rref_in_py 2024-11-01T17:42:17.6268953Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_kwargs_not_passed 2024-11-01T17:42:17.6270332Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_remote_script_module 2024-11-01T17:42:17.6271920Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_return_local_script_class_rref_in_py_and_use_in_script 2024-11-01T17:42:17.6273612Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_rref_python_annotation 2024-11-01T17:42:17.6275029Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_torchscript_function 2024-11-01T17:42:17.6276485Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_unexepected_kwarg_is_specified 2024-11-01T17:42:17.6277996Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_user_rrefs_confirmed_remote 2024-11-01T17:42:17.6279444Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_forward_async 2024-11-01T17:42:17.6281245Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire 2024-11-01T17:42:17.6283153Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeThreeWorkersRemoteModuleTest::test_create_remote_module_from_module_rref 2024-11-01T17:42:17.6284793Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDdpComparisonTest::test_ddp_comparison 2024-11-01T17:42:17.6286554Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_dynamic_rpc_existing_rank_can_communicate_with_new_rank 2024-11-01T17:42:17.6288334Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_infer_backend_from_options 2024-11-01T17:42:17.6289984Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_mismatched_type_for_options 2024-11-01T17:42:17.6291655Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_multi_builtin_remote_ret_sparse 2024-11-01T17:42:17.6293251Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_multi_rpc_sparse 2024-11-01T17:42:17.6294901Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_self_remote_rref_as_remote_arg_sparse 2024-11-01T17:42:17.6296665Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_self_remote_rref_as_self_rpc_arg_sparse 2024-11-01T17:42:17.6298329Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_send_to_rank_sparse 2024-11-01T17:42:17.6299908Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_stress_heavy_rpc_sparse 2024-11-01T17:42:17.6301633Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_different_dtypes_sparse 2024-11-01T17:42:17.6303498Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_graph_for_py_nested_call_itself_sparse 2024-11-01T17:42:17.6304530Z 2024-11-01T17:42:17.6305120Z Running distributed/rpc/test_tensorpipe_agent 3/4 ... [2024-11-01 17:42:17.610178] 2024-11-01T17:42:17.6305785Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:42:17.6307684Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/rpc/test_tensorpipe_agent.py', '--shard-id=3', '--num-shards=4', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:42:17.610675] 2024-11-01T18:00:59.8956359Z 2024-11-01T18:00:59.8958332Z distributed/rpc/test_tensorpipe_agent 3/4 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.test_tensorpipe_agent_3.4_a7dd6ef3df8f50db_.log 2024-11-01T18:00:59.9056936Z Running 117 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_add, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_add_with_id, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_multi_chained, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_multi_fanout, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_multi_fanout_async, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_simple, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_with_future_ctor, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_record_function_cbs_jit_call, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_build_rpc_profiling_key, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_builtin_remote_ret, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_in_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_none, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_with_error, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_deadlock, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_disable_gil_profiling, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_dist_init_decorator, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_duplicate_name_2, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_future_in_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_future_wait_twice, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_ignore_rref_leak, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_init_rpc_then_pg, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_local_shutdown_with_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_my_parameter_server, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_nested_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_non_garbage_collected_user_rref_due_to_local_circular_dependency, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_nonzero, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_remote_events_profiled_single_threaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_autograd_context, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_remote_builtin_single_threaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_async_rpc_single_threaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_remote_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_sync_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_class_constructor, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_class_instance_method, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_multi_async_call, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_rpc_rref_args, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_rref_args_user_share, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_tensors, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_udf_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_profiling_remote_record_function, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_get_future, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_non_blocking, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_slow_init, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_with_error_blocking, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_user_rrefs_confirmed, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_user_rrefs_confirmed_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_raise_in_body, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_with_exception, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_with_partial_exception, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_workers_twice_dense, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_world_size_one, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_autograd_context, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_different_dtypes, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_invalid_args, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_multiple_roots, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_multiple_round_trips, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_no_grad_on_tensor, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_node_failure, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_node_failure_python_udf, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_python_udf_error, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_simple_script_call, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_verify_hooks, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backwards_nested_python_udf, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_context_cleanup_nested_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_context_cleanup_tensor_no_grad, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_builtin_remote_call, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_py_nested_remote_call_itself, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_python_call, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_mixed_requires_grad, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_nested_backward_accumulate_grads, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_no_grad_copy_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_post_hooks, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_trainer_ps, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistOptimizerTest::test_dist_optim_exception, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistOptimizerTest::test_dist_optim_exception_on_constructor, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_add_done_callback, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_args_and_kwargs_contain_different_types, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_args_kwargs_are_neither_passed, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_function_remote_multi, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_function_simple, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_call_fork_in_jit_with_profiling, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_call_rpc_with_profiling, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_callback_simple, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_future_python_annotation, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_local_rref_local_value, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_more_than_needed_args_are_specified, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_my_script_module_with_rrefs, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_remote_script_throw, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_return_local_script_module_rref_in_py_and_use_in_script, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_some_kwargs_are_populated_by_defaults, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitDistAutogradTest::test_restore_context_after_swtich_to_jit_thread, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_bad_module, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_forward_async_script, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_train_eval, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_unsupported_methods, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeReinforcementLearningRpcTest::test_rl_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_builtin_remote_ret_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_builtin_remote_self_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_multi_py_udf_remote_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_nested_rpc_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_rref_get_type_timeout_blocking, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_rref_get_type_timeout_non_blocking, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_self_py_udf_remote_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_self_remote_rref_as_self_remote_arg_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_tensorpipe_options_throw_on_timedelta_timeout, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_wait_all_workers_twice_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_world_size_one_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_rref_multi_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_simple_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backwards_nested_python_udf_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_context_cleanup_nested_rpc_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_context_cleanup_tensor_no_grad_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_embedding_bag_with_no_grad_tensors, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_multiple_backward_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_no_graph_with_tensors_not_require_grad_remote_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_remote_complex_args_sparse 2024-11-01T18:00:59.9125569Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_add 2024-11-01T18:00:59.9126816Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_add_with_id 2024-11-01T18:00:59.9128194Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_multi_chained 2024-11-01T18:00:59.9129650Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_multi_fanout 2024-11-01T18:00:59.9131117Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_multi_fanout_async 2024-11-01T18:00:59.9132557Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_simple 2024-11-01T18:00:59.9134097Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_with_future_ctor 2024-11-01T18:00:59.9135609Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_record_function_cbs_jit_call 2024-11-01T18:00:59.9137075Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_build_rpc_profiling_key 2024-11-01T18:00:59.9138538Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_builtin_remote_ret 2024-11-01T18:00:59.9139873Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_in_rpc 2024-11-01T18:00:59.9141179Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_none 2024-11-01T18:00:59.9142493Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_with_error 2024-11-01T18:00:59.9143792Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_deadlock 2024-11-01T18:00:59.9145096Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_disable_gil_profiling 2024-11-01T18:00:59.9146463Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_dist_init_decorator 2024-11-01T18:00:59.9147812Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_duplicate_name_2 2024-11-01T18:00:59.9149126Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_future_in_rpc 2024-11-01T18:00:59.9150431Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_future_wait_twice 2024-11-01T18:00:59.9151743Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_ignore_rref_leak 2024-11-01T18:00:59.9153125Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_init_rpc_then_pg 2024-11-01T18:00:59.9154493Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_local_shutdown_with_rpc 2024-11-01T18:00:59.9155874Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_my_parameter_server 2024-11-01T18:00:59.9157199Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_nested_remote 2024-11-01T18:00:59.9158898Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_non_garbage_collected_user_rref_due_to_local_circular_dependency 2024-11-01T18:00:59.9160432Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_nonzero 2024-11-01T18:00:59.9161888Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_remote_events_profiled_single_threaded 2024-11-01T18:00:59.9163449Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_autograd_context 2024-11-01T18:00:59.9165012Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_remote_builtin_single_threaded 2024-11-01T18:00:59.9166656Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_async_rpc_single_threaded 2024-11-01T18:00:59.9168229Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_remote_rpc 2024-11-01T18:00:59.9169709Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_sync_rpc 2024-11-01T18:00:59.9171235Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_class_constructor 2024-11-01T18:00:59.9172632Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_class_instance_method 2024-11-01T18:00:59.9189116Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_multi_async_call 2024-11-01T18:00:59.9190777Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_rpc_rref_args 2024-11-01T18:00:59.9192153Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_rref_args_user_share 2024-11-01T18:00:59.9193577Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_tensors 2024-11-01T18:00:59.9194863Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_udf_remote 2024-11-01T18:00:59.9196279Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_profiling_remote_record_function 2024-11-01T18:00:59.9197708Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_get_future 2024-11-01T18:00:59.9199059Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_non_blocking 2024-11-01T18:00:59.9200429Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_slow_init 2024-11-01T18:00:59.9201837Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_with_error_blocking 2024-11-01T18:00:59.9203245Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_user_rrefs_confirmed 2024-11-01T18:00:59.9204652Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_user_rrefs_confirmed_remote 2024-11-01T18:00:59.9205989Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all 2024-11-01T18:00:59.9207286Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_raise_in_body 2024-11-01T18:00:59.9208672Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_with_exception 2024-11-01T18:00:59.9210129Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_with_partial_exception 2024-11-01T18:00:59.9211592Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_workers_twice_dense 2024-11-01T18:00:59.9213054Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_world_size_one 2024-11-01T18:00:59.9214433Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_autograd_context 2024-11-01T18:00:59.9215924Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_different_dtypes 2024-11-01T18:00:59.9217440Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_invalid_args 2024-11-01T18:00:59.9218942Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_multiple_roots 2024-11-01T18:00:59.9220504Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_multiple_round_trips 2024-11-01T18:00:59.9222075Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_no_grad_on_tensor 2024-11-01T18:00:59.9223691Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_node_failure 2024-11-01T18:00:59.9225256Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_node_failure_python_udf 2024-11-01T18:00:59.9226846Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_python_udf_error 2024-11-01T18:00:59.9228472Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_simple_script_call 2024-11-01T18:00:59.9230004Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_verify_hooks 2024-11-01T18:00:59.9231524Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backwards_nested_python_udf 2024-11-01T18:00:59.9233158Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_context_cleanup_nested_rpc 2024-11-01T18:00:59.9234735Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_context_cleanup_tensor_no_grad 2024-11-01T18:00:59.9236325Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_builtin_remote_call 2024-11-01T18:00:59.9237962Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_py_nested_remote_call_itself 2024-11-01T18:00:59.9239550Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_python_call 2024-11-01T18:00:59.9241033Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_mixed_requires_grad 2024-11-01T18:00:59.9242586Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_nested_backward_accumulate_grads 2024-11-01T18:00:59.9244132Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_no_grad_copy_sparse 2024-11-01T18:00:59.9245536Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_post_hooks 2024-11-01T18:00:59.9246912Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_trainer_ps 2024-11-01T18:00:59.9248358Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistOptimizerTest::test_dist_optim_exception 2024-11-01T18:00:59.9249952Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistOptimizerTest::test_dist_optim_exception_on_constructor 2024-11-01T18:00:59.9251464Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_add_done_callback 2024-11-01T18:00:59.9252959Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_args_and_kwargs_contain_different_types 2024-11-01T18:00:59.9254520Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_args_kwargs_are_neither_passed 2024-11-01T18:00:59.9256020Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_function_remote_multi 2024-11-01T18:00:59.9257468Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_function_simple 2024-11-01T18:00:59.9259119Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_call_fork_in_jit_with_profiling 2024-11-01T18:00:59.9260607Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_call_rpc_with_profiling 2024-11-01T18:00:59.9262131Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_callback_simple 2024-11-01T18:00:59.9263553Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_future_python_annotation 2024-11-01T18:00:59.9264995Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_local_rref_local_value 2024-11-01T18:00:59.9266572Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_more_than_needed_args_are_specified 2024-11-01T18:00:59.9268096Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_my_script_module_with_rrefs 2024-11-01T18:00:59.9269530Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_remote_script_throw 2024-11-01T18:00:59.9271109Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_return_local_script_module_rref_in_py_and_use_in_script 2024-11-01T18:00:59.9272878Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_some_kwargs_are_populated_by_defaults 2024-11-01T18:00:59.9274562Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitDistAutogradTest::test_restore_context_after_swtich_to_jit_thread 2024-11-01T18:00:59.9276133Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_bad_module 2024-11-01T18:00:59.9277569Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_forward_async_script 2024-11-01T18:00:59.9278997Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_train_eval 2024-11-01T18:00:59.9280428Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_unsupported_methods 2024-11-01T18:00:59.9281925Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeReinforcementLearningRpcTest::test_rl_rpc 2024-11-01T18:00:59.9283492Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_builtin_remote_ret_sparse 2024-11-01T18:00:59.9285126Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_builtin_remote_self_sparse 2024-11-01T18:00:59.9286778Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_multi_py_udf_remote_sparse 2024-11-01T18:00:59.9288372Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_nested_rpc_sparse 2024-11-01T18:00:59.9289987Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_rref_get_type_timeout_blocking 2024-11-01T18:00:59.9291688Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_rref_get_type_timeout_non_blocking 2024-11-01T18:00:59.9293359Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_self_py_udf_remote_sparse 2024-11-01T18:00:59.9295073Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_self_remote_rref_as_self_remote_arg_sparse 2024-11-01T18:00:59.9296904Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_tensorpipe_options_throw_on_timedelta_timeout 2024-11-01T18:00:59.9298645Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_wait_all_workers_twice_sparse 2024-11-01T18:00:59.9300351Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_world_size_one_sparse 2024-11-01T18:00:59.9302033Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_rref_multi_sparse 2024-11-01T18:00:59.9303789Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_simple_sparse 2024-11-01T18:00:59.9305640Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backwards_nested_python_udf_sparse 2024-11-01T18:00:59.9307494Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_context_cleanup_nested_rpc_sparse 2024-11-01T18:00:59.9309353Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_context_cleanup_tensor_no_grad_sparse 2024-11-01T18:00:59.9311224Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_embedding_bag_with_no_grad_tensors 2024-11-01T18:00:59.9313067Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_multiple_backward_sparse 2024-11-01T18:00:59.9315059Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_no_graph_with_tensors_not_require_grad_remote_sparse 2024-11-01T18:00:59.9316955Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_remote_complex_args_sparse 2024-11-01T18:00:59.9317932Z 2024-11-01T18:00:59.9318653Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 ... [2024-11-01 18:00:59.903594] 2024-11-01T18:00:59.9319403Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:00:59.9321338Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:00:59.904073] 2024-11-01T18:01:03.6801791Z 2024-11-01T18:01:03.6803737Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_binary_cmp_1.1_fadc226d4bc2949b_.log 2024-11-01T18:01:03.6807810Z Running 4 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_allclose, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_allclose_tensor_specs, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_equal, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_equal_tensor_specs 2024-11-01T18:01:03.6810892Z 2024-11-01T18:01:03.6811411Z Running distributed/_shard/sharded_tensor/ops/test_init 1/1 ... [2024-11-01 18:01:03.680808] 2024-11-01T18:01:03.6812126Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:01:03.6815971Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:01:03.681301] 2024-11-01T18:01:07.4577153Z 2024-11-01T18:01:07.4579030Z distributed/_shard/sharded_tensor/ops/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_init_1.1_de5d3e61fb494ad7_.log 2024-11-01T18:01:07.4582680Z Running 3 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_kaiming_uniform, test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_normal, test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_uniform 2024-11-01T18:01:07.4585173Z 2024-11-01T18:01:07.4585687Z Running distributed/fsdp/test_fsdp_mixed_precision 1/1 ... [2024-11-01 18:01:07.458327] 2024-11-01T18:01:07.4586528Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:01:07.4591458Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:01:07.458848] 2024-11-01T18:04:47.0512883Z 2024-11-01T18:04:47.0515000Z distributed/fsdp/test_fsdp_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_mixed_precision_1.1_200f1d5808a7ac86_.log 2024-11-01T18:04:47.0565123Z Running 57 items in this shard: test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_eval_root_cast_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval_buffers, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval_comm, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_grads_reduced_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_input_grads_with_param_mixed_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_no_reshard_after_forward, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_resnet, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_batchnorm_convert_sync_bn_False, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_batchnorm_convert_sync_bn_True, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_default, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_only_params_and_bufs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_params_and_reduce_diff, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_reduce, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_grads_reduced_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_mixed_precision_e2e_full_shard, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_mixed_precision_no_reshard_after_forward, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionIgnoredModules::test_mixed_precision_with_ignored_module, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule_skip_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule_skip_inputs_error, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_different_precisions, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_different_precisions_error, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_external_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPTrainEval::test_train_ema_eval_flow 2024-11-01T18:04:47.0611005Z 2024-11-01T18:04:47.0611493Z Running distributed/test_launcher 1/1 ... [2024-11-01 18:04:47.052137] 2024-11-01T18:04:47.0612100Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:04:47.0613868Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/test_launcher.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:04:47.052643] 2024-11-01T18:04:52.2809818Z 2024-11-01T18:04:52.2811843Z distributed/test_launcher 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_launcher_1.1_9b95a6b6e63b9272_.log 2024-11-01T18:04:52.2813704Z Running 1 items in this shard: test/distributed/test_launcher.py::TestDistributedLaunch::test_launch_user_script 2024-11-01T18:04:52.2814441Z 2024-11-01T18:04:52.2819096Z Running distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 ... [2024-11-01 18:04:52.281639] 2024-11-01T18:04:52.2819852Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:04:52.2824735Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:04:52.282140] 2024-11-01T18:05:05.3266477Z 2024-11-01T18:05:05.3268509Z distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_1.1_89e909d52e77a2f2_.log 2024-11-01T18:05:05.3316600Z Running 69 items in this shard: test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorMetadata::test_serialize_and_deserialize, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestCreateTensorFromParams::test_empty, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardParameter::test_shard_parameter, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardParameter::test_shard_parameter_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardTensor::test_shard_tensor, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardTensor::test_shard_tensor_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardTensor::test_shard_tensor_with_empty_shard, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestModuleHookApi::test_collect_local_shard, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestModuleHookApi::test_reshard_output, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestLocalTensor::test_local_tensor, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestLocalTensor::test_local_tensor_error, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_cleanup, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_complete_world_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_like, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_full, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_ones, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_rand, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_zeros, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_gather_even, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_gather_uneven, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_insufficient_sharding_dims, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_invalid_pg_rpc_ranks, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_invalid_sharding, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_load_state_dict_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_multiple_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_partial_world_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_sharded_tensor_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_sharded_tensor_sizes, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_sharding_columns, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_state_dict, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_state_dict_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_state_dict_no_sharded_tensors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_create_sharded_tensor_with_ones, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_gather_even, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_gather_uneven, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_grid_sharding, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_multiple_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_partial_world_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_device, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_to_cpu, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_to_cuda, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_to_test, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_uneven_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_with_rpc_names, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalTensor::test_init_from_local_tensor, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalTensor::test_init_from_local_tensor_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_and_global_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_and_global_metadata_invalid_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_pin_memory, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_property_cross_ranks, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_shards_gaps, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_shards_overlap, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_st_base_init_from_local_shards_and_global_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorCustomOps::test_custom_op, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorCustomOps::test_custom_op_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorCustomOps::test_custom_op_override, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardMetadata::test_create_shard_with_no_placement, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardMetadata::test_shard_metadata_init, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorSubGroupInit::test_sub_process_group_placement_validation, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorSubGroupInit::test_sub_process_group_sharded_tensor_init, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestCreateTensorNoProcessGroupMode::test_init_from_local_shards_and_global_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestCreateTensorNoProcessGroupMode::test_non_contiguous_local_shards 2024-11-01T18:05:05.3362555Z 2024-11-01T18:05:05.3363063Z Running distributed/test_c10d_spawn_gloo 1/1 ... [2024-11-01 18:05:05.327395] 2024-11-01T18:05:05.3363711Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:05:05.3365598Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:05:05.327889] 2024-11-01T18:06:26.0062792Z 2024-11-01T18:06:26.0064813Z distributed/test_c10d_spawn_gloo 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_gloo_1.1_a7f6edfa8b5d96a5_.log 2024-11-01T18:06:26.0075461Z Running 11 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cpu, test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cuda, test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_rnn, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all_single, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_allreduce, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_broadcast, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_gather, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_reduce, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_scatter 2024-11-01T18:06:26.0082880Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cpu 2024-11-01T18:06:26.0084336Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cuda 2024-11-01T18:06:26.0085784Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_rnn 2024-11-01T18:06:26.0087196Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather 2024-11-01T18:06:26.0088555Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all 2024-11-01T18:06:26.0089953Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all_single 2024-11-01T18:06:26.0091330Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_allreduce 2024-11-01T18:06:26.0092683Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_broadcast 2024-11-01T18:06:26.0094007Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_gather 2024-11-01T18:06:26.0095323Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_reduce 2024-11-01T18:06:26.0096647Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_scatter 2024-11-01T18:06:26.0097389Z 2024-11-01T18:06:26.0097843Z Running distributed/test_store 1/1 ... [2024-11-01 18:06:26.007296] 2024-11-01T18:06:26.0098425Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:06:26.0100236Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/test_store.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:06:26.007871] 2024-11-01T18:12:50.1899760Z 2024-11-01T18:12:50.1901302Z distributed/test_store 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_store_1.1_f1be3c7d64aa3a06_.log 2024-11-01T18:12:50.1950392Z Running 88 items in this shard: test/distributed/test_store.py::FileStoreTest::test_append, test/distributed/test_store.py::FileStoreTest::test_compare_set, test/distributed/test_store.py::FileStoreTest::test_init_pg_and_rpc_with_same_file, test/distributed/test_store.py::FileStoreTest::test_multi_get, test/distributed/test_store.py::FileStoreTest::test_multi_set, test/distributed/test_store.py::FileStoreTest::test_refcount, test/distributed/test_store.py::FileStoreTest::test_set_get_check, test/distributed/test_store.py::FileStoreTest::test_simple_wait, test/distributed/test_store.py::HashStoreTest::test_append, test/distributed/test_store.py::HashStoreTest::test_compare_set, test/distributed/test_store.py::HashStoreTest::test_multi_get, test/distributed/test_store.py::HashStoreTest::test_multi_set, test/distributed/test_store.py::HashStoreTest::test_set_get_check, test/distributed/test_store.py::HashStoreTest::test_simple_wait, test/distributed/test_store.py::PrefixStoreTest::test_get_underlying_store, test/distributed/test_store.py::PrefixFileStoreTest::test_append, test/distributed/test_store.py::PrefixFileStoreTest::test_compare_set, test/distributed/test_store.py::PrefixFileStoreTest::test_multi_get, test/distributed/test_store.py::PrefixFileStoreTest::test_multi_set, test/distributed/test_store.py::PrefixFileStoreTest::test_set_get_check, test/distributed/test_store.py::PrefixFileStoreTest::test_simple_wait, test/distributed/test_store.py::TCPStoreTest::test_address_already_in_use, test/distributed/test_store.py::TCPStoreTest::test_append, test/distributed/test_store.py::TCPStoreTest::test_compare_set, test/distributed/test_store.py::TCPStoreTest::test_init_pg_and_rpc_with_same_socket, test/distributed/test_store.py::TCPStoreTest::test_multi_get, test/distributed/test_store.py::TCPStoreTest::test_multi_set, test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_fixed_world_size, test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_nonfixed_world_size, test/distributed/test_store.py::TCPStoreTest::test_multitenancy, test/distributed/test_store.py::TCPStoreTest::test_numkeys_delkeys, test/distributed/test_store.py::TCPStoreTest::test_repr, test/distributed/test_store.py::TCPStoreTest::test_set_get_check, test/distributed/test_store.py::TCPStoreTest::test_simple_wait, test/distributed/test_store.py::TCPStoreTest::test_store_timeout_on_missing_clients, test/distributed/test_store.py::TCPStoreTest::test_take_over_listen_socket, test/distributed/test_store.py::TCPStoreTest::test_world_size_0_raises, test/distributed/test_store.py::LibUvTCPStoreTest::test_address_already_in_use, test/distributed/test_store.py::LibUvTCPStoreTest::test_append, test/distributed/test_store.py::LibUvTCPStoreTest::test_compare_set, test/distributed/test_store.py::LibUvTCPStoreTest::test_init_pg_and_rpc_with_same_socket, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_get, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_set, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_fixed_world_size, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_nonfixed_world_size, test/distributed/test_store.py::LibUvTCPStoreTest::test_multitenancy, test/distributed/test_store.py::LibUvTCPStoreTest::test_numkeys_delkeys, test/distributed/test_store.py::LibUvTCPStoreTest::test_repr, test/distributed/test_store.py::LibUvTCPStoreTest::test_set_get_check, test/distributed/test_store.py::LibUvTCPStoreTest::test_simple_wait, test/distributed/test_store.py::LibUvTCPStoreTest::test_store_timeout_on_missing_clients, test/distributed/test_store.py::LibUvTCPStoreTest::test_take_over_listen_socket, test/distributed/test_store.py::LibUvTCPStoreTest::test_world_size_0_raises, test/distributed/test_store.py::PrefixTCPStoreTest::test_append, test/distributed/test_store.py::PrefixTCPStoreTest::test_compare_set, test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_get, test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_set, test/distributed/test_store.py::PrefixTCPStoreTest::test_set_get_check, test/distributed/test_store.py::PrefixTCPStoreTest::test_simple_wait, test/distributed/test_store.py::PrefixTCPStoreTest::test_underlying_non_prefix_store, test/distributed/test_store.py::PythonStoreTest::test_set_get, test/distributed/test_store.py::RendezvousTest::test_unknown_handler, test/distributed/test_store.py::RendezvousTest::test_url_with_node_params, test/distributed/test_store.py::RendezvousEnvTest::test_nominal, test/distributed/test_store.py::RendezvousFileTest::test_common_errors, test/distributed/test_store.py::RendezvousFileTest::test_nominal, test/distributed/test_store.py::RendezvousTCPTest::test_common_errors, test/distributed/test_store.py::RendezvousTCPTest::test_dns_timeout, test/distributed/test_store.py::RendezvousTCPTest::test_nominal, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_doest_break_client, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_set, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_url_with_libuv, test/distributed/test_store.py::TestPythonStore::test_append_roundtrip, test/distributed/test_store.py::TestPythonStore::test_extended_methods_fallbacks, test/distributed/test_store.py::TestPythonStore::test_has_extended_api_passthrough, test/distributed/test_store.py::TestPythonStore::test_has_extended_api_roundtrip, test/distributed/test_store.py::TestPythonStore::test_multi_get_roundtrip, test/distributed/test_store.py::TestPythonStore::test_multi_set_roundtrip, test/distributed/test_store.py::TestPythonStore::test_optional_methods_fail, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_file_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_hash_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_prefix_file_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store_uv, test/distributed/test_store.py::TimeoutTest::test_interrupt_doesnt_break_wait, test/distributed/test_store.py::InitPgWithNonUvStore::test_with_env_var, test/distributed/test_store.py::InitPgWithNonUvStore::test_with_url_param, test/distributed/test_store.py::TestClientProtocol::test_client_connect 2024-11-01T18:12:50.1986225Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_append 2024-11-01T18:12:50.1987319Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_compare_set 2024-11-01T18:12:50.1988454Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_init_pg_and_rpc_with_same_file 2024-11-01T18:12:50.1989592Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_multi_get 2024-11-01T18:12:50.1990611Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_multi_set 2024-11-01T18:12:50.1991623Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_refcount 2024-11-01T18:12:50.1992659Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_set_get_check 2024-11-01T18:12:50.1993779Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_simple_wait 2024-11-01T18:12:50.1994790Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_append 2024-11-01T18:12:50.1995786Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_compare_set 2024-11-01T18:12:50.1996810Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_multi_get 2024-11-01T18:12:50.1997825Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_multi_set 2024-11-01T18:12:50.1998870Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_set_get_check 2024-11-01T18:12:50.1999915Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_simple_wait 2024-11-01T18:12:50.2001024Z Running 1 items in this shard: test/distributed/test_store.py::PrefixStoreTest::test_get_underlying_store 2024-11-01T18:12:50.2002141Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_append 2024-11-01T18:12:50.2003220Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_compare_set 2024-11-01T18:12:50.2004326Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_multi_get 2024-11-01T18:12:50.2005418Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_multi_set 2024-11-01T18:12:50.2006662Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_set_get_check 2024-11-01T18:12:50.2007793Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_simple_wait 2024-11-01T18:12:50.2008927Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_address_already_in_use 2024-11-01T18:12:50.2009987Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_append 2024-11-01T18:12:50.2011054Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_compare_set 2024-11-01T18:12:50.2012194Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_init_pg_and_rpc_with_same_socket 2024-11-01T18:12:50.2013321Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_get 2024-11-01T18:12:50.2014323Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_set 2024-11-01T18:12:50.2015469Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_fixed_world_size 2024-11-01T18:12:50.2016748Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_nonfixed_world_size 2024-11-01T18:12:50.2017921Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multitenancy 2024-11-01T18:12:50.2018964Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_numkeys_delkeys 2024-11-01T18:12:50.2019967Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_repr 2024-11-01T18:12:50.2020960Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_set_get_check 2024-11-01T18:12:50.2021993Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_simple_wait 2024-11-01T18:12:50.2023130Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_store_timeout_on_missing_clients 2024-11-01T18:12:50.2024340Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_take_over_listen_socket 2024-11-01T18:12:50.2025471Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_world_size_0_raises 2024-11-01T18:12:50.2026620Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_address_already_in_use 2024-11-01T18:12:50.2027734Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_append 2024-11-01T18:12:50.2028804Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_compare_set 2024-11-01T18:12:50.2030005Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_init_pg_and_rpc_with_same_socket 2024-11-01T18:12:50.2031198Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_get 2024-11-01T18:12:50.2032263Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_set 2024-11-01T18:12:50.2033556Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_fixed_world_size 2024-11-01T18:12:50.2034897Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_nonfixed_world_size 2024-11-01T18:12:50.2036137Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multitenancy 2024-11-01T18:12:50.2037272Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_numkeys_delkeys 2024-11-01T18:12:50.2038347Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_repr 2024-11-01T18:12:50.2039412Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_set_get_check 2024-11-01T18:12:50.2040509Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_simple_wait 2024-11-01T18:12:50.2041787Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_store_timeout_on_missing_clients 2024-11-01T18:12:50.2043043Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_take_over_listen_socket 2024-11-01T18:12:50.2044240Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_world_size_0_raises 2024-11-01T18:12:50.2045351Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_append 2024-11-01T18:12:50.2046511Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_compare_set 2024-11-01T18:12:50.2047605Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_get 2024-11-01T18:12:50.2048681Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_set 2024-11-01T18:12:50.2049774Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_set_get_check 2024-11-01T18:12:50.2050882Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_simple_wait 2024-11-01T18:12:50.2052076Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_underlying_non_prefix_store 2024-11-01T18:12:50.2053225Z Running 1 items in this shard: test/distributed/test_store.py::PythonStoreTest::test_set_get 2024-11-01T18:12:50.2054281Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTest::test_unknown_handler 2024-11-01T18:12:50.2055411Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTest::test_url_with_node_params 2024-11-01T18:12:50.2056505Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousEnvTest::test_nominal 2024-11-01T18:12:50.2057584Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousFileTest::test_common_errors 2024-11-01T18:12:50.2058855Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousFileTest::test_nominal 2024-11-01T18:12:50.2059956Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_common_errors 2024-11-01T18:12:50.2061055Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_dns_timeout 2024-11-01T18:12:50.2062118Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_nominal 2024-11-01T18:12:50.2063318Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_doest_break_client 2024-11-01T18:12:50.2064599Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_set 2024-11-01T18:12:50.2065812Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_url_with_libuv 2024-11-01T18:12:50.2066969Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_append_roundtrip 2024-11-01T18:12:50.2068147Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_extended_methods_fallbacks 2024-11-01T18:12:50.2069391Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_has_extended_api_passthrough 2024-11-01T18:12:50.2070627Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_has_extended_api_roundtrip 2024-11-01T18:12:50.2071814Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_multi_get_roundtrip 2024-11-01T18:12:50.2073035Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_multi_set_roundtrip 2024-11-01T18:12:50.2074201Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_optional_methods_fail 2024-11-01T18:12:50.2075384Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_file_store 2024-11-01T18:12:50.2076555Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_hash_store 2024-11-01T18:12:50.2077890Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_prefix_file_store 2024-11-01T18:12:50.2079120Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store 2024-11-01T18:12:50.2080312Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store_uv 2024-11-01T18:12:50.2081505Z Running 1 items in this shard: test/distributed/test_store.py::TimeoutTest::test_interrupt_doesnt_break_wait 2024-11-01T18:12:50.2082748Z Running 1 items in this shard: test/distributed/test_store.py::InitPgWithNonUvStore::test_with_env_var 2024-11-01T18:12:50.2083895Z Running 1 items in this shard: test/distributed/test_store.py::InitPgWithNonUvStore::test_with_url_param 2024-11-01T18:12:50.2085023Z Running 1 items in this shard: test/distributed/test_store.py::TestClientProtocol::test_client_connect 2024-11-01T18:12:50.2085690Z 2024-11-01T18:12:50.2086234Z Running distributed/test_c10d_spawn_ucc 1/1 ... [2024-11-01 18:12:50.193219] 2024-11-01T18:12:50.2086861Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:12:50.2088738Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/test_c10d_spawn_ucc.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:12:50.193697] 2024-11-01T18:13:17.6910538Z 2024-11-01T18:13:17.6913021Z distributed/test_c10d_spawn_ucc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_ucc_1.1_b689a844060459e5_.log 2024-11-01T18:13:17.6917611Z Running 6 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_gather, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all_single, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_allreduce, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_broadcast, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_reduce 2024-11-01T18:13:17.6921613Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_gather 2024-11-01T18:13:17.6922947Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all 2024-11-01T18:13:17.6924364Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all_single 2024-11-01T18:13:17.6925816Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_allreduce 2024-11-01T18:13:17.6927130Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_broadcast 2024-11-01T18:13:17.6928437Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_reduce 2024-11-01T18:13:17.6929172Z 2024-11-01T18:13:17.6929577Z Running distributed/test_c10d_nccl 1/1 ... [2024-11-01 18:13:17.691886] 2024-11-01T18:13:17.6930175Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:13:17.6932026Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:13:17.692396] 2024-11-01T18:13:19.2189279Z 2024-11-01T18:13:19.2191302Z distributed/test_c10d_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_1.1_7fc53c2a8ba43f89_.log 2024-11-01T18:13:19.2192394Z 2024-11-01T18:13:19.2198032Z Running distributed/test_c10d_spawn_nccl 1/1 ... [2024-11-01 18:13:19.219587] 2024-11-01T18:13:19.2198734Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:13:19.2204731Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/test_c10d_spawn_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:13:19.220100] 2024-11-01T18:13:58.7306513Z 2024-11-01T18:13:58.7307955Z distributed/test_c10d_spawn_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_nccl_1.1_b6d52629c6fc5411_.log 2024-11-01T18:13:58.7314659Z Running 9 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather_base, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all_single, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_allreduce, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_broadcast, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter_non_contiguous 2024-11-01T18:13:58.7320601Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather 2024-11-01T18:13:58.7322000Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather_base 2024-11-01T18:13:58.7323390Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all 2024-11-01T18:13:58.7324797Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all_single 2024-11-01T18:13:58.7326194Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_allreduce 2024-11-01T18:13:58.7327547Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_broadcast 2024-11-01T18:13:58.7328870Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce 2024-11-01T18:13:58.7330233Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter 2024-11-01T18:13:58.7331703Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter_non_contiguous 2024-11-01T18:13:58.7332576Z 2024-11-01T18:13:58.7333027Z Running distributed/test_c10d_ops_nccl 1/1 ... [2024-11-01 18:13:58.731358] 2024-11-01T18:13:58.7333649Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:13:58.7335444Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/test_c10d_ops_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:13:58.731858] 2024-11-01T18:14:00.2338915Z 2024-11-01T18:14:00.2340816Z distributed/test_c10d_ops_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_ops_nccl_1.1_6668d062e4bba6af_.log 2024-11-01T18:14:00.2341904Z 2024-11-01T18:14:00.2347299Z Running distributed/elastic/events/lib_test 1/1 ... [2024-11-01 18:14:00.234499] 2024-11-01T18:14:00.2347984Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:14:00.2353676Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/elastic/events/lib_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:14:00.235014] 2024-11-01T18:14:04.3119397Z 2024-11-01T18:14:04.3122144Z distributed/elastic/events/lib_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.events.lib_test_1.1_0764544c3a746fc8_.log 2024-11-01T18:14:04.3131865Z Running 8 items in this shard: test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_created, test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser, test/distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str 2024-11-01T18:14:04.3139256Z 2024-11-01T18:14:04.3140208Z Running distributed/elastic/metrics/api_test 1/1 ... [2024-11-01 18:14:04.312719] 2024-11-01T18:14:04.3141404Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:14:04.3144236Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:14:04.313330] 2024-11-01T18:14:08.2903602Z 2024-11-01T18:14:08.2905847Z distributed/elastic/metrics/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.metrics.api_test_1.1_1a8a731d5fe637e4_.log 2024-11-01T18:14:08.2908532Z Running 3 items in this shard: test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_get_metric_name, test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_inheritance, test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_profile 2024-11-01T18:14:08.2910160Z 2024-11-01T18:14:08.2912139Z Running distributed/elastic/multiprocessing/api_test 1/1 ... [2024-11-01 18:14:08.290947] 2024-11-01T18:14:08.2913000Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:14:08.2917815Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:14:08.291457] 2024-11-01T18:14:43.8265764Z 2024-11-01T18:14:43.8268868Z distributed/elastic/multiprocessing/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.multiprocessing.api_test_1.1_b705c42a77083530_.log 2024-11-01T18:14:43.8288995Z Running 25 items in this shard: test/distributed/elastic/multiprocessing/api_test.py::RunProcResultsTest::test_get_failures, test/distributed/elastic/multiprocessing/api_test.py::RunProcResultsTest::test_is_failed, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_str_bad_input, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_value, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_value_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_args_env_len_mismatch, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_function_large_ret_val, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_function_raise, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_function_with_tensor, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_invalid_log_dir, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_multiprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_multiprocessing_context_poll_raises_exception, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_pcontext_wait, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_pcontext_wait_on_a_child_thread, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_to_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_void_function, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_wait_for_all_child_procs_to_exit, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_binary_exit, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_binary_incorrect_entrypoint, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_binary_raises, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_subprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_validate_full_rank, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsFuncTest::test_function, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsBinaryTest::test_binary, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsBinaryTest::test_binary_redirect_and_tee 2024-11-01T18:14:43.8304731Z 2024-11-01T18:14:43.8305275Z Running distributed/elastic/timer/api_test 1/1 ... [2024-11-01 18:14:43.827337] 2024-11-01T18:14:43.8305935Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:14:43.8307778Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/elastic/timer/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:14:43.827850] 2024-11-01T18:14:45.3023805Z 2024-11-01T18:14:45.3025504Z distributed/elastic/timer/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.api_test_1.1_2827ff97f8d10b8f_.log 2024-11-01T18:14:45.3026644Z 2024-11-01T18:14:45.3030364Z Running distributed/elastic/timer/local_timer_example 1/1 ... [2024-11-01 18:14:45.302826] 2024-11-01T18:14:45.3031166Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:14:45.3036090Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:14:45.303309] 2024-11-01T18:15:08.3172458Z 2024-11-01T18:15:08.3174381Z distributed/elastic/timer/local_timer_example 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.local_timer_example_1.1_68e64129a9ef9a5f_.log 2024-11-01T18:15:08.3178042Z Running 2 items in this shard: test/distributed/elastic/timer/local_timer_example.py::LocalTimerExample::test_example_start_method_spawn, test/distributed/elastic/timer/local_timer_example.py::LocalTimerExample::test_torch_mp_example 2024-11-01T18:15:08.3179543Z 2024-11-01T18:15:08.3182197Z Running distributed/elastic/timer/local_timer_test 1/1 ... [2024-11-01 18:15:08.317963] 2024-11-01T18:15:08.3182924Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:15:08.3187664Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/elastic/timer/local_timer_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:15:08.318467] 2024-11-01T18:15:17.9028985Z 2024-11-01T18:15:17.9031912Z distributed/elastic/timer/local_timer_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.local_timer_test_1.1_299d5ef3741de756_.log 2024-11-01T18:15:17.9042396Z Running 14 items in this shard: test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_client_interaction, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_exception_propagation, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_get_timer_recursive, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_happy_path, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_no_client, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_timer, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get_less_than_size, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get_size, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_acquire_release, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_expired_timers, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_valid_timers, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_watchdog_call_count, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_watchdog_empty_queue 2024-11-01T18:15:17.9051387Z 2024-11-01T18:15:17.9051943Z Running distributed/elastic/utils/distributed_test 1/1 ... [2024-11-01 18:15:17.903550] 2024-11-01T18:15:17.9052727Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:15:17.9054771Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/elastic/utils/distributed_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:15:17.904020] 2024-11-01T18:15:26.4868522Z 2024-11-01T18:15:26.4871540Z distributed/elastic/utils/distributed_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.distributed_test_1.1_8b826889fcadb69d_.log 2024-11-01T18:15:26.4883145Z Running 8 items in this shard: test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_multi, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_no_port_multi, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_single_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_timeout_on_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_timeout_on_worker, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_with_libuv_support, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_port_already_in_use_on_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_port_already_in_use_on_worker 2024-11-01T18:15:26.4891659Z 2024-11-01T18:15:26.4892700Z Running distributed/elastic/utils/logging_test 1/1 ... [2024-11-01 18:15:26.487893] 2024-11-01T18:15:26.4893927Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:15:26.4896636Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:15:26.488504] 2024-11-01T18:15:30.5155950Z 2024-11-01T18:15:30.5157986Z distributed/elastic/utils/logging_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.logging_test_1.1_a9336d3529114d58_.log 2024-11-01T18:15:30.5160482Z Running 2 items in this shard: test/distributed/elastic/utils/logging_test.py::LoggingTest::test_derive_module_name, test/distributed/elastic/utils/logging_test.py::LoggingTest::test_logger_name 2024-11-01T18:15:30.5161685Z 2024-11-01T18:15:30.5165065Z Running distributed/elastic/utils/util_test 1/1 ... [2024-11-01 18:15:30.516267] 2024-11-01T18:15:30.5165741Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:15:30.5170466Z Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'distributed/elastic/utils/util_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 18:15:30.516754] 2024-11-01T18:15:34.6958211Z 2024-11-01T18:15:34.6960693Z distributed/elastic/utils/util_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.util_test_1.1_d7b4b80707f843ba_.log 2024-11-01T18:15:34.6967709Z Running 12 items in this shard: test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier_hash_store, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier_timeout_operations, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier_timeout_rank_tracing, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_get_all_rank_0, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_get_all_rank_n, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_synchronize, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_synchronize_hash_store, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_custom_name, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_different, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_none 2024-11-01T18:15:34.6973534Z 2024-11-01T18:15:35.3754451Z Running test batch 'tests to run' cost 5686.2 seconds 2024-11-01T18:15:35.9812798Z 2024-11-01T18:15:35.9813312Z real 94m52.619s 2024-11-01T18:15:35.9813943Z user 203m9.448s 2024-11-01T18:15:35.9814319Z sys 34m42.785s 2024-11-01T18:15:35.9814639Z + assert_git_not_dirty 2024-11-01T18:15:35.9815243Z + [[ linux-jammy-py3.9-gcc11 != *rocm* ]] 2024-11-01T18:15:35.9815765Z + [[ linux-jammy-py3.9-gcc11 != *xla* ]] 2024-11-01T18:15:35.9819130Z ++ git status --porcelain 2024-11-01T18:15:35.9819820Z ++ grep -v '?? third_party' 2024-11-01T18:16:01.1689542Z ++ true 2024-11-01T18:16:01.1689879Z + git_status= 2024-11-01T18:16:01.1690555Z + [[ -n '' ]] 2024-11-01T18:16:01.1691274Z + [[ linux-jammy-py3.9-gcc11 == *cuda* ]] 2024-11-01T18:16:01.1693066Z + [[ linux-jammy-py3.9-gcc11 == *rocm* ]] 2024-11-01T18:16:01.1693547Z + [[ 1 == 1 ]] 2024-11-01T18:16:01.1693833Z + test_rpc 2024-11-01T18:16:01.1694180Z + echo 'Testing RPC C++ tests' 2024-11-01T18:16:01.1694543Z Testing RPC C++ tests 2024-11-01T18:16:01.1697874Z + ln -sf /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/bin 2024-11-01T18:16:01.1717638Z + ln -sf /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/bin 2024-11-01T18:16:01.1727188Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/bin 2024-11-01T18:16:01.1728025Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc 2024-11-01T18:16:01.2758222Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-11-01T18:16:01.2759580Z import pkg_resources 2024-11-01T18:16:05.6454903Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-11-01T18:16:05.6585704Z Found test times from artifacts 2024-11-01T18:16:05.7112205Z Found test times from artifacts 2024-11-01T18:16:05.7134652Z Running all tests 2024-11-01T18:16:05.7138236Z Running parallel tests on 3 processes 2024-11-01T18:16:05.7138999Z Name: tests to run (est. time: 0.0min) 2024-11-01T18:16:05.7139682Z Serial tests (1): 2024-11-01T18:16:05.7140208Z cpp/test_cpp_rpc 1/1 2024-11-01T18:16:05.7141053Z Parallel tests (0): 2024-11-01T18:16:05.7141571Z Name: excluded (est. time: 0.0min) 2024-11-01T18:16:05.7142185Z Serial tests (0): 2024-11-01T18:16:05.7142684Z Parallel tests (0): 2024-11-01T18:16:05.7387432Z Running cpp/test_cpp_rpc 1/1 ... [2024-11-01 18:16:05.738289] 2024-11-01T18:16:05.7388460Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T18:16:05.7402721Z Executing ['pytest', '/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '3', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-6c0da635215d7a97.xml', '-x', '--reruns=2'] ... [2024-11-01 18:16:05.739658] 2024-11-01T18:16:11.0245893Z 2024-11-01T18:16:11.0247750Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_09ab296419a7fbf2_.log 2024-11-01T18:16:11.0249049Z 2024-11-01T18:16:11.7042075Z Running test batch 'tests to run' cost 5.99 seconds 2024-11-01T18:16:12.3046584Z + cleanup_workspace 2024-11-01T18:16:12.3047758Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-11-01T18:16:12.3049048Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-11-01T18:16:12.3050110Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-11-01T18:16:12.3050836Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-11-01T18:16:12.3051755Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-11-01T18:16:12.3052808Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-11-01T18:16:12.3053574Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-11-01T18:16:12.7978143Z ##[group]Run cat test/**/*_toprint.log || true 2024-11-01T18:16:12.7978669Z cat test/**/*_toprint.log || true 2024-11-01T18:16:12.7987815Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T18:16:12.7988340Z env: 2024-11-01T18:16:12.7988608Z GIT_DEFAULT_BRANCH: main 2024-11-01T18:16:12.7989219Z DOCKER_CONTAINER_ID: ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e 2024-11-01T18:16:12.7989896Z ##[endgroup] 2024-11-01T18:16:12.8084857Z cat: 'test/**/*_toprint.log': No such file or directory 2024-11-01T18:16:12.8116809Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-11-01T18:16:12.8117274Z kill "$MONITOR_SCRIPT_PID" 2024-11-01T18:16:12.8122739Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T18:16:12.8123249Z env: 2024-11-01T18:16:12.8123517Z GIT_DEFAULT_BRANCH: main 2024-11-01T18:16:12.8124149Z DOCKER_CONTAINER_ID: ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e 2024-11-01T18:16:12.8124830Z MONITOR_SCRIPT_PID: 39788 2024-11-01T18:16:12.8125181Z ##[endgroup] 2024-11-01T18:16:12.8332794Z Prepare all required actions 2024-11-01T18:16:12.8333241Z Getting action download info 2024-11-01T18:16:13.0485831Z Download action repository 'actions/upload-artifact@v3' (SHA:ff15f0306b3f739f7b6fd43fb5d26cd321bd4de5) 2024-11-01T18:16:13.1876805Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-11-01T18:16:13.1877286Z with: 2024-11-01T18:16:13.1877698Z file-suffix: test-distributed-1-2-linux.2xlarge_32396543718 2024-11-01T18:16:13.1878244Z s3-bucket: gha-artifacts 2024-11-01T18:16:13.1878574Z env: 2024-11-01T18:16:13.1878821Z GIT_DEFAULT_BRANCH: main 2024-11-01T18:16:13.1879435Z DOCKER_CONTAINER_ID: ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e 2024-11-01T18:16:13.1880105Z ##[endgroup] 2024-11-01T18:16:13.1910155Z ##[group]Run # Remove any previous test jsons if they exist 2024-11-01T18:16:13.1910793Z # Remove any previous test jsons if they exist 2024-11-01T18:16:13.1911310Z rm -f test-jsons-*.zip 2024-11-01T18:16:13.1911950Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2024-11-01T18:16:13.1918107Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T18:16:13.1918769Z env: 2024-11-01T18:16:13.1919020Z GIT_DEFAULT_BRANCH: main 2024-11-01T18:16:13.1919644Z DOCKER_CONTAINER_ID: ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e 2024-11-01T18:16:13.1920461Z FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_32396543718 2024-11-01T18:16:13.1920987Z ##[endgroup] 2024-11-01T18:16:13.2092263Z adding: test/test-reports/td_exclusions-780e49e7887c40374d11.json (deflated 86%) 2024-11-01T18:16:13.2093307Z adding: test/test-reports/td_exclusions-2e0f4e170e6b686d5088.json (deflated 18%) 2024-11-01T18:16:13.2750748Z ##[group]Run # Remove any previous test reports if they exist 2024-11-01T18:16:13.2751620Z # Remove any previous test reports if they exist 2024-11-01T18:16:13.2752146Z rm -f test-reports-*.zip 2024-11-01T18:16:13.2752880Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2024-11-01T18:16:13.2759414Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T18:16:13.2759925Z env: 2024-11-01T18:16:13.2760190Z GIT_DEFAULT_BRANCH: main 2024-11-01T18:16:13.2760817Z DOCKER_CONTAINER_ID: ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e 2024-11-01T18:16:13.2761635Z FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_32396543718 2024-11-01T18:16:13.2762162Z ##[endgroup] 2024-11-01T18:16:13.2849265Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_compile/distributed._tensor.test_dtensor_compile-8d460e0a23088864.xml (deflated 85%) 2024-11-01T18:16:13.2851216Z adding: test/test-reports/python-pytest/distributed.test_inductor_collectives/distributed.test_inductor_collectives-dd32ce59804db922.xml (deflated 94%) 2024-11-01T18:16:13.2853176Z adding: test/test-reports/python-pytest/distributed._tensor.debug.test_op_coverage/distributed._tensor.debug.test_op_coverage-93298eb87eedc4b7.xml (deflated 34%) 2024-11-01T18:16:13.2855204Z adding: test/test-reports/python-pytest/distributed.flight_recorder.test_fr_analysis/distributed.flight_recorder.test_fr_analysis-02a19c2fe0382962.xml (deflated 36%) 2024-11-01T18:16:13.2857131Z adding: test/test-reports/python-pytest/distributed._tools.test_mod_tracker/distributed._tools.test_mod_tracker-7cc5d2171c065472.xml (deflated 64%) 2024-11-01T18:16:13.2859184Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_nested_dict/distributed.checkpoint.test_nested_dict-95bc1c2043306306.xml (deflated 51%) 2024-11-01T18:16:13.2861187Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_logger/distributed._shard.sharded_tensor.test_logger-9769dff3d75ab0af.xml (deflated 37%) 2024-11-01T18:16:13.2863338Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_micro_pipeline_tp/distributed.tensor.parallel.test_micro_pipeline_tp-ffbf004795821980.xml (deflated 95%) 2024-11-01T18:16:13.2865505Z adding: test/test-reports/python-pytest/distributed._tools.test_sac_ilp/distributed._tools.test_sac_ilp-96c4a290b3895226.xml (deflated 75%) 2024-11-01T18:16:13.2867327Z adding: test/test-reports/python-pytest/distributed.pipelining.test_backward/distributed.pipelining.test_backward-968336006f3da1d5.xml (deflated 68%) 2024-11-01T18:16:13.2869272Z adding: test/test-reports/python-pytest/distributed.pipelining.test_transformer/distributed.pipelining.test_transformer-438e6a6cec3d2af9.xml (deflated 39%) 2024-11-01T18:16:13.2871194Z adding: test/test-reports/python-pytest/distributed._tools.test_sac_estimator/distributed._tools.test_sac_estimator-4dd6a08babad1554.xml (deflated 63%) 2024-11-01T18:16:13.2873208Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dedup_tensors/distributed.checkpoint.test_dedup_tensors-59e1633be3210442.xml (deflated 37%) 2024-11-01T18:16:13.2875121Z adding: test/test-reports/python-pytest/distributed.pipelining.test_pipe/distributed.pipelining.test_pipe-eb40e4e163966a0f.xml (deflated 65%) 2024-11-01T18:16:13.2876948Z adding: test/test-reports/python-pytest/distributed._tensor.test_common_rules/distributed._tensor.test_common_rules-0b5c3b56eab54cdf.xml (deflated 80%) 2024-11-01T18:16:13.2878888Z adding: test/test-reports/python-pytest/distributed._tensor.test_attention/distributed._tensor.test_attention-2dcee7cd9063e719.xml (deflated 74%) 2024-11-01T18:16:13.2880819Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_save_load_api/distributed.checkpoint.test_save_load_api-981a2b0a0d483333.xml (deflated 34%) 2024-11-01T18:16:13.2882820Z adding: test/test-reports/python-pytest/distributed.checkpoint.e2e.test_fsdp_ep/distributed.checkpoint.e2e.test_fsdp_ep-db121ebe6d4c67bf.xml (deflated 34%) 2024-11-01T18:16:13.2884746Z adding: test/test-reports/python-pytest/distributed.test_c10d_functional_native/distributed.test_c10d_functional_native-0fde980f9f6b0ef5.xml (deflated 93%) 2024-11-01T18:16:13.2886671Z adding: test/test-reports/python-pytest/distributed._tools.test_runtime_estimator/distributed._tools.test_runtime_estimator-82f03d9825624b25.xml (deflated 63%) 2024-11-01T18:16:13.2888444Z adding: test/test-reports/python-pytest/distributed._tensor.test_api/distributed._tensor.test_api-d7864ac69926355b.xml (deflated 78%) 2024-11-01T18:16:13.2890249Z adding: test/test-reports/python-pytest/distributed.pipelining.test_microbatch/distributed.pipelining.test_microbatch-23b2574a6f09d596.xml (deflated 54%) 2024-11-01T18:16:13.2892205Z adding: test/test-reports/python-pytest/distributed._tensor.test_experimental_ops/distributed._tensor.test_experimental_ops-3abc04b61a245b0d.xml (deflated 59%) 2024-11-01T18:16:13.2894058Z adding: test/test-reports/python-pytest/distributed._tensor.test_tensor_ops/distributed._tensor.test_tensor_ops-97029a74475dcd97.xml (deflated 88%) 2024-11-01T18:16:13.2896122Z adding: test/test-reports/python-pytest/distributed._tensor.experimental.test_register_sharding/distributed._tensor.experimental.test_register_sharding-88d885e87320cf23.xml (deflated 52%) 2024-11-01T18:16:13.2898077Z adding: test/test-reports/python-pytest/distributed._tensor.test_init/distributed._tensor.test_init-5c1a67a72f78429d.xml (deflated 75%) 2024-11-01T18:16:13.2899857Z adding: test/test-reports/python-pytest/distributed._tools.test_memory_tracker/distributed._tools.test_memory_tracker-926c04ab1b743d55.xml (deflated 44%) 2024-11-01T18:16:13.2901757Z adding: test/test-reports/python-pytest/distributed._tensor.test_xla_integration/distributed._tensor.test_xla_integration-3cc535dbfaa6a70b.xml (deflated 74%) 2024-11-01T18:16:13.2903820Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_state/distributed._composable.fsdp.test_fully_shard_state-f1effdfff59cdc43.xml (deflated 81%) 2024-11-01T18:16:13.2905871Z adding: test/test-reports/python-pytest/distributed.pipelining.test_unflatten/distributed.pipelining.test_unflatten-944f69fb990654e9.xml (deflated 41%) 2024-11-01T18:16:13.2907870Z adding: test/test-reports/python-pytest/distributed._composable.test_checkpoint/distributed._composable.test_checkpoint-22251250505b4aa2.xml (deflated 70%) 2024-11-01T18:16:13.2909756Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsspec/distributed.checkpoint.test_fsspec-66470acb792d459e.xml (deflated 65%) 2024-11-01T18:16:13.2911852Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_util/distributed._composable.fully_shard.test_fully_shard_util-80bedf40a8c7372e.xml (deflated 61%) 2024-11-01T18:16:13.2914031Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_traverse/distributed.checkpoint.test_traverse-d4121e1f827bb063.xml (deflated 75%) 2024-11-01T18:16:13.2915958Z adding: test/test-reports/python-pytest/distributed.test_compute_comm_reordering/distributed.test_compute_comm_reordering-a3f18ca8b721bbde.xml (deflated 86%) 2024-11-01T18:16:13.2917891Z adding: test/test-reports/python-pytest/distributed._tensor.test_convolution_ops/distributed._tensor.test_convolution_ops-8b5f911b3b034e6c.xml (deflated 52%) 2024-11-01T18:16:13.2919714Z adding: test/test-reports/python-pytest/distributed._tensor.test_math_ops/distributed._tensor.test_math_ops-d58894d6fef45bf4.xml (deflated 82%) 2024-11-01T18:16:13.2921731Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dtensor_resharding/distributed.checkpoint.test_dtensor_resharding-f0aaf226ccfb94a5.xml (deflated 72%) 2024-11-01T18:16:13.2923778Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_format_utils/distributed.checkpoint.test_format_utils-ce82c811ace1651f.xml (deflated 61%) 2024-11-01T18:16:13.2925783Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_tp_checkpoint/distributed.checkpoint.test_tp_checkpoint-a18cbeaf8fa6a027.xml (deflated 54%) 2024-11-01T18:16:13.2927819Z adding: test/test-reports/python-pytest/distributed.checkpoint.fsdp.test_fsdp_dsd/distributed.checkpoint.fsdp.test_fsdp_dsd-cce4883b2c070111.xml (deflated 77%) 2024-11-01T18:16:13.2929799Z adding: test/test-reports/python-pytest/distributed._tools.test_fsdp2_mem_tracker/distributed._tools.test_fsdp2_mem_tracker-c37a02d5d2993795.xml (deflated 60%) 2024-11-01T18:16:13.2931691Z adding: test/test-reports/python-pytest/distributed._tensor.test_redistribute/distributed._tensor.test_redistribute-83fdada4088f422b.xml (deflated 83%) 2024-11-01T18:16:13.2933585Z adding: test/test-reports/python-pytest/distributed.elastic.test_control_plane/distributed.elastic.test_control_plane-a2811bb302d1cbea.xml (deflated 75%) 2024-11-01T18:16:13.2935569Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_style/distributed.tensor.parallel.test_tp_style-435ef6deb1907cae.xml (deflated 80%) 2024-11-01T18:16:13.2937722Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_overlap/distributed._composable.fsdp.test_fully_shard_overlap-c4fed3f027e135c6.xml (deflated 62%) 2024-11-01T18:16:13.2939648Z adding: test/test-reports/python-pytest/distributed.test_device_mesh/distributed.test_device_mesh-23004f4e81b681f9.xml (deflated 87%) 2024-11-01T18:16:13.2941379Z adding: test/test-reports/python-pytest/distributed._tensor.test_matrix_ops/distributed._tensor.test_matrix_ops-3ff02fd2b81f742b.xml (deflated 77%) 2024-11-01T18:16:13.2943589Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_mixed_precision/distributed._composable.fully_shard.test_fully_shard_mixed_precision-0ea58b673ae7660d.xml (deflated 47%) 2024-11-01T18:16:13.2945689Z adding: test/test-reports/python-pytest/distributed.test_functional_api/distributed.test_functional_api-81e96001c5d0cbc1.xml (deflated 89%) 2024-11-01T18:16:13.2947579Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_model_state/distributed.checkpoint.test_fsdp_model_state-148087937646cd24.xml (deflated 55%) 2024-11-01T18:16:13.2949568Z adding: test/test-reports/python-pytest/distributed._tensor.test_optimizers/distributed._tensor.test_optimizers-f9b3a8144c56fd44.xml (deflated 82%) 2024-11-01T18:16:13.2951593Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_parallelize_api/distributed.tensor.parallel.test_parallelize_api-3fc8eb08fdf75428.xml (deflated 83%) 2024-11-01T18:16:13.2953569Z adding: test/test-reports/python-pytest/distributed.test_c10d_logger/distributed.test_c10d_logger-66094a3997277e6c.xml (deflated 65%) 2024-11-01T18:16:13.2955298Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_utils/distributed.checkpoint.test_utils-1ac6a6cf98890478.xml (deflated 71%) 2024-11-01T18:16:13.2957340Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_compile/distributed._composable.fsdp.test_fully_shard_compile-1c3b7bf231127384.xml (deflated 91%) 2024-11-01T18:16:13.2959693Z adding: test/test-reports/python-pytest/distributed._tensor.debug.test_comm_mode/distributed._tensor.debug.test_comm_mode-84007d58342fc5ef.xml (deflated 60%) 2024-11-01T18:16:13.2962598Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_clip_grad_norm_/distributed._composable.fsdp.test_fully_shard_clip_grad_norm_-b7fa0c118a171fba.xml (deflated 53%) 2024-11-01T18:16:13.2965163Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_mixed_precision/distributed._composable.fsdp.test_fully_shard_mixed_precision-036eedeffc81ee2e.xml (deflated 83%) 2024-11-01T18:16:13.2967278Z adding: test/test-reports/python-pytest/distributed._composable.test_compose/distributed._composable.test_compose-39734e864c5ae511.xml (deflated 86%) 2024-11-01T18:16:13.2969097Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_pure_fp16/distributed.fsdp.test_fsdp_pure_fp16-84e7e32316f73493.xml (deflated 63%) 2024-11-01T18:16:13.2970923Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_overlap/distributed.fsdp.test_fsdp_overlap-216197c73226cc2d.xml (deflated 64%) 2024-11-01T18:16:13.2973286Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_model_checkpoint/distributed._composable.fully_shard.test_fully_shard_model_checkpoint-b4ee14d5ae4cec02.xml (deflated 79%) 2024-11-01T18:16:13.2975681Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_frozen/distributed._composable.fsdp.test_fully_shard_frozen-676476d1e9bb0a83.xml (deflated 62%) 2024-11-01T18:16:13.2977638Z adding: test/test-reports/python-pytest/distributed._tensor.test_view_ops/distributed._tensor.test_view_ops-db46b64fdf14698f.xml (deflated 65%) 2024-11-01T18:16:13.2979285Z adding: test/test-reports/python-pytest/distributed.test_fake_pg/distributed.test_fake_pg-f26cf9c6e8e0ec99.xml (deflated 82%) 2024-11-01T18:16:13.2981176Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_file_system_checkpoint/distributed.checkpoint.test_file_system_checkpoint-47afd8066b372a1a.xml (deflated 79%) 2024-11-01T18:16:13.2983163Z adding: test/test-reports/python-pytest/distributed.rpc.test_share_memory/distributed.rpc.test_share_memory-68411864e68aec84.xml (deflated 48%) 2024-11-01T18:16:13.2985214Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_state_dict/distributed._composable.fsdp.test_fully_shard_state_dict-8b8b35869fad0530.xml (deflated 75%) 2024-11-01T18:16:13.2987286Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_state_dict/distributed.fsdp.test_fsdp_state_dict-38eb3c131edd1aab.xml (deflated 97%) 2024-11-01T18:16:13.2989109Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_apply/distributed.fsdp.test_fsdp_apply-9cfbfdeba2069e28.xml (deflated 71%) 2024-11-01T18:16:13.2990938Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-b7e3c198604c1ac2.xml (deflated 45%) 2024-11-01T18:16:13.2993220Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_init/distributed._composable.fully_shard.test_fully_shard_init-0b7db8a13e4ecad4.xml (deflated 85%) 2024-11-01T18:16:13.2995375Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_ignored_modules/distributed.fsdp.test_fsdp_ignored_modules-4bca208fa34ac167.xml (deflated 86%) 2024-11-01T18:16:13.2997277Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_grad_acc/distributed.fsdp.test_fsdp_grad_acc-015eab18b278c375.xml (deflated 84%) 2024-11-01T18:16:13.2999172Z adding: test/test-reports/python-pytest/distributed.fsdp.test_checkpoint_wrapper/distributed.fsdp.test_checkpoint_wrapper-ce3d306df75215bd.xml (deflated 76%) 2024-11-01T18:16:13.3001039Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_memory/distributed.fsdp.test_fsdp_memory-34d507bbddc44dee.xml (deflated 64%) 2024-11-01T18:16:13.3002921Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_flatten_params/distributed.fsdp.test_fsdp_flatten_params-1b8f0723a3dd7c30.xml (deflated 85%) 2024-11-01T18:16:13.3005012Z adding: test/test-reports/python-pytest/distributed._composable.test_replicate_with_compiler/distributed._composable.test_replicate_with_compiler-53ff93439615968e.xml (deflated 84%) 2024-11-01T18:16:13.3007190Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_freezing_weights/distributed.fsdp.test_fsdp_freezing_weights-bfd4a254216850ab.xml (deflated 96%) 2024-11-01T18:16:13.3009223Z adding: test/test-reports/python-pytest/distributed.fsdp.test_hsdp_dtensor_state_dict/distributed.fsdp.test_hsdp_dtensor_state_dict-a32dee8c85d96829.xml (deflated 71%) 2024-11-01T18:16:13.3011128Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-675c456f725155a0.xml (deflated 44%) 2024-11-01T18:16:13.3013189Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_extensions/distributed._composable.fsdp.test_fully_shard_extensions-2f21acded8a304a3.xml (deflated 80%) 2024-11-01T18:16:13.3015142Z adding: test/test-reports/python-pytest/distributed.fsdp.test_wrap/distributed.fsdp.test_wrap-0b5b42d25b11a2ff.xml (deflated 94%) 2024-11-01T18:16:13.3016763Z adding: test/test-reports/python-pytest/distributed.test_c10d_pypg/distributed.test_c10d_pypg-aa1090a216673cc7.xml (deflated 94%) 2024-11-01T18:16:13.3018546Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_state_dict/distributed.checkpoint.test_state_dict-4b7617827bb08df4.xml (deflated 85%) 2024-11-01T18:16:13.3020642Z adding: test/test-reports/python-pytest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks-33e021a10b9512d0.xml (deflated 85%) 2024-11-01T18:16:13.3022704Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-5c2afac1c82a6fb5.xml (deflated 93%) 2024-11-01T18:16:13.3024557Z adding: test/test-reports/python-pytest/distributed.algorithms.test_join/distributed.algorithms.test_join-0b2a33872e7a9937.xml (deflated 79%) 2024-11-01T18:16:13.3026435Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-b005865d0a8faec6.xml (deflated 77%) 2024-11-01T18:16:13.3028282Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_misc/distributed.fsdp.test_fsdp_misc-742200ef840ccb67.xml (deflated 89%) 2024-11-01T18:16:13.3030121Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_hybrid_shard/distributed.fsdp.test_fsdp_hybrid_shard-d9ce95b65283ecc3.xml (deflated 81%) 2024-11-01T18:16:13.3032130Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_sharded_grad_scaler/distributed.fsdp.test_fsdp_sharded_grad_scaler-da5c3a4a662312f9.xml (deflated 93%) 2024-11-01T18:16:13.3034337Z adding: test/test-reports/python-pytest/distributed.optim.test_zero_redundancy_optimizer/distributed.optim.test_zero_redundancy_optimizer-c2992e25dcac5a8e.xml (deflated 94%) 2024-11-01T18:16:13.3036328Z adding: test/test-reports/python-pytest/distributed._shard.test_sharder/distributed._shard.test_sharder-66ba80c3c9543108.xml (deflated 66%) 2024-11-01T18:16:13.3038570Z adding: test/test-reports/python-pytest/distributed._shard.sharding_plan.test_sharding_plan/distributed._shard.sharding_plan.test_sharding_plan-38ae23cbd4987102.xml (deflated 75%) 2024-11-01T18:16:13.3041474Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_use_orig_params/distributed.fsdp.test_fsdp_use_orig_params-6db379ce2e7c78f1.xml (deflated 90%) 2024-11-01T18:16:13.3043485Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_multiple_forward/distributed.fsdp.test_fsdp_multiple_forward-ecb4d527f0c2a42b.xml (deflated 46%) 2024-11-01T18:16:13.3045798Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_optim_checkpoint/distributed._composable.fully_shard.test_fully_shard_optim_checkpoint-fe118fa9fbfccb79.xml (deflated 67%) 2024-11-01T18:16:13.3047954Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-5a67956f946117f3.xml (deflated 36%) 2024-11-01T18:16:13.3049741Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-55c2e72ec365fcd1.xml (deflated 36%) 2024-11-01T18:16:13.3051625Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-d2310f2230371572.xml (deflated 36%) 2024-11-01T18:16:13.3053398Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-1e0a28678e41b2dc.xml (deflated 37%) 2024-11-01T18:16:13.3055177Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-8f7190b25c539f9d.xml (deflated 35%) 2024-11-01T18:16:13.3056947Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-b46df76996cb1977.xml (deflated 37%) 2024-11-01T18:16:13.3058950Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-9429689a399cc3a8.xml (deflated 36%) 2024-11-01T18:16:13.3060746Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-2945297d9fcf6c1b.xml (deflated 35%) 2024-11-01T18:16:13.3062508Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-f272d6a0fb0e16e7.xml (deflated 36%) 2024-11-01T18:16:13.3064282Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-dffc42d234c99070.xml (deflated 36%) 2024-11-01T18:16:13.3066076Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-5f95abf243acbd88.xml (deflated 36%) 2024-11-01T18:16:13.3067855Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-6b4f41b3f55e6816.xml (deflated 36%) 2024-11-01T18:16:13.3069623Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-4d108f1443d81a66.xml (deflated 36%) 2024-11-01T18:16:13.3071397Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-8126799b56665e3c.xml (deflated 36%) 2024-11-01T18:16:13.3073248Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-94b26d5f1d737db4.xml (deflated 35%) 2024-11-01T18:16:13.3075012Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-754b79795d933218.xml (deflated 36%) 2024-11-01T18:16:13.3076769Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-2003cdf3debeca26.xml (deflated 36%) 2024-11-01T18:16:13.3078652Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-d3054c8909849d4a.xml (deflated 36%) 2024-11-01T18:16:13.3080425Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-51929e0d6a52c3b6.xml (deflated 36%) 2024-11-01T18:16:13.3082207Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-ea8d4348f329c075.xml (deflated 35%) 2024-11-01T18:16:13.3083984Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-bfe8ba24db3f5dcc.xml (deflated 36%) 2024-11-01T18:16:13.3085768Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-9c3ec7bea335be44.xml (deflated 35%) 2024-11-01T18:16:13.3087543Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-c253b6ab561eac3e.xml (deflated 36%) 2024-11-01T18:16:13.3089310Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-73a99d275873e9c8.xml (deflated 35%) 2024-11-01T18:16:13.3091081Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-a6519ddfeb9d5f01.xml (deflated 35%) 2024-11-01T18:16:13.3092938Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-1fe4cf6fcc82e28a.xml (deflated 35%) 2024-11-01T18:16:13.3094714Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-b6d95397ee982ae4.xml (deflated 35%) 2024-11-01T18:16:13.3096460Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-b7757943186ca216.xml (deflated 35%) 2024-11-01T18:16:13.3098206Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-16fc6e9511a4b271.xml (deflated 88%) 2024-11-01T18:16:13.3100176Z adding: test/test-reports/python-pytest/distributed._shard.sharded_optim.test_sharded_optim/distributed._shard.sharded_optim.test_sharded_optim-095bc2d6e7560110.xml (deflated 67%) 2024-11-01T18:16:13.3102375Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_tensor_ops/distributed._shard.sharded_tensor.ops.test_tensor_ops-9e509f5f6096445c.xml (deflated 83%) 2024-11-01T18:16:13.3104414Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6c791b202a903db6.xml (deflated 35%) 2024-11-01T18:16:13.3106284Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-63b17597c0946f87.xml (deflated 36%) 2024-11-01T18:16:13.3108161Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-95de0b154a9739b7.xml (deflated 37%) 2024-11-01T18:16:13.3110039Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-73402032cb133c1e.xml (deflated 35%) 2024-11-01T18:16:13.3111907Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4fc61df1bb8d662d.xml (deflated 36%) 2024-11-01T18:16:13.3113845Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0416dabdee2ca036.xml (deflated 35%) 2024-11-01T18:16:13.3115722Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-feedf1e0b6b0bc86.xml (deflated 35%) 2024-11-01T18:16:13.3117607Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b35bac74d1b9358c.xml (deflated 36%) 2024-11-01T18:16:13.3119490Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0e5cc73c0d799be6.xml (deflated 36%) 2024-11-01T18:16:13.3121420Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-46139f983f8459c4.xml (deflated 35%) 2024-11-01T18:16:13.3123295Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-cb366df50cef3c8e.xml (deflated 35%) 2024-11-01T18:16:13.3125177Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-eeb159746358c6f4.xml (deflated 35%) 2024-11-01T18:16:13.3127044Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-cf823a97835ed541.xml (deflated 36%) 2024-11-01T18:16:13.3128917Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-668516cdc7c84775.xml (deflated 36%) 2024-11-01T18:16:13.3130787Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-908e23b2bee1bc68.xml (deflated 35%) 2024-11-01T18:16:13.3132647Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e809e3d59a498ec5.xml (deflated 36%) 2024-11-01T18:16:13.3134578Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-9030caa2faed0ef6.xml (deflated 35%) 2024-11-01T18:16:13.3136455Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c7ae7298ca19b3a4.xml (deflated 56%) 2024-11-01T18:16:13.3138328Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e3ee3df0a53e85a5.xml (deflated 35%) 2024-11-01T18:16:13.3140201Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e8c161fc0ae2f8d7.xml (deflated 35%) 2024-11-01T18:16:13.3142073Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c998a4272e8c0819.xml (deflated 34%) 2024-11-01T18:16:13.3143949Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d8be25c0d7fd0ca1.xml (deflated 35%) 2024-11-01T18:16:13.3145826Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c7d291a0d3d2b1ad.xml (deflated 35%) 2024-11-01T18:16:13.3147693Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3f9847a1d60af52a.xml (deflated 35%) 2024-11-01T18:16:13.3149561Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-cf2702372a30afbb.xml (deflated 35%) 2024-11-01T18:16:13.3151411Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-26266357294ab6c0.xml (deflated 36%) 2024-11-01T18:16:13.3153362Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-2f7debaef4e83fef.xml (deflated 38%) 2024-11-01T18:16:13.3155239Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0159c7330415972a.xml (deflated 36%) 2024-11-01T18:16:13.3157104Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0df43f2898a5481b.xml (deflated 36%) 2024-11-01T18:16:13.3159141Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-afd2f48b88f90f1b.xml (deflated 36%) 2024-11-01T18:16:13.3161029Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6f1cb2e905f66ad2.xml (deflated 35%) 2024-11-01T18:16:13.3163029Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0da4f919a3f011fa.xml (deflated 36%) 2024-11-01T18:16:13.3164912Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-796a8db91601dcae.xml (deflated 35%) 2024-11-01T18:16:13.3166792Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-dae6aefed8fe41f0.xml (deflated 35%) 2024-11-01T18:16:13.3168645Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-38839162a0ca6b81.xml (deflated 36%) 2024-11-01T18:16:13.3170506Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-380bfa4f3d2653d3.xml (deflated 36%) 2024-11-01T18:16:13.3172370Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-91a2ce9d696cf2af.xml (deflated 36%) 2024-11-01T18:16:13.3174249Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-dde1eb1dc966f8cf.xml (deflated 35%) 2024-11-01T18:16:13.3176126Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-86d163a019c1c089.xml (deflated 35%) 2024-11-01T18:16:13.3178078Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c774a99ef07034d3.xml (deflated 35%) 2024-11-01T18:16:13.3179954Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8bb162891a18fcc3.xml (deflated 34%) 2024-11-01T18:16:13.3181814Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-32e4a5009425843c.xml (deflated 36%) 2024-11-01T18:16:13.3183682Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-95d50e3996b09750.xml (deflated 35%) 2024-11-01T18:16:13.3185547Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-10add4486281b154.xml (deflated 35%) 2024-11-01T18:16:13.3187407Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-5aaa5de2f28c5c12.xml (deflated 34%) 2024-11-01T18:16:13.3189280Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-5620ed5ffbeb5403.xml (deflated 34%) 2024-11-01T18:16:13.3191149Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-75f68667dbce002b.xml (deflated 34%) 2024-11-01T18:16:13.3193114Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-46a37c94002dacce.xml (deflated 34%) 2024-11-01T18:16:13.3194993Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-603c90799e54a152.xml (deflated 35%) 2024-11-01T18:16:13.3196945Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-052c780ef3b6c75c.xml (deflated 35%) 2024-11-01T18:16:13.3198833Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-9d95a17d39fb4dde.xml (deflated 35%) 2024-11-01T18:16:13.3200715Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-9ed2daa49dcd218a.xml (deflated 35%) 2024-11-01T18:16:13.3202584Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b03409273bd98270.xml (deflated 35%) 2024-11-01T18:16:13.3204439Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-589704196e785e9c.xml (deflated 35%) 2024-11-01T18:16:13.3206357Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-9b9d4cb48393a6b1.xml (deflated 36%) 2024-11-01T18:16:13.3208242Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e9fa60eb92bbe6dd.xml (deflated 36%) 2024-11-01T18:16:13.3210120Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6c3c996ff29920a7.xml (deflated 34%) 2024-11-01T18:16:13.3211988Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d03fd0014868ff4e.xml (deflated 35%) 2024-11-01T18:16:13.3213872Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d1ae48ca42dee99d.xml (deflated 35%) 2024-11-01T18:16:13.3215749Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-76fdf9ba039f57aa.xml (deflated 35%) 2024-11-01T18:16:13.3217629Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-9aac7e1cb772efeb.xml (deflated 35%) 2024-11-01T18:16:13.3219567Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-058afb8b1bd7d48f.xml (deflated 36%) 2024-11-01T18:16:13.3221438Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-deb14573e2efc1d3.xml (deflated 35%) 2024-11-01T18:16:13.3223312Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b19ac74dae220619.xml (deflated 35%) 2024-11-01T18:16:13.3225177Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-ffe7c2a80e9f0b63.xml (deflated 35%) 2024-11-01T18:16:13.3227074Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-76d99c5cd4bf7615.xml (deflated 37%) 2024-11-01T18:16:13.3228945Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-24554ea141b60faa.xml (deflated 35%) 2024-11-01T18:16:13.3230825Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7ac5e3ec7474e7f5.xml (deflated 35%) 2024-11-01T18:16:13.3232700Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a249d08f811a8690.xml (deflated 35%) 2024-11-01T18:16:13.3234626Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c4add1ddf7cb9b32.xml (deflated 35%) 2024-11-01T18:16:13.3236500Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0e52114cdbb4293f.xml (deflated 35%) 2024-11-01T18:16:13.3238382Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a8cfbdea1beed970.xml (deflated 36%) 2024-11-01T18:16:13.3240262Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a807ff8392c086bc.xml (deflated 36%) 2024-11-01T18:16:13.3242115Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f4082733ee1f3fa1.xml (deflated 36%) 2024-11-01T18:16:13.3243979Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-519ea228a67accac.xml (deflated 36%) 2024-11-01T18:16:13.3245853Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3f8e4e1fba4914c7.xml (deflated 36%) 2024-11-01T18:16:13.3247786Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e63c6038508a7b61.xml (deflated 37%) 2024-11-01T18:16:13.3249652Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-85d9412a2e9cd2c0.xml (deflated 37%) 2024-11-01T18:16:13.3251529Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-61af3d71b395831b.xml (deflated 37%) 2024-11-01T18:16:13.3253397Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7ea9c41620587b2a.xml (deflated 38%) 2024-11-01T18:16:13.3255262Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c18e7c3d184605d4.xml (deflated 38%) 2024-11-01T18:16:13.3257129Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-94b8896b7a523e2b.xml (deflated 37%) 2024-11-01T18:16:13.3259157Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-cb5c71baea99ca5f.xml (deflated 37%) 2024-11-01T18:16:13.3261132Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-250b9954e41e4049.xml (deflated 37%) 2024-11-01T18:16:13.3262997Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1d5d15b799af85a2.xml (deflated 37%) 2024-11-01T18:16:13.3264875Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d30c45bfcf494ad6.xml (deflated 36%) 2024-11-01T18:16:13.3266747Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0ea3226813a79b74.xml (deflated 35%) 2024-11-01T18:16:13.3268613Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-16df54700956b411.xml (deflated 35%) 2024-11-01T18:16:13.3270482Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4db0bcfe384aff3d.xml (deflated 35%) 2024-11-01T18:16:13.3272350Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f26f78ee107a31c9.xml (deflated 36%) 2024-11-01T18:16:13.3274296Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-298fa80ad6b1afcc.xml (deflated 36%) 2024-11-01T18:16:13.3276167Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8e2a45652a610a5a.xml (deflated 35%) 2024-11-01T18:16:13.3278018Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-50309db020f1fb3e.xml (deflated 36%) 2024-11-01T18:16:13.3279884Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-cab5dc110c070e33.xml (deflated 35%) 2024-11-01T18:16:13.3281761Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c718480852daa3db.xml (deflated 35%) 2024-11-01T18:16:13.3283629Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-007901dc402c4de7.xml (deflated 35%) 2024-11-01T18:16:13.3285490Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4283975549a629ac.xml (deflated 35%) 2024-11-01T18:16:13.3287352Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6a3f2586f0522738.xml (deflated 36%) 2024-11-01T18:16:13.3289299Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1d8a887127e5f527.xml (deflated 35%) 2024-11-01T18:16:13.3291178Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-98e70ab98f36674e.xml (deflated 35%) 2024-11-01T18:16:13.3293055Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7e3a7ddbac95e408.xml (deflated 36%) 2024-11-01T18:16:13.3294920Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6e65599133d41bf6.xml (deflated 36%) 2024-11-01T18:16:13.3296773Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f4bcdad3aa605960.xml (deflated 35%) 2024-11-01T18:16:13.3298646Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a6a28c799c905809.xml (deflated 35%) 2024-11-01T18:16:13.3300529Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-5edfff44ae9842e4.xml (deflated 35%) 2024-11-01T18:16:13.3302405Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-809ba36d02721755.xml (deflated 35%) 2024-11-01T18:16:13.3304331Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-82b4b46d619ecb5d.xml (deflated 35%) 2024-11-01T18:16:13.3306203Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d07dc1839d85e9b2.xml (deflated 35%) 2024-11-01T18:16:13.3308075Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-beeda086b824f437.xml (deflated 35%) 2024-11-01T18:16:13.3309943Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-ff29610da4c8e107.xml (deflated 36%) 2024-11-01T18:16:13.3311809Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-18be799e72f626f6.xml (deflated 35%) 2024-11-01T18:16:13.3313750Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1281d4566c5b81a5.xml (deflated 36%) 2024-11-01T18:16:13.3315600Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-58674495d59bf14a.xml (deflated 34%) 2024-11-01T18:16:13.3317460Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3a8830db0e186821.xml (deflated 35%) 2024-11-01T18:16:13.3319318Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-764619e6f5484450.xml (deflated 35%) 2024-11-01T18:16:13.3321189Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-45a80f3f81af4dc6.xml (deflated 35%) 2024-11-01T18:16:13.3323059Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c9b5722eb393e7cd.xml (deflated 36%) 2024-11-01T18:16:13.3324937Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-90f3ff4f74c9597c.xml (deflated 36%) 2024-11-01T18:16:13.3326805Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-2a25d287d614d9bf.xml (deflated 36%) 2024-11-01T18:16:13.3328671Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-18495d2af6ea6cdc.xml (deflated 36%) 2024-11-01T18:16:13.3330534Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0680ea29aa5efe7d.xml (deflated 36%) 2024-11-01T18:16:13.3332474Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-360bb06b991d74ba.xml (deflated 36%) 2024-11-01T18:16:13.3334340Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4a127e214c55ef4c.xml (deflated 36%) 2024-11-01T18:16:13.3336208Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-fae1af038f79d77f.xml (deflated 35%) 2024-11-01T18:16:13.3338081Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e1b8e98279d75112.xml (deflated 35%) 2024-11-01T18:16:13.3339946Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d2fcb4e087a75513.xml (deflated 35%) 2024-11-01T18:16:13.3341815Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d6f6742d6d433b2c.xml (deflated 35%) 2024-11-01T18:16:13.3343680Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-631b0fe9d9dcd026.xml (deflated 35%) 2024-11-01T18:16:13.3345601Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-610712fa5a95c87b.xml (deflated 36%) 2024-11-01T18:16:13.3347466Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1e7aae6b9f948425.xml (deflated 36%) 2024-11-01T18:16:13.3349339Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-ccf5b13d239fe356.xml (deflated 35%) 2024-11-01T18:16:13.3351192Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8ff6f10bb2c11638.xml (deflated 35%) 2024-11-01T18:16:13.3353118Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-853754c7fe8a14fe.xml (deflated 35%) 2024-11-01T18:16:13.3354989Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-2f1a4500f78df932.xml (deflated 35%) 2024-11-01T18:16:13.3356867Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c01d4875a211fbb6.xml (deflated 35%) 2024-11-01T18:16:13.3358908Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-760bab14299d97fe.xml (deflated 36%) 2024-11-01T18:16:13.3360789Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-62d9dc87e66c01aa.xml (deflated 35%) 2024-11-01T18:16:13.3362665Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c25ef0edbc80aadd.xml (deflated 35%) 2024-11-01T18:16:13.3364544Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3099c933e01b5254.xml (deflated 35%) 2024-11-01T18:16:13.3366413Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3e6f581b58711acf.xml (deflated 34%) 2024-11-01T18:16:13.3368267Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-394401978915670b.xml (deflated 34%) 2024-11-01T18:16:13.3370194Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1502f5e12a2225b2.xml (deflated 34%) 2024-11-01T18:16:13.3372071Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-390faaf2dd3c91fe.xml (deflated 35%) 2024-11-01T18:16:13.3374360Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8d927cd2aaf63df4.xml (deflated 35%) 2024-11-01T18:16:13.3376247Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c203ab521a1f2b55.xml (deflated 35%) 2024-11-01T18:16:13.3378127Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0fe27d64347f558b.xml (deflated 35%) 2024-11-01T18:16:13.3380000Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-227f4f71cc7d4d16.xml (deflated 35%) 2024-11-01T18:16:13.3381881Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f923e901a9b68521.xml (deflated 35%) 2024-11-01T18:16:13.3383759Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8b2efe5d534faa4f.xml (deflated 35%) 2024-11-01T18:16:13.3385630Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b6f0b2d17fcba2e1.xml (deflated 35%) 2024-11-01T18:16:13.3387486Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-55a17bef17412760.xml (deflated 36%) 2024-11-01T18:16:13.3389434Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-5b86146ad799dca2.xml (deflated 35%) 2024-11-01T18:16:13.3391311Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a04c22a29fbae091.xml (deflated 35%) 2024-11-01T18:16:13.3393262Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c4a425ee0da630e7.xml (deflated 35%) 2024-11-01T18:16:13.3395148Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0650b2ccf966636d.xml (deflated 35%) 2024-11-01T18:16:13.3397020Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-bc879e0775b9f8da.xml (deflated 35%) 2024-11-01T18:16:13.3398889Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f0026740e780249c.xml (deflated 35%) 2024-11-01T18:16:13.3400753Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-73cc9d2b80d2773c.xml (deflated 35%) 2024-11-01T18:16:13.3402624Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4ca67885b2034c7f.xml (deflated 34%) 2024-11-01T18:16:13.3404495Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b986eef80af9a2cd.xml (deflated 36%) 2024-11-01T18:16:13.3406363Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-49cb8c7365aad2ea.xml (deflated 36%) 2024-11-01T18:16:13.3408234Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-cdad70c1225a7b14.xml (deflated 35%) 2024-11-01T18:16:13.3410243Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-946cb8f3740035ad.xml (deflated 35%) 2024-11-01T18:16:13.3412112Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b48ac15a0d68abf3.xml (deflated 35%) 2024-11-01T18:16:13.3413991Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b513939d1d512ae6.xml (deflated 35%) 2024-11-01T18:16:13.3415849Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7a8bea5230c2c874.xml (deflated 35%) 2024-11-01T18:16:13.3417786Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-160a8b25e8b0809a.xml (deflated 34%) 2024-11-01T18:16:13.3419665Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f6fe1a9da421602d.xml (deflated 35%) 2024-11-01T18:16:13.3421542Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-ce9cdfcb43894df5.xml (deflated 35%) 2024-11-01T18:16:13.3423420Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-fc9b4d2a28e88b26.xml (deflated 35%) 2024-11-01T18:16:13.3425291Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-2d91a1c804126d15.xml (deflated 35%) 2024-11-01T18:16:13.3427153Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c7560efa9124468d.xml (deflated 34%) 2024-11-01T18:16:13.3429022Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-2ea533fcb3a81e64.xml (deflated 35%) 2024-11-01T18:16:13.3430951Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e2827e247844e864.xml (deflated 35%) 2024-11-01T18:16:13.3432892Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c8a9b4a3b2489105.xml (deflated 35%) 2024-11-01T18:16:13.3434750Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-ea4566b929b57a30.xml (deflated 35%) 2024-11-01T18:16:13.3436620Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-21df7f20b21210d2.xml (deflated 34%) 2024-11-01T18:16:13.3438490Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1383dba352a673f2.xml (deflated 35%) 2024-11-01T18:16:13.3440357Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e8da61dda8aac7e7.xml (deflated 35%) 2024-11-01T18:16:13.3442235Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-9e0faf6aaa3b93c6.xml (deflated 35%) 2024-11-01T18:16:13.3444110Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1e887cf1fe9d8f71.xml (deflated 36%) 2024-11-01T18:16:13.3445986Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-ac31fe4dfedbb858.xml (deflated 35%) 2024-11-01T18:16:13.3447865Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d32ed1805aea57c7.xml (deflated 37%) 2024-11-01T18:16:13.3449740Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f9f2fbbd3a77948a.xml (deflated 36%) 2024-11-01T18:16:13.3451603Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b9b1934e93d7ed47.xml (deflated 36%) 2024-11-01T18:16:13.3453473Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f56d4bfbdbd5d23a.xml (deflated 37%) 2024-11-01T18:16:13.3455339Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6d1c90607e241627.xml (deflated 37%) 2024-11-01T18:16:13.3457203Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-36299492c84b9074.xml (deflated 36%) 2024-11-01T18:16:13.3459342Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7f257739cfea9a92.xml (deflated 37%) 2024-11-01T18:16:13.3461219Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0330f88636917d69.xml (deflated 38%) 2024-11-01T18:16:13.3463088Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0d5e0c89fed05bff.xml (deflated 38%) 2024-11-01T18:16:13.3464964Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-ef40b86680f591c4.xml (deflated 36%) 2024-11-01T18:16:13.3466836Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8f62f68b3f8dbf72.xml (deflated 36%) 2024-11-01T18:16:13.3468697Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-10f9509897dc9781.xml (deflated 36%) 2024-11-01T18:16:13.3470548Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a7e10ff81b7861f4.xml (deflated 36%) 2024-11-01T18:16:13.3472604Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-74ae175cd1852622.xml (deflated 36%) 2024-11-01T18:16:13.3474558Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-800f76b5518b8b10.xml (deflated 37%) 2024-11-01T18:16:13.3476430Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-304990a57c6fd4d4.xml (deflated 37%) 2024-11-01T18:16:13.3478300Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4fd09b703362965e.xml (deflated 37%) 2024-11-01T18:16:13.3480173Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-2459b8294be36846.xml (deflated 36%) 2024-11-01T18:16:13.3482048Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-598a932daf3a3150.xml (deflated 37%) 2024-11-01T18:16:13.3483919Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-67b309d20548c710.xml (deflated 37%) 2024-11-01T18:16:13.3485983Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_binary_cmp/distributed._shard.sharded_tensor.ops.test_binary_cmp-dbe61cb896029e9b.xml (deflated 81%) 2024-11-01T18:16:13.3488142Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_init/distributed._shard.sharded_tensor.ops.test_init-208d0ac32b13cb17.xml (deflated 76%) 2024-11-01T18:16:13.3490183Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_mixed_precision/distributed.fsdp.test_fsdp_mixed_precision-e8a9ca0da6ddd007.xml (deflated 95%) 2024-11-01T18:16:13.3491970Z adding: test/test-reports/python-pytest/distributed.test_launcher/distributed.test_launcher-f016c21cd9fcb94e.xml (deflated 40%) 2024-11-01T18:16:13.3493897Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_sharded_tensor/distributed._shard.sharded_tensor.test_sharded_tensor-8ed15f487be89927.xml (deflated 95%) 2024-11-01T18:16:13.3495862Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-df15098d4eccf6cf.xml (deflated 36%) 2024-11-01T18:16:13.3497598Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-7a053e87b312921e.xml (deflated 44%) 2024-11-01T18:16:13.3499327Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-677255e2296a29d0.xml (deflated 43%) 2024-11-01T18:16:13.3501132Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-b3fc150927fcd814.xml (deflated 44%) 2024-11-01T18:16:13.3502869Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-cda68a3ad603d04f.xml (deflated 44%) 2024-11-01T18:16:13.3504597Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-9613c2d6c4a0a885.xml (deflated 44%) 2024-11-01T18:16:13.3506310Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-ed5444d3065fb6dd.xml (deflated 44%) 2024-11-01T18:16:13.3508042Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-9f5aa5842a7507fe.xml (deflated 44%) 2024-11-01T18:16:13.3509768Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-9196217c784cd9a3.xml (deflated 44%) 2024-11-01T18:16:13.3511499Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-4cb33b24b38c6aa8.xml (deflated 44%) 2024-11-01T18:16:13.3513300Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-4b7b8cbe5b17d4a9.xml (deflated 44%) 2024-11-01T18:16:13.3514997Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-58d95614f2a63b70.xml (deflated 37%) 2024-11-01T18:16:13.3516503Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-716491542fe38b05.xml (deflated 37%) 2024-11-01T18:16:13.3518012Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d878ffcc6578ffeb.xml (deflated 40%) 2024-11-01T18:16:13.3519531Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-ee91f045a0a7d980.xml (deflated 36%) 2024-11-01T18:16:13.3521023Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d6200506ff1b568e.xml (deflated 37%) 2024-11-01T18:16:13.3522535Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-a67c529c5be34e9f.xml (deflated 36%) 2024-11-01T18:16:13.3524048Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-feea728d38281e82.xml (deflated 36%) 2024-11-01T18:16:13.3525555Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-3aad337c91743ede.xml (deflated 37%) 2024-11-01T18:16:13.3527070Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-2da5f69f3176994c.xml (deflated 36%) 2024-11-01T18:16:13.3528575Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-ba690e67cf31e71c.xml (deflated 37%) 2024-11-01T18:16:13.3530084Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-f2fec4961339cb01.xml (deflated 36%) 2024-11-01T18:16:13.3531596Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5ae1a368237fdebf.xml (deflated 37%) 2024-11-01T18:16:13.3533089Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e40ea6167409aff1.xml (deflated 36%) 2024-11-01T18:16:13.3534598Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-06afc075b98fa3a4.xml (deflated 37%) 2024-11-01T18:16:13.3536097Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-63d90967bb8f06e3.xml (deflated 36%) 2024-11-01T18:16:13.3537607Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-9eb7d80b75171df7.xml (deflated 35%) 2024-11-01T18:16:13.3539120Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-6acda11e45f792e2.xml (deflated 36%) 2024-11-01T18:16:13.3540631Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-2fd77ab3212c82e0.xml (deflated 35%) 2024-11-01T18:16:13.3542226Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-a3219426a13345a1.xml (deflated 35%) 2024-11-01T18:16:13.3543741Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-b133aacab16dee2b.xml (deflated 36%) 2024-11-01T18:16:13.3545242Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-939ece88d1d00be4.xml (deflated 35%) 2024-11-01T18:16:13.3546753Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-b1a6bf72880c6519.xml (deflated 36%) 2024-11-01T18:16:13.3548262Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-b73fda8bb8a72c0b.xml (deflated 35%) 2024-11-01T18:16:13.3549778Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-9d6ffdc71367fa0f.xml (deflated 36%) 2024-11-01T18:16:13.3551291Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-c0e5ed00b0a4f71f.xml (deflated 40%) 2024-11-01T18:16:13.3552935Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d136ab7785d3eaec.xml (deflated 36%) 2024-11-01T18:16:13.3554454Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-37938373920bc882.xml (deflated 36%) 2024-11-01T18:16:13.3556028Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-4f2ee3e676277038.xml (deflated 35%) 2024-11-01T18:16:13.3557521Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-a2e42be4ab4cd5d3.xml (deflated 34%) 2024-11-01T18:16:13.3559179Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-8bb21320e929344f.xml (deflated 36%) 2024-11-01T18:16:13.3560685Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e419968da6321491.xml (deflated 36%) 2024-11-01T18:16:13.3562202Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-4c94d3b082bfda71.xml (deflated 36%) 2024-11-01T18:16:13.3563705Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5c5156af5cd6982e.xml (deflated 35%) 2024-11-01T18:16:13.3565216Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-cf10055408071180.xml (deflated 36%) 2024-11-01T18:16:13.3566710Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-702e62873316fdcf.xml (deflated 36%) 2024-11-01T18:16:13.3568204Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-019fa4efbbf8993f.xml (deflated 35%) 2024-11-01T18:16:13.3569704Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-612a6d63e198450f.xml (deflated 35%) 2024-11-01T18:16:13.3571196Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-42055b6caed05170.xml (deflated 34%) 2024-11-01T18:16:13.3572708Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-ce363cb225dc8274.xml (deflated 35%) 2024-11-01T18:16:13.3574210Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-6774ddf6f374405c.xml (deflated 35%) 2024-11-01T18:16:13.3575719Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-46683388c17a1daf.xml (deflated 40%) 2024-11-01T18:16:13.3577224Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-85c199f0571881e5.xml (deflated 34%) 2024-11-01T18:16:13.3578728Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d9fef213c07ee006.xml (deflated 34%) 2024-11-01T18:16:13.3580221Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-cc110fbe35fef0fe.xml (deflated 33%) 2024-11-01T18:16:13.3581733Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-fba74bc4452fe222.xml (deflated 34%) 2024-11-01T18:16:13.3583340Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-2f4a6a7f9b3cc873.xml (deflated 35%) 2024-11-01T18:16:13.3584859Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-9db27d8db1c9ea31.xml (deflated 35%) 2024-11-01T18:16:13.3586385Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-af4053033c2cafa3.xml (deflated 35%) 2024-11-01T18:16:13.3587896Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-fa8b1b0d6b2a7846.xml (deflated 35%) 2024-11-01T18:16:13.3589414Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-2cd1bbde5a0447ba.xml (deflated 35%) 2024-11-01T18:16:13.3590923Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d5312d843d68117d.xml (deflated 35%) 2024-11-01T18:16:13.3592419Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-07573b1763c8a932.xml (deflated 34%) 2024-11-01T18:16:13.3593997Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5426a4b6df99ca68.xml (deflated 34%) 2024-11-01T18:16:13.3595598Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-3c9e4cf96f25787a.xml (deflated 36%) 2024-11-01T18:16:13.3597106Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-bac7e30b89f1fe7e.xml (deflated 36%) 2024-11-01T18:16:13.3598614Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-11d26927f85b6c04.xml (deflated 35%) 2024-11-01T18:16:13.3600120Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e27745dd753f3e4a.xml (deflated 35%) 2024-11-01T18:16:13.3601624Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-7e86d23f6b244fa2.xml (deflated 35%) 2024-11-01T18:16:13.3603132Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-7bc974e2e9746abb.xml (deflated 36%) 2024-11-01T18:16:13.3604633Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-a8cf5b9410494264.xml (deflated 35%) 2024-11-01T18:16:13.3606144Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-0f530441f23e8c99.xml (deflated 37%) 2024-11-01T18:16:13.3607645Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-58328a67f6316030.xml (deflated 35%) 2024-11-01T18:16:13.3609141Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-03e0bd912bb7d2d6.xml (deflated 35%) 2024-11-01T18:16:13.3610652Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-f028bbc262657bbe.xml (deflated 34%) 2024-11-01T18:16:13.3612157Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d0ccc37772ee4592.xml (deflated 35%) 2024-11-01T18:16:13.3613665Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5f44d6d6ecb1b700.xml (deflated 34%) 2024-11-01T18:16:13.3615168Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-435b18b43ee83f9a.xml (deflated 35%) 2024-11-01T18:16:13.3616656Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-02933167142d7908.xml (deflated 36%) 2024-11-01T18:16:13.3618160Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-2faa4e8aace8cb0e.xml (deflated 35%) 2024-11-01T18:16:13.3619667Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d7946e416c41d275.xml (deflated 35%) 2024-11-01T18:16:13.3621174Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e9427e7b2672661b.xml (deflated 35%) 2024-11-01T18:16:13.3622666Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-c153e324896ebc13.xml (deflated 35%) 2024-11-01T18:16:13.3624228Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-27ddf78f02be42f2.xml (deflated 35%) 2024-11-01T18:16:13.3625738Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-1022c7d7ed063f01.xml (deflated 35%) 2024-11-01T18:16:13.3627241Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-fbdefaa2324f7b29.xml (deflated 35%) 2024-11-01T18:16:13.3628735Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-bb9822fdb148ef52.xml (deflated 35%) 2024-11-01T18:16:13.3630243Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-05ba69d20127dc80.xml (deflated 35%) 2024-11-01T18:16:13.3631742Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-1eb46664ee2cf515.xml (deflated 35%) 2024-11-01T18:16:13.3633320Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-9d9a4c9f28806639.xml (deflated 36%) 2024-11-01T18:16:13.3634828Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-a379cdc8602714ba.xml (deflated 35%) 2024-11-01T18:16:13.3636418Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d45e30f25f75f014.xml (deflated 35%) 2024-11-01T18:16:13.3637929Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-282dceb9aa722445.xml (deflated 35%) 2024-11-01T18:16:13.3639435Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-3aec8cd3ac070a69.xml (deflated 35%) 2024-11-01T18:16:13.3640935Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-3ae758a458e70de8.xml (deflated 35%) 2024-11-01T18:16:13.3642441Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-a336a8f43073bfd7.xml (deflated 36%) 2024-11-01T18:16:13.3643953Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-ff3de7e6cd2bc1c0.xml (deflated 33%) 2024-11-01T18:16:13.3645469Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-4af2cdd76fcffe96.xml (deflated 33%) 2024-11-01T18:16:13.3646985Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e44e2621dd4e1d8b.xml (deflated 35%) 2024-11-01T18:16:13.3648606Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-c1625e06da3a164e.xml (deflated 45%) 2024-11-01T18:16:13.3650313Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-46e6fad0457344b6.xml (deflated 46%) 2024-11-01T18:16:13.3652017Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-54a355c5592563f9.xml (deflated 46%) 2024-11-01T18:16:13.3653701Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-151128a6cf1cc41a.xml (deflated 45%) 2024-11-01T18:16:13.3655402Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-332dc88f48737f51.xml (deflated 45%) 2024-11-01T18:16:13.3657103Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_ucc/distributed.test_c10d_spawn_ucc-b06dd42d0566f089.xml (deflated 46%) 2024-11-01T18:16:13.3658988Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-e6ac28f46172e55c.xml (deflated 46%) 2024-11-01T18:16:13.3660750Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-cbfd2eb28bfb11dc.xml (deflated 45%) 2024-11-01T18:16:13.3662496Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-58aeb5a4a375a896.xml (deflated 46%) 2024-11-01T18:16:13.3664341Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-f5bcacdd2b8b8438.xml (deflated 46%) 2024-11-01T18:16:13.3666080Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-fb63550ae4077b17.xml (deflated 46%) 2024-11-01T18:16:13.3667814Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-2514920be43a0489.xml (deflated 46%) 2024-11-01T18:16:13.3669524Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-1f2b7c6a1d7423b1.xml (deflated 46%) 2024-11-01T18:16:13.3671258Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-2ae7a065b7040ff8.xml (deflated 45%) 2024-11-01T18:16:13.3673050Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_nccl/distributed.test_c10d_spawn_nccl-773ac0d575a57e16.xml (deflated 45%) 2024-11-01T18:16:13.3674871Z adding: test/test-reports/python-pytest/distributed.elastic.events.lib_test/distributed.elastic.events.lib_test-2b498340882108b7.xml (deflated 76%) 2024-11-01T18:16:13.3676770Z adding: test/test-reports/python-pytest/distributed.elastic.metrics.api_test/distributed.elastic.metrics.api_test-754678b6fd2cbd04.xml (deflated 59%) 2024-11-01T18:16:13.3678904Z adding: test/test-reports/python-pytest/distributed.elastic.multiprocessing.api_test/distributed.elastic.multiprocessing.api_test-7ed51c345799266b.xml (deflated 84%) 2024-11-01T18:16:13.3681039Z adding: test/test-reports/python-pytest/distributed.elastic.timer.local_timer_example/distributed.elastic.timer.local_timer_example-901cb88791113b5d.xml (deflated 51%) 2024-11-01T18:16:13.3683130Z adding: test/test-reports/python-pytest/distributed.elastic.timer.local_timer_test/distributed.elastic.timer.local_timer_test-673ca91b4eb3582c.xml (deflated 81%) 2024-11-01T18:16:13.3685201Z adding: test/test-reports/python-pytest/distributed.elastic.utils.distributed_test/distributed.elastic.utils.distributed_test-1b2a189d8c690780.xml (deflated 78%) 2024-11-01T18:16:13.3687232Z adding: test/test-reports/python-pytest/distributed.elastic.utils.logging_test/distributed.elastic.utils.logging_test-cedba09a0483eac9.xml (deflated 51%) 2024-11-01T18:16:13.3689165Z adding: test/test-reports/python-pytest/distributed.elastic.utils.util_test/distributed.elastic.utils.util_test-0fcf648e115ff9c1.xml (deflated 81%) 2024-11-01T18:16:13.3690732Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-6c0da635215d7a97.xml (deflated 64%) 2024-11-01T18:16:13.3716996Z ##[group]Run # Remove any previous usage logs if they exist 2024-11-01T18:16:13.3717655Z # Remove any previous usage logs if they exist 2024-11-01T18:16:13.3718145Z rm -f logs-*.zip 2024-11-01T18:16:13.3718826Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-11-01T18:16:13.3719641Z # so check to see if the file exists first 2024-11-01T18:16:13.3720152Z if [ -f 'usage_log.txt' ]; then 2024-11-01T18:16:13.3720681Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-11-01T18:16:13.3721171Z fi 2024-11-01T18:16:13.3721668Z if find "test/test-reports" -name "*.log" 2>/dev/null | grep -q .; then 2024-11-01T18:16:13.3722474Z  zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' 2024-11-01T18:16:13.3723038Z fi 2024-11-01T18:16:13.3732345Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T18:16:13.3732849Z env: 2024-11-01T18:16:13.3733116Z GIT_DEFAULT_BRANCH: main 2024-11-01T18:16:13.3733724Z DOCKER_CONTAINER_ID: ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e 2024-11-01T18:16:13.3734541Z FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_32396543718 2024-11-01T18:16:13.3735065Z ##[endgroup] 2024-11-01T18:16:13.3830053Z adding: usage_log.txt (deflated 92%) 2024-11-01T18:16:13.3946789Z adding: test/test-reports/distributed.elastic.utils.distributed_test_1.1_8b826889fcadb69d_.log (deflated 90%) 2024-11-01T18:16:13.3948187Z adding: test/test-reports/distributed._tensor.test_dtensor_compile_1.1_a469e7b066894dae_.log (deflated 81%) 2024-11-01T18:16:13.3949519Z adding: test/test-reports/distributed.elastic.utils.logging_test_1.1_a9336d3529114d58_.log (deflated 58%) 2024-11-01T18:16:13.3950818Z adding: test/test-reports/distributed.test_inductor_collectives_1.1_4353786c487e4573_.log (deflated 90%) 2024-11-01T18:16:13.3952376Z adding: test/test-reports/distributed.elastic.utils.util_test_1.1_d7b4b80707f843ba_.log (deflated 75%) 2024-11-01T18:16:13.3953746Z adding: test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_ee90816efebb1da2_.log (deflated 54%) 2024-11-01T18:16:13.3955011Z adding: test/test-reports/distributed.test_c10d_spawn_gloo_1.1_a7f6edfa8b5d96a5_.log (deflated 92%) 2024-11-01T18:16:13.3956306Z adding: test/test-reports/distributed.flight_recorder.test_fr_analysis_1.1_c81edbc1986bbb29_.log (deflated 55%) 2024-11-01T18:16:13.3957608Z adding: test/test-reports/distributed._tools.test_mod_tracker_1.1_16ce026496f28e59_.log (deflated 63%) 2024-11-01T18:16:13.3959014Z adding: test/test-reports/distributed.checkpoint.test_nested_dict_1.1_e22ef8621120214b_.log (deflated 59%) 2024-11-01T18:16:13.3969227Z adding: test/test-reports/distributed.test_store_1.1_f1be3c7d64aa3a06_.log (deflated 94%) 2024-11-01T18:16:13.3970481Z adding: test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_00614d8de49ee238_.log (deflated 55%) 2024-11-01T18:16:13.3971744Z adding: test/test-reports/distributed.test_c10d_spawn_ucc_1.1_b689a844060459e5_.log (deflated 90%) 2024-11-01T18:16:13.3973088Z adding: test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_a1aaaf109cded443_.log (deflated 90%) 2024-11-01T18:16:13.3974357Z adding: test/test-reports/distributed.test_c10d_nccl_1.1_7fc53c2a8ba43f89_.log (stored 0%) 2024-11-01T18:16:13.3975597Z adding: test/test-reports/distributed.pipelining.test_schedule_multiproc_1.1_d7d9026c87f2cd40_.log (deflated 2%) 2024-11-01T18:16:13.3976881Z adding: test/test-reports/distributed._tools.test_sac_ilp_1.1_f81764fda347aea3_.log (deflated 64%) 2024-11-01T18:16:13.3977964Z adding: test/test-reports/cpp.test_cpp_rpc_1.1_09ab296419a7fbf2_.log (deflated 78%) 2024-11-01T18:16:13.3979117Z adding: test/test-reports/distributed.pipelining.test_backward_1.1_8ce8d971dc553868_.log (deflated 66%) 2024-11-01T18:16:13.3980409Z adding: test/test-reports/distributed.pipelining.test_transformer_1.1_a0aac115c39999a5_.log (deflated 54%) 2024-11-01T18:16:13.3981684Z adding: test/test-reports/distributed._tools.test_sac_estimator_1.1_2886b7459565973d_.log (deflated 59%) 2024-11-01T18:16:13.3982978Z adding: test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_60c0aaa5133f4847_.log (deflated 52%) 2024-11-01T18:16:13.3984246Z adding: test/test-reports/distributed.pipelining.test_pipe_1.1_ac97b86dfe40f9f2_.log (deflated 62%) 2024-11-01T18:16:13.3985493Z adding: test/test-reports/distributed._tensor.test_common_rules_1.1_df1f450b2118e537_.log (deflated 81%) 2024-11-01T18:16:13.3986724Z adding: test/test-reports/distributed._tensor.test_attention_1.1_a3ea0e487f04bccc_.log (deflated 67%) 2024-11-01T18:16:13.3988014Z adding: test/test-reports/distributed.checkpoint.test_save_load_api_1.1_85c83615d47cc128_.log (deflated 54%) 2024-11-01T18:16:13.3989339Z adding: test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_46404614e0971eb0_.log (deflated 57%) 2024-11-01T18:16:13.3990631Z adding: test/test-reports/distributed.test_c10d_functional_native_1.1_a60bc979e6a4c6b6_.log (deflated 86%) 2024-11-01T18:16:13.3991914Z adding: test/test-reports/distributed._tools.test_runtime_estimator_1.1_0a7283d274c3c6b3_.log (deflated 59%) 2024-11-01T18:16:13.3993199Z adding: test/test-reports/distributed._tensor.test_api_1.1_08e79625c7df2a42_.log (deflated 78%) 2024-11-01T18:16:13.3994423Z adding: test/test-reports/distributed.pipelining.test_microbatch_1.1_23b407a2a50b255b_.log (deflated 58%) 2024-11-01T18:16:13.3995860Z adding: test/test-reports/distributed._tensor.test_experimental_ops_1.1_c2a35f2aaeac9ea2_.log (deflated 66%) 2024-11-01T18:16:13.3997143Z adding: test/test-reports/distributed._tensor.test_tensor_ops_1.1_530eebccad0742f6_.log (deflated 87%) 2024-11-01T18:16:13.3998482Z adding: test/test-reports/distributed.optim.test_zero_redundancy_optimizer_1.1_ea5bd3325b6a8bf4_.log (deflated 94%) 2024-11-01T18:16:13.3999940Z adding: test/test-reports/distributed._tensor.experimental.test_register_sharding_1.1_1ceedf54c093af65_.log (deflated 64%) 2024-11-01T18:16:13.4001263Z adding: test/test-reports/distributed._tensor.test_init_1.1_ab1369296e26e546_.log (deflated 76%) 2024-11-01T18:16:13.4002471Z adding: test/test-reports/distributed._tools.test_memory_tracker_1.1_7b995ee20278e400_.log (deflated 54%) 2024-11-01T18:16:13.4003744Z adding: test/test-reports/distributed._tensor.test_xla_integration_1.1_5d84593e13fedf1b_.log (deflated 66%) 2024-11-01T18:16:13.4004970Z adding: test/test-reports/distributed.test_c10d_spawn_nccl_1.1_b6d52629c6fc5411_.log (deflated 92%) 2024-11-01T18:16:13.4006285Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_f6769fe9317d31b5_.log (deflated 72%) 2024-11-01T18:16:13.4007673Z adding: test/test-reports/distributed.pipelining.test_stage_1.1_8e1167c086be5b0c_.log (deflated 2%) 2024-11-01T18:16:13.4008933Z adding: test/test-reports/distributed.pipelining.test_unflatten_1.1_fd702e8802ac731f_.log (deflated 54%) 2024-11-01T18:16:13.4010225Z adding: test/test-reports/distributed._composable.test_checkpoint_1.1_4b3cfe1805dfcb23_.log (deflated 69%) 2024-11-01T18:16:13.4011502Z adding: test/test-reports/distributed.checkpoint.test_fsspec_1.1_41962c754bb3eaf7_.log (deflated 57%) 2024-11-01T18:16:13.4012717Z adding: test/test-reports/distributed._shard.test_sharder_1.1_64eff4f11e4145c6_.log (deflated 59%) 2024-11-01T18:16:13.4014063Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_util_1.1_be6eac3a6366af29_.log (deflated 60%) 2024-11-01T18:16:13.4015436Z adding: test/test-reports/distributed.checkpoint.test_traverse_1.1_895540db10557f2a_.log (deflated 71%) 2024-11-01T18:16:13.4016731Z adding: test/test-reports/distributed.test_compute_comm_reordering_1.1_2148f20a2fdefc8e_.log (deflated 76%) 2024-11-01T18:16:13.4018008Z adding: test/test-reports/distributed._tensor.test_convolution_ops_1.1_8e9b172c3a17f6c4_.log (deflated 63%) 2024-11-01T18:16:13.4019250Z adding: test/test-reports/distributed._tensor.test_math_ops_1.1_b4ef298050f823a8_.log (deflated 83%) 2024-11-01T18:16:13.4020415Z adding: test/test-reports/distributed.test_c10d_ops_nccl_1.1_6668d062e4bba6af_.log (stored 0%) 2024-11-01T18:16:13.4021688Z adding: test/test-reports/distributed.checkpoint.test_dtensor_resharding_1.1_57063fd90a312c7c_.log (deflated 76%) 2024-11-01T18:16:13.4023180Z adding: test/test-reports/distributed.checkpoint.test_format_utils_1.1_1ddf99ddcc148d79_.log (deflated 86%) 2024-11-01T18:16:13.4024513Z adding: test/test-reports/distributed.checkpoint.test_tp_checkpoint_1.1_9ff78521a680d030_.log (deflated 63%) 2024-11-01T18:16:13.4025858Z adding: test/test-reports/distributed.checkpoint.fsdp.test_fsdp_dsd_1.1_444aedad98422d4a_.log (deflated 77%) 2024-11-01T18:16:13.4027180Z adding: test/test-reports/distributed._tools.test_fsdp2_mem_tracker_1.1_8423f211690d712e_.log (deflated 66%) 2024-11-01T18:16:13.4028440Z adding: test/test-reports/distributed._tensor.test_redistribute_1.1_e1f15058288238cc_.log (deflated 84%) 2024-11-01T18:16:13.4029706Z adding: test/test-reports/distributed.elastic.test_control_plane_1.1_9e6d4dcd4bae34fd_.log (deflated 73%) 2024-11-01T18:16:13.4031026Z adding: test/test-reports/distributed.tensor.parallel.test_tp_style_1.1_e6d8409a9aa7ab1d_.log (deflated 81%) 2024-11-01T18:16:13.4032279Z adding: test/test-reports/distributed.test_collective_utils_1.1_a5ffcda6ee014335_.log (stored 0%) 2024-11-01T18:16:13.4033744Z adding: test/test-reports/distributed._shard.sharding_plan.test_sharding_plan_1.1_86b7da3c28a21120_.log (deflated 66%) 2024-11-01T18:16:13.4035197Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_overlap_1.1_ea4ba97d6bd6aef9_.log (deflated 58%) 2024-11-01T18:16:13.4036508Z adding: test/test-reports/distributed.test_device_mesh_1.1_84784ae416633333_.log (deflated 87%) 2024-11-01T18:16:13.4037700Z adding: test/test-reports/distributed._tensor.test_matrix_ops_1.1_a2be7291d7988348_.log (deflated 79%) 2024-11-01T18:16:13.4039140Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_mixed_precision_1.1_cf278296d88e0f60_.log (deflated 58%) 2024-11-01T18:16:13.4040523Z adding: test/test-reports/distributed.test_functional_api_1.1_a2ea1506a3fcde53_.log (deflated 86%) 2024-11-01T18:16:13.4041753Z adding: test/test-reports/distributed.elastic.events.lib_test_1.1_0764544c3a746fc8_.log (deflated 72%) 2024-11-01T18:16:13.4043061Z adding: test/test-reports/distributed.checkpoint.test_fsdp_model_state_1.1_62329a94e97347cd_.log (deflated 64%) 2024-11-01T18:16:13.4044338Z adding: test/test-reports/distributed._tensor.test_optimizers_1.1_19d7c0109d621187_.log (deflated 82%) 2024-11-01T18:16:13.4045693Z adding: test/test-reports/distributed.elastic.metrics.api_test_1.1_1a8a731d5fe637e4_.log (deflated 62%) 2024-11-01T18:16:13.4047078Z adding: test/test-reports/distributed.tensor.parallel.test_parallelize_api_1.1_1922db24e879738e_.log (deflated 84%) 2024-11-01T18:16:13.4048358Z adding: test/test-reports/distributed.test_c10d_logger_1.1_4746d7f07d998299_.log (deflated 59%) 2024-11-01T18:16:13.4049546Z adding: test/test-reports/distributed.checkpoint.test_utils_1.1_68898c9e76bd447a_.log (deflated 66%) 2024-11-01T18:16:13.4050828Z adding: test/test-reports/distributed.fsdp.test_fsdp_use_orig_params_1.1_7b87f24658f71cd4_.log (deflated 86%) 2024-11-01T18:16:13.4052421Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_compile_1.1_896262ff9593c0e4_.log (deflated 85%) 2024-11-01T18:16:13.4053802Z adding: test/test-reports/distributed._tensor.debug.test_comm_mode_1.1_a767bc3608f14b93_.log (deflated 61%) 2024-11-01T18:16:13.4055084Z adding: test/test-reports/distributed.fsdp.test_fsdp_hybrid_shard_1.1_799418be87dd7274_.log (deflated 73%) 2024-11-01T18:16:13.4056501Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_clip_grad_norm__1.1_f763df747841611d_.log (deflated 63%) 2024-11-01T18:16:13.4057949Z adding: test/test-reports/distributed.fsdp.test_fsdp_sharded_grad_scaler_1.1_52e9796290d64be8_.log (deflated 88%) 2024-11-01T18:16:13.4059542Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_0b9303d06f4f6a53_.log (deflated 74%) 2024-11-01T18:16:13.4060924Z adding: test/test-reports/distributed._composable.test_compose_1.1_365449705f3e36d6_.log (deflated 78%) 2024-11-01T18:16:13.4062179Z adding: test/test-reports/distributed.fsdp.test_fsdp_pure_fp16_1.1_b0ffd1b18b5b1cb4_.log (deflated 58%) 2024-11-01T18:16:13.4063516Z adding: test/test-reports/distributed.elastic.multiprocessing.api_test_1.1_b705c42a77083530_.log (deflated 84%) 2024-11-01T18:16:13.4064914Z adding: test/test-reports/distributed.optim.test_apply_optimizer_in_backward_1.1_d620c8cdf22b1603_.log (deflated 33%) 2024-11-01T18:16:13.4066239Z adding: test/test-reports/distributed.fsdp.test_fsdp_overlap_1.1_45bc8e5c6ed91616_.log (deflated 59%) 2024-11-01T18:16:13.4067458Z adding: test/test-reports/distributed.fsdp.test_fsdp_misc_1.1_57b1dd8d775c0295_.log (deflated 90%) 2024-11-01T18:16:13.4068891Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_model_checkpoint_1.1_55767205334ffa1e_.log (deflated 73%) 2024-11-01T18:16:13.4070350Z adding: test/test-reports/distributed.fsdp.test_fsdp_multiple_forward_1.1_0fe75bc31284f6e0_.log (deflated 54%) 2024-11-01T18:16:13.4071739Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_b26fa0a37fd21fd0_.log (deflated 67%) 2024-11-01T18:16:13.4073218Z adding: test/test-reports/distributed._tensor.test_view_ops_1.1_ed58dcdfe9e67fa8_.log (deflated 67%) 2024-11-01T18:16:13.4074390Z adding: test/test-reports/distributed.test_fake_pg_1.1_231d4a36c18fa7d6_.log (deflated 73%) 2024-11-01T18:16:13.4075542Z adding: test/test-reports/distributed.elastic.timer.api_test_1.1_2827ff97f8d10b8f_.log (stored 0%) 2024-11-01T18:16:13.4076881Z adding: test/test-reports/distributed.checkpoint.test_file_system_checkpoint_1.1_7f7e45671e56d868_.log (deflated 73%) 2024-11-01T18:16:13.4078303Z adding: test/test-reports/distributed.rpc.test_share_memory_1.1_7b1a234f874ad3d2_.log (deflated 57%) 2024-11-01T18:16:13.4087389Z adding: test/test-reports/distributed.rpc.test_tensorpipe_agent_3.4_a7dd6ef3df8f50db_.log (deflated 97%) 2024-11-01T18:16:13.4089942Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_state_dict_1.1_177838ce76b8a1de_.log (deflated 76%) 2024-11-01T18:16:13.4098808Z adding: test/test-reports/distributed.fsdp.test_fsdp_state_dict_1.1_fd44cf44517c0946_.log (deflated 96%) 2024-11-01T18:16:13.4101162Z adding: test/test-reports/distributed.fsdp.test_fsdp_apply_1.1_1c9c85b75525a841_.log (deflated 63%) 2024-11-01T18:16:13.4103500Z adding: test/test-reports/distributed.fsdp.test_fsdp_traversal_1.1_8036c0671fba494a_.log (deflated 53%) 2024-11-01T18:16:13.4106290Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_binary_cmp_1.1_fadc226d4bc2949b_.log (deflated 70%) 2024-11-01T18:16:13.4109084Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_init_1.1_d4d8a8330349fcab_.log (deflated 77%) 2024-11-01T18:16:13.4111712Z adding: test/test-reports/distributed.fsdp.test_fsdp_ignored_modules_1.1_f6ee96b5d7120393_.log (deflated 85%) 2024-11-01T18:16:13.4114194Z adding: test/test-reports/distributed.fsdp.test_fsdp_grad_acc_1.1_6f5726a54111d402_.log (deflated 74%) 2024-11-01T18:16:13.4116561Z adding: test/test-reports/distributed.fsdp.test_checkpoint_wrapper_1.1_0ac5f03ab832e366_.log (deflated 74%) 2024-11-01T18:16:13.4118913Z adding: test/test-reports/distributed.fsdp.test_fsdp_memory_1.1_622fe548029732e7_.log (deflated 60%) 2024-11-01T18:16:13.4121228Z adding: test/test-reports/distributed.fsdp.test_fsdp_flatten_params_1.1_2e816716b0bb5106_.log (deflated 80%) 2024-11-01T18:16:13.4161954Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_init_1.1_de5d3e61fb494ad7_.log (deflated 67%) 2024-11-01T18:16:13.4164814Z adding: test/test-reports/distributed._composable.test_replicate_with_compiler_1.1_f079b1e9bf507609_.log (deflated 77%) 2024-11-01T18:16:13.4167211Z adding: test/test-reports/distributed.fsdp.test_fsdp_freezing_weights_1.1_546683fc4aa86702_.log (deflated 94%) 2024-11-01T18:16:13.4169580Z adding: test/test-reports/distributed.elastic.timer.local_timer_example_1.1_68e64129a9ef9a5f_.log (deflated 68%) 2024-11-01T18:16:13.4171936Z adding: test/test-reports/distributed.fsdp.test_hsdp_dtensor_state_dict_1.1_0df314066c2d45f7_.log (deflated 75%) 2024-11-01T18:16:13.4174150Z adding: test/test-reports/distributed.fsdp.test_fsdp_uneven_1.1_8dae878b6c58bc7f_.log (deflated 52%) 2024-11-01T18:16:13.4176334Z adding: test/test-reports/distributed.fsdp.test_fsdp_mixed_precision_1.1_200f1d5808a7ac86_.log (deflated 92%) 2024-11-01T18:16:13.4178825Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_extensions_1.1_16cafc7859d812d8_.log (deflated 73%) 2024-11-01T18:16:13.4181109Z adding: test/test-reports/distributed.fsdp.test_wrap_1.1_4845fb334e39fd1b_.log (deflated 90%) 2024-11-01T18:16:13.4183163Z adding: test/test-reports/distributed.test_c10d_pypg_1.1_df943223382cde97_.log (deflated 90%) 2024-11-01T18:16:13.4185227Z adding: test/test-reports/distributed.checkpoint.test_state_dict_1.1_bc93eb1775f48ba3_.log (deflated 90%) 2024-11-01T18:16:13.4187275Z adding: test/test-reports/distributed.test_launcher_1.1_9b95a6b6e63b9272_.log (deflated 59%) 2024-11-01T18:16:13.4188580Z adding: test/test-reports/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks_1.1_659d63b1accbdcb4_.log (deflated 76%) 2024-11-01T18:16:13.4190253Z adding: test/test-reports/distributed.fsdp.test_fsdp_checkpoint_1.1_f83e6bd475faa888_.log (deflated 87%) 2024-11-01T18:16:13.4191554Z adding: test/test-reports/distributed.algorithms.test_join_1.1_a71032f07223b852_.log (deflated 81%) 2024-11-01T18:16:13.4192873Z adding: test/test-reports/distributed.fsdp.test_fsdp_clip_grad_norm_1.1_6fe1b389972e5d09_.log (deflated 67%) 2024-11-01T18:16:13.4194347Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_optim_checkpoint_1.1_df75bebe9677188f_.log (deflated 65%) 2024-11-01T18:16:13.4195730Z adding: test/test-reports/distributed.rpc.test_faulty_agent_1.1_f91759641ecc5142_.log (deflated 96%) 2024-11-01T18:16:13.4196916Z adding: test/test-reports/distributed.fsdp.test_fsdp_comm_1.1_57cda1d7dabef5f9_.log (deflated 79%) 2024-11-01T18:16:13.4198187Z adding: test/test-reports/distributed.elastic.timer.local_timer_test_1.1_299d5ef3741de756_.log (deflated 77%) 2024-11-01T18:16:13.4199560Z adding: test/test-reports/distributed._shard.sharded_optim.test_sharded_optim_1.1_5617a5a5f10fb30d_.log (deflated 62%) 2024-11-01T18:16:13.4200980Z adding: test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_1.1_89e909d52e77a2f2_.log (deflated 91%) 2024-11-01T18:16:13.4202537Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_tensor_ops_1.1_e494b3f036d6a6be_.log (deflated 71%) 2024-11-01T18:16:13.4203891Z adding: test/test-reports/distributed.rpc.test_tensorpipe_agent_1.4_a409cb129b840c80_.log (deflated 97%) 2024-11-01T18:16:13.4230691Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-11-01T18:16:13.4231434Z # Remove any previous debugging artifacts if they exist 2024-11-01T18:16:13.4231981Z rm -f debug-*.zip 2024-11-01T18:16:13.4232356Z if [ -d 'test/debug' ]; then 2024-11-01T18:16:13.4232958Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-11-01T18:16:13.4233440Z fi 2024-11-01T18:16:13.4238984Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T18:16:13.4239486Z env: 2024-11-01T18:16:13.4239738Z GIT_DEFAULT_BRANCH: main 2024-11-01T18:16:13.4240366Z DOCKER_CONTAINER_ID: ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e 2024-11-01T18:16:13.4241188Z FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_32396543718 2024-11-01T18:16:13.4241716Z ##[endgroup] 2024-11-01T18:16:13.4325566Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-11-01T18:16:13.4326007Z with: 2024-11-01T18:16:13.4326273Z s3-bucket: gha-artifacts 2024-11-01T18:16:13.4326691Z s3-prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T18:16:13.4327168Z retention-days: 14 2024-11-01T18:16:13.4327489Z if-no-files-found: warn 2024-11-01T18:16:13.4327842Z path: test-jsons-*.zip 2024-11-01T18:16:13.4328167Z name: artifact 2024-11-01T18:16:13.4328446Z region: us-east-1 2024-11-01T18:16:13.4328731Z env: 2024-11-01T18:16:13.4328990Z GIT_DEFAULT_BRANCH: main 2024-11-01T18:16:13.4329613Z DOCKER_CONTAINER_ID: ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e 2024-11-01T18:16:13.4330283Z ##[endgroup] 2024-11-01T18:16:13.8401133Z NOTE: s3-prefix specified, ignoring name parameter 2024-11-01T18:16:13.8402181Z With the provided path, there will be 1 file uploaded 2024-11-01T18:16:13.8402834Z Uploading to s3 prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T18:16:14.0193223Z Starting upload of test-jsons-test-distributed-1-2-linux.2xlarge_32396543718.zip 2024-11-01T18:16:14.1222216Z Finished upload of test-jsons-test-distributed-1-2-linux.2xlarge_32396543718.zip 2024-11-01T18:16:14.1423658Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-11-01T18:16:14.1424115Z with: 2024-11-01T18:16:14.1424384Z s3-bucket: gha-artifacts 2024-11-01T18:16:14.1424808Z s3-prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T18:16:14.1425285Z retention-days: 14 2024-11-01T18:16:14.1425614Z if-no-files-found: error 2024-11-01T18:16:14.1425974Z path: test-reports-*.zip 2024-11-01T18:16:14.1426460Z name: artifact 2024-11-01T18:16:14.1426743Z region: us-east-1 2024-11-01T18:16:14.1427036Z env: 2024-11-01T18:16:14.1427297Z GIT_DEFAULT_BRANCH: main 2024-11-01T18:16:14.1427928Z DOCKER_CONTAINER_ID: ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e 2024-11-01T18:16:14.1428602Z ##[endgroup] 2024-11-01T18:16:14.5164901Z NOTE: s3-prefix specified, ignoring name parameter 2024-11-01T18:16:14.5165956Z With the provided path, there will be 1 file uploaded 2024-11-01T18:16:14.5167141Z Uploading to s3 prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T18:16:14.5207767Z Starting upload of test-reports-test-distributed-1-2-linux.2xlarge_32396543718.zip 2024-11-01T18:16:14.6929390Z Finished upload of test-reports-test-distributed-1-2-linux.2xlarge_32396543718.zip 2024-11-01T18:16:14.7129298Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-11-01T18:16:14.7129751Z with: 2024-11-01T18:16:14.7130023Z s3-bucket: gha-artifacts 2024-11-01T18:16:14.7130459Z s3-prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T18:16:14.7130932Z retention-days: 14 2024-11-01T18:16:14.7131256Z if-no-files-found: ignore 2024-11-01T18:16:14.7131767Z path: logs-*.zip 2024-11-01T18:16:14.7132068Z name: artifact 2024-11-01T18:16:14.7132348Z region: us-east-1 2024-11-01T18:16:14.7132643Z env: 2024-11-01T18:16:14.7132966Z GIT_DEFAULT_BRANCH: main 2024-11-01T18:16:14.7133631Z DOCKER_CONTAINER_ID: ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e 2024-11-01T18:16:14.7134303Z ##[endgroup] 2024-11-01T18:16:15.0882908Z NOTE: s3-prefix specified, ignoring name parameter 2024-11-01T18:16:15.0883745Z With the provided path, there will be 1 file uploaded 2024-11-01T18:16:15.0884396Z Uploading to s3 prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T18:16:15.0924060Z Starting upload of logs-test-distributed-1-2-linux.2xlarge_32396543718.zip 2024-11-01T18:16:15.3823071Z Finished upload of logs-test-distributed-1-2-linux.2xlarge_32396543718.zip 2024-11-01T18:16:15.4003696Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-11-01T18:16:15.4004467Z with: 2024-11-01T18:16:15.4004741Z s3-bucket: gha-artifacts 2024-11-01T18:16:15.4005184Z s3-prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T18:16:15.4005668Z retention-days: 14 2024-11-01T18:16:15.4005999Z if-no-files-found: ignore 2024-11-01T18:16:15.4006353Z path: debug-*.zip 2024-11-01T18:16:15.4006657Z name: artifact 2024-11-01T18:16:15.4006942Z region: us-east-1 2024-11-01T18:16:15.4007233Z env: 2024-11-01T18:16:15.4007502Z GIT_DEFAULT_BRANCH: main 2024-11-01T18:16:15.4008126Z DOCKER_CONTAINER_ID: ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e 2024-11-01T18:16:15.4008803Z ##[endgroup] 2024-11-01T18:16:15.7699502Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-11-01T18:16:15.7877963Z ##[group]Run # shellcheck disable=SC2156 2024-11-01T18:16:15.7878445Z # shellcheck disable=SC2156 2024-11-01T18:16:15.7879275Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-11-01T18:16:15.7885653Z shell: /usr/bin/bash -e {0} 2024-11-01T18:16:15.7886023Z env: 2024-11-01T18:16:15.7886278Z GIT_DEFAULT_BRANCH: main 2024-11-01T18:16:15.7886903Z DOCKER_CONTAINER_ID: ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e 2024-11-01T18:16:15.7887578Z ##[endgroup] 2024-11-01T18:16:16.0219243Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-11-01T18:16:16.0219827Z with: 2024-11-01T18:16:16.0220056Z env: 2024-11-01T18:16:16.0220315Z GIT_DEFAULT_BRANCH: main 2024-11-01T18:16:16.0220934Z DOCKER_CONTAINER_ID: ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e 2024-11-01T18:16:16.0221607Z ##[endgroup] 2024-11-01T18:16:16.0242717Z ##[group]Run set -eou pipefail 2024-11-01T18:16:16.0243120Z set -eou pipefail 2024-11-01T18:16:16.0243465Z  2024-11-01T18:16:16.0243978Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-11-01T18:16:16.0244884Z for _ in $(seq 1440); do 2024-11-01T18:16:16.0245367Z  # Break if no ssh session exists anymore 2024-11-01T18:16:16.0245880Z  if [ "$(who)" = "" ]; then 2024-11-01T18:16:16.0246275Z  break 2024-11-01T18:16:16.0246576Z  fi 2024-11-01T18:16:16.0246906Z  echo "." 2024-11-01T18:16:16.0247224Z  sleep 5 2024-11-01T18:16:16.0247517Z done 2024-11-01T18:16:16.0253696Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T18:16:16.0254208Z env: 2024-11-01T18:16:16.0254473Z GIT_DEFAULT_BRANCH: main 2024-11-01T18:16:16.0255099Z DOCKER_CONTAINER_ID: ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e 2024-11-01T18:16:16.0255776Z ##[endgroup] 2024-11-01T18:16:16.0280980Z Holding runner for 2 hours until all ssh sessions have logged out 2024-11-01T18:16:16.0375117Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T18:16:16.0375904Z # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T18:16:16.0376505Z # shellcheck disable=SC2046 2024-11-01T18:16:16.0377118Z docker stop $(docker ps -q) || true 2024-11-01T18:16:16.0377603Z # Prune all of the docker images 2024-11-01T18:16:16.0378056Z docker system prune -af 2024-11-01T18:16:16.0383665Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T18:16:16.0384156Z env: 2024-11-01T18:16:16.0384422Z GIT_DEFAULT_BRANCH: main 2024-11-01T18:16:16.0385045Z DOCKER_CONTAINER_ID: ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e 2024-11-01T18:16:16.0385714Z ##[endgroup] 2024-11-01T18:16:26.6078440Z ec832529b912 2024-11-01T18:16:26.9590285Z Deleted Containers: 2024-11-01T18:16:26.9591097Z ec832529b912ef88c538942ecf3e70b3419b2f2f86c4dd6b588b0104516efd4e 2024-11-01T18:16:26.9591705Z 2024-11-01T18:16:30.6843048Z Deleted Images: 2024-11-01T18:16:30.6844981Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T18:16:30.6846968Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.9-gcc11@sha256:68d308b5ab9729d840ef6882af267b00db6e15ae40d6c8de9da61fef478f8a18 2024-11-01T18:16:30.6848431Z deleted: sha256:acd5ab414b6754b1ba1fbfceff9d37aa2af65d736e970c726d477963e8596d49 2024-11-01T18:16:30.6849348Z deleted: sha256:93d4c75f8615120fabeeba1f8213410a94a4ba687993261f88de5cc89bcbf906 2024-11-01T18:16:30.6850248Z deleted: sha256:b63de2ee0c28ad19c4b1c104bf86117de9b8910b607e03ae4d23205b4d148692 2024-11-01T18:16:30.6851135Z deleted: sha256:0a53ccbaa95fb1a992cc6c5e65510999516c30369c756bf31ca672cbc64fb9b3 2024-11-01T18:16:30.6852031Z deleted: sha256:d78307e26844a1efaffd88d6f73ce0f57b842be250bb227e42eeef4747662f84 2024-11-01T18:16:30.6852930Z deleted: sha256:c31baeac25f19cf5310f19aae7d284b1210c38b3420e5380985a90845bbb3ce6 2024-11-01T18:16:30.6853824Z deleted: sha256:8777d65601fce44cfc563e1c10bc5fcab4bc999fcf5fb993482e8b87be10893f 2024-11-01T18:16:30.6854723Z deleted: sha256:e916dd94c9d50b354fcaaa29e4d62f47f9de54074151253dabb04146dab6e011 2024-11-01T18:16:30.6855637Z deleted: sha256:1b6b584aa7ce85e635c8bbddc60ac91b9739e6024f8fbf9f06fb4dcbd29845d1 2024-11-01T18:16:30.6856868Z deleted: sha256:2c1df409b0bc3c603cff4d93e404ad37b8f239a22c249b619fece0ae4ea9401d 2024-11-01T18:16:30.6857757Z deleted: sha256:64d683cb97b5d2594640455cc9abda679180d03fde8042fe1eb42f5c11339211 2024-11-01T18:16:30.6858817Z deleted: sha256:52d56c43eadccf8502b8b135fcb8977834757560c7203223db9636e09e33d84c 2024-11-01T18:16:30.6859724Z deleted: sha256:e5fa9a8a5cdfe96d9803f8d75ab7bfa100dac25e8f5a2796c4da5263defc2a41 2024-11-01T18:16:30.6860615Z deleted: sha256:05394b4a9c017e772570d96d044015ec9a91ebc9a4d1801eb4542a1277ab20c3 2024-11-01T18:16:30.6861482Z deleted: sha256:fa175c9721a866b0a80d601ae54178dbcfd1958c1f707946999873c113ae9d67 2024-11-01T18:16:30.6862346Z deleted: sha256:1e8f530a59c243c9b668032780b0f009c4f01d39414229567de23fbaffec9cfa 2024-11-01T18:16:30.6863209Z deleted: sha256:6db5bde9f97b983b138f7bda4f891595437b5d1aa505bb05d66f3ef85c45127b 2024-11-01T18:16:30.6864110Z deleted: sha256:8469ffb0d1c0dc2e20854f1a4008c12f3b78335682af5ba92d1930a5adbc7028 2024-11-01T18:16:30.6865011Z deleted: sha256:89302d65daebdd48960bcdfcc4baafd9b1b999d15347822b5c224d412bad1075 2024-11-01T18:16:30.6865912Z deleted: sha256:c3306463289e9c13cb9693fffa9d751830069dc6dea34fb0f246bd6a2d68f501 2024-11-01T18:16:30.6866803Z deleted: sha256:7cadc42c996bfd4e5ef638a0c39f4833b0e8b56c799ef47ef6b0796e718a8e2c 2024-11-01T18:16:30.6867679Z deleted: sha256:66dd4211375278d63533557df9880e017092a2279bcf1aa4b2026ceb1f79aee6 2024-11-01T18:16:30.6868534Z deleted: sha256:6a3e6ae5f03a8e430cfb7fc8ba67044c4193e8687608d5807166fdc637d76c51 2024-11-01T18:16:30.6869433Z deleted: sha256:7fb11ba653fd3f0ac5254b80dc431bcc70ace76abfa5a6c54db12b5fada1f628 2024-11-01T18:16:30.6870327Z deleted: sha256:1df534e180693f6c931b18ad42094de6c6da937b8a947d0df367ae372f07c252 2024-11-01T18:16:30.6871215Z deleted: sha256:b4674aad1e9f30bc4a21079d55e516bedc8d0dbbc4fbb6e33353d436107f7d0e 2024-11-01T18:16:30.6872102Z deleted: sha256:b157d9219906e9a181856f4ea238e28d2ef3a7e2a9620682a2f97938a57c16c7 2024-11-01T18:16:30.6873236Z deleted: sha256:ec1b84ee35628aea2e4be97caecf75e470a7f3fc0ffcf7c9ece8b6b5c0350179 2024-11-01T18:16:30.6874149Z deleted: sha256:c641d29fa165b9df2e600522be43a800058e264119d786ae5169ea44d9c6d633 2024-11-01T18:16:30.6875006Z deleted: sha256:f6cab744e96c900941f08148a16559b0bf8d082d3708437293d9efde965b7e05 2024-11-01T18:16:30.6875896Z deleted: sha256:0e7d20ba5f6c8240de8a882de5db511016bda1151d8789c59da7a22c8abb871a 2024-11-01T18:16:30.6876795Z deleted: sha256:56c722cb5dea3a08f1a0d7ec11d78a96c79621694093c8277d19ecab3ce88c65 2024-11-01T18:16:30.6877696Z deleted: sha256:e91a23eb11d67c2e9366e9feeabf19fb04dbce8560b4aa25e5f59409473360e6 2024-11-01T18:16:30.6878600Z deleted: sha256:6c701e81fdbe6d18019ad98862dab19332ff13df1f7b7deb79906deeb406662f 2024-11-01T18:16:30.6879485Z deleted: sha256:8ffc0852321d71307b2d814b44439d326ccd3585e620fd5f9d2b9d14c2f57c76 2024-11-01T18:16:30.6880345Z deleted: sha256:0231028281bd3abe9274ff1789f476b5d04f462b397b7bfd7aabec966895429b 2024-11-01T18:16:30.6881227Z deleted: sha256:d2a62f6ff0b36141922e69d55eee63f81172048e7e47eb02d317f8d43569a63e 2024-11-01T18:16:30.6882106Z deleted: sha256:936d9afc1499b541e2af4d535c445c1a92f2781bd362bdee530836e3ca2805ca 2024-11-01T18:16:30.6882996Z deleted: sha256:e4adf4a1ffb66e2392341556aa2d581b152bdfbd6249f925d873fce18e7daa9f 2024-11-01T18:16:30.6883883Z deleted: sha256:02509fb4f4a80806b2b7a773fd3ec833d429ddf548ee59c6280e28c541d007de 2024-11-01T18:16:30.6884926Z deleted: sha256:171f4ecd31f7318928abdb2d2235d1de266a9b5a50b92b0b0e134b1337a82419 2024-11-01T18:16:30.6885805Z deleted: sha256:caddba564eb4b90759f3f9d507b22a338c8d04a98d022c5225d9496885c23819 2024-11-01T18:16:30.6886705Z deleted: sha256:2f61b6d34ee2ab83fc8ed06c11087a7590732dd8d5a4d9428e7da9afbcd93e1f 2024-11-01T18:16:30.6887597Z deleted: sha256:0884213b056d2d710178f6e379521e1ee5ba0a52bd76eaa718147e082bdfd37a 2024-11-01T18:16:30.6888500Z deleted: sha256:6e2bd1e98ead15cfae2a20c0bef74b19098b8f339dd9cbbcf89d26d23076744d 2024-11-01T18:16:30.6889412Z deleted: sha256:13c0075bc3e89c37804dfeb3260a6a84741e753284bdeb4b8ed36431042edcd1 2024-11-01T18:16:30.6890311Z deleted: sha256:4ba7d9d48d20ffed1c4a0da01450453f959c1543f598b513ef2d4acbaf0f5ec2 2024-11-01T18:16:30.6891359Z deleted: sha256:ce931da4e7c258b233bff780e36e85ac997d8b49afa07b4535cc304cd31a11e7 2024-11-01T18:16:30.6892275Z deleted: sha256:3aaa5d72e932a5a15d8c59b85a871e9ee8bb84fd440ac8ee2add33d814484324 2024-11-01T18:16:30.6893189Z deleted: sha256:d1e2263c25c8e26ae2a2cd0647d58dbb585b7854e3a3bcdba930c920fa5af61d 2024-11-01T18:16:30.6894079Z deleted: sha256:a98147b873bd881afb3f40ac863183865857d7e51d03c77e62680a12d67bcdec 2024-11-01T18:16:30.6895412Z deleted: sha256:66f106e0ac50ae8a76edd59595174b50c242c2c2ac23acec7ef73c82d66297a1 2024-11-01T18:16:30.6896905Z deleted: sha256:b5d2c21b1dd9bc44170d5905fb53feb18cc36bffab40a77563393e1486846dd4 2024-11-01T18:16:30.6898032Z deleted: sha256:b72cce60f852d323a47ea10b266624ed08b955a6c05f06f0b49b7c20216388aa 2024-11-01T18:16:30.6898909Z deleted: sha256:74c7c4685920472ace75441f01522a55c24af0ccaa115d00d9dbe18df18e30ee 2024-11-01T18:16:30.6899809Z deleted: sha256:8630ef75057e74e403d57a98f74309dba1e7f85b1ed408525faa637bee221790 2024-11-01T18:16:30.6900702Z deleted: sha256:97528f5f870aa4a450943a8fc8c5a1c1c7c283f402bab249b6b508f9b27c3c3d 2024-11-01T18:16:30.6901581Z deleted: sha256:278371c37229c8c9e7351d8f0d93348f15b7cc5f3cdac826c0445d69aa2d6a10 2024-11-01T18:16:30.6902473Z deleted: sha256:a9c94a6aa0d059039d1ba4bcbc77966ecee565900a8cc818aa0285d1ed66c4ad 2024-11-01T18:16:30.6903389Z deleted: sha256:39bb64d0c3a9b6cf6b76d7cd6eaa1ca2eeb20297d2cd72ac5c44755b30c3aef6 2024-11-01T18:16:30.6904280Z deleted: sha256:644445c3b37180be4e526c5b00c7b41d00d0ccd98ef74c19ea25a0af4008ff2e 2024-11-01T18:16:30.6905168Z deleted: sha256:9a75bbfa0a4ea81a239dd0220499dce009590f2c04d00ee632c2f83113386a8a 2024-11-01T18:16:30.6906053Z deleted: sha256:d21955a1bb21e611e644c48daa5282a91a2916d0e786d1a7dd192109316b9557 2024-11-01T18:16:30.6906932Z deleted: sha256:fcaa556b6854c19b200a05f96fe84b229840e1de575ca7f50d213230fe25b26f 2024-11-01T18:16:30.6907931Z deleted: sha256:9cebf1d2e389c1903a89af6e5c69f174a5269e3e6638bd8909f829cf63187e5a 2024-11-01T18:16:30.6908843Z deleted: sha256:aaaf586d8bf742b00c3c36dc64756ca8caf46fb3993fbd74cf0b5ffc9dc7068e 2024-11-01T18:16:30.6909734Z deleted: sha256:4ccd5e5d28cbc0024a056a771e9255c7d7f2652a84e7f986cb759bb78eec3d42 2024-11-01T18:16:30.6910622Z deleted: sha256:e1d4b9f05a40316ed16ac9d2782512a3c8c93ff0138a4ad9da7818e4727ca5f9 2024-11-01T18:16:30.6911519Z deleted: sha256:25910dca7d8ea1cf745d816a127da821049bfa380e6ddefb0b087d200226648b 2024-11-01T18:16:30.6912424Z deleted: sha256:3d78aa41921e460e91e7aad88c3d8c7087bdf8be4fadecbd735cb9bc77247abe 2024-11-01T18:16:30.6913397Z deleted: sha256:8f7d14624ecb8bf5b3a22e2821004915a10523800b48ec38ca41321838901943 2024-11-01T18:16:30.6914284Z deleted: sha256:bbd6146f62f341eae7965473aa9bf21fdaff3b915eaf3ad68b485161344527b0 2024-11-01T18:16:30.6915182Z deleted: sha256:6e4831d26492ce0d0f9cd5dd20eb43fca2756f0d53250dfe73007d1411ad0c5e 2024-11-01T18:16:30.6916071Z deleted: sha256:4c6fec9010edcd2fc690a58502aa7e7c40b4eb068efef822d78144219ba05aff 2024-11-01T18:16:30.6916963Z deleted: sha256:bb0a52775a7a09803c91c960dd86264b220b897371fae4e11a3c390ff0f76055 2024-11-01T18:16:30.6917846Z deleted: sha256:73e5b22cc89ad45e5730af3f6c908c454f5aeeb29b5d77995490543d662c2886 2024-11-01T18:16:30.6918725Z deleted: sha256:9b8350f631b365a72c8c3f19b5f579691d959aa38becbde32cdd39c6267d6454 2024-11-01T18:16:30.6919604Z deleted: sha256:e62a33aecd92746d8329b9212f36c1a173de2a65f36a488631a88ca162c64e73 2024-11-01T18:16:30.6920473Z deleted: sha256:6544f00a8f0bd424906c4493d6148760d4c94638022f4e7cb90681bf8444b084 2024-11-01T18:16:30.6921319Z deleted: sha256:95a2497c520f10dc257e56e30a553efb2b8bf46ec505989e04360488256f758c 2024-11-01T18:16:30.6922201Z deleted: sha256:ba8f96b6ecf286da1f96032fc117f49dea1872b03599708e3e95fe170482b21f 2024-11-01T18:16:30.6923095Z deleted: sha256:2cb3bb0ae28ef1443d20823b5d817e342cc811292ffd2aae5463d9f9ad5f0575 2024-11-01T18:16:30.6923997Z deleted: sha256:55fca8b9b71ddd36a31f9416d73be55815bd0149fe0dd8f2b407e3c2529a2b19 2024-11-01T18:16:30.6924896Z deleted: sha256:92656f9347466a332eba67bfdb5fa55ffce4ea0ca154e5964673c3cc361c481e 2024-11-01T18:16:30.6925853Z deleted: sha256:72e581b367a1bbf15e628dc140eb5c4959a42627139674b1d866a36e8635f2d3 2024-11-01T18:16:30.6926716Z deleted: sha256:3817e88611d8a5885cd27cec7ff3582a072913fcdf526679c5e3ade4bc7d3cb8 2024-11-01T18:16:30.6927616Z deleted: sha256:86dc99fe4b04edc49424ef6166cacb649da3b96bd7f3ddeabe874e214d90e21a 2024-11-01T18:16:30.6928527Z deleted: sha256:7267a3bed3d7c10acdbb147bfa514df85f9616b3eff70a65f480f2e94c62ab11 2024-11-01T18:16:30.6929436Z deleted: sha256:d4f1a7de408e56eb63c85e8a2f9f4bb6a869cf603460dac9db302a63d99467ea 2024-11-01T18:16:30.6930360Z deleted: sha256:de0ebf8dc2a0fe35ba3f5017a1e7a479fbaf89cd1f29df8b9fa71e5404ac5e72 2024-11-01T18:16:30.6931272Z deleted: sha256:16045d13bfcf7601cf505c900ce4b924de4276b2298cbcf4368ca549e6c9b253 2024-11-01T18:16:30.6932157Z deleted: sha256:1de9e018f602ea1e82e2fd77e92c6cee5f9b8194701b3fea384bbb7747fdb9d0 2024-11-01T18:16:30.6933062Z deleted: sha256:9f07dc8d8caa9ef8ba9c434b154dfefcdae201f4aa1ee96f9985d367cbdad914 2024-11-01T18:16:30.6933980Z deleted: sha256:1de940da5e07249accfa1463ae4a0ba8970537e754f9d66c156838fb550e4b47 2024-11-01T18:16:30.6934866Z deleted: sha256:c63e4c08818cb327718369cbf8b4ff8428b3d4626f1847768bc31d76217c6c2d 2024-11-01T18:16:30.6935765Z deleted: sha256:0b2b7bcf246325532e79fec05292983083c5d48af4d68f28b98dbd3fc1d6dbee 2024-11-01T18:16:30.6936656Z deleted: sha256:f8486aa1b48ce6b5902d475899ec53f3c5c840ea3ff8fcc4cee0fbb91e53f2d9 2024-11-01T18:16:30.6937551Z deleted: sha256:2573e0d8158209ed54ab25c87bcdcb00bd3d2539246960a3d592a1c599d70465 2024-11-01T18:16:30.6938070Z 2024-11-01T18:16:30.6938198Z Total reclaimed space: 21.58GB 2024-11-01T18:16:30.7019658Z Post job cleanup. 2024-11-01T18:16:30.7079024Z Post job cleanup. 2024-11-01T18:16:30.8045304Z [command]/usr/bin/git version 2024-11-01T18:16:30.8091870Z git version 2.40.1 2024-11-01T18:16:30.8130069Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/68c57926-2b1f-410d-a080-65cf9529ad3c' before making global git config changes 2024-11-01T18:16:30.8131438Z Adding repository directory to the temporary git global config as a safe directory 2024-11-01T18:16:30.8135000Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T18:16:30.8172560Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-11-01T18:16:30.8206121Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-11-01T18:16:30.8616560Z Entering 'android/libs/fbjni' 2024-11-01T18:16:30.8667232Z Entering 'third_party/FP16' 2024-11-01T18:16:30.8718129Z Entering 'third_party/FXdiv' 2024-11-01T18:16:30.8770061Z Entering 'third_party/NNPACK' 2024-11-01T18:16:30.8819747Z Entering 'third_party/NVTX' 2024-11-01T18:16:30.8870102Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T18:16:30.8920663Z Entering 'third_party/XNNPACK' 2024-11-01T18:16:30.8986542Z Entering 'third_party/benchmark' 2024-11-01T18:16:30.9037506Z Entering 'third_party/composable_kernel' 2024-11-01T18:16:30.9092271Z Entering 'third_party/cpp-httplib' 2024-11-01T18:16:30.9141682Z Entering 'third_party/cpuinfo' 2024-11-01T18:16:30.9191966Z Entering 'third_party/cudnn_frontend' 2024-11-01T18:16:30.9241694Z Entering 'third_party/cutlass' 2024-11-01T18:16:30.9299758Z Entering 'third_party/eigen' 2024-11-01T18:16:30.9350483Z Entering 'third_party/fbgemm' 2024-11-01T18:16:30.9401672Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T18:16:30.9450535Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T18:16:30.9499542Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T18:16:30.9555720Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T18:16:30.9606840Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T18:16:30.9656342Z Entering 'third_party/flatbuffers' 2024-11-01T18:16:30.9709983Z Entering 'third_party/fmt' 2024-11-01T18:16:30.9760951Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T18:16:30.9808552Z Entering 'third_party/gloo' 2024-11-01T18:16:30.9857804Z Entering 'third_party/googletest' 2024-11-01T18:16:30.9907131Z Entering 'third_party/ideep' 2024-11-01T18:16:30.9955222Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T18:16:31.0010449Z Entering 'third_party/ittapi' 2024-11-01T18:16:31.0061040Z Entering 'third_party/kineto' 2024-11-01T18:16:31.0108805Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T18:16:31.0157669Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T18:16:31.0208476Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T18:16:31.0257555Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T18:16:31.0306587Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T18:16:31.0354101Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T18:16:31.0408567Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T18:16:31.0456468Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T18:16:31.0506216Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T18:16:31.0556275Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T18:16:31.0609975Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T18:16:31.0659845Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T18:16:31.0710144Z Entering 'third_party/mimalloc' 2024-11-01T18:16:31.0760932Z Entering 'third_party/nccl/nccl' 2024-11-01T18:16:31.0812877Z Entering 'third_party/nlohmann' 2024-11-01T18:16:31.0865285Z Entering 'third_party/onnx' 2024-11-01T18:16:31.0931709Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T18:16:31.0984427Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T18:16:31.1036858Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T18:16:31.1085694Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T18:16:31.1134863Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T18:16:31.1185400Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T18:16:31.1235442Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T18:16:31.1285932Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T18:16:31.1334381Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T18:16:31.1381900Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T18:16:31.1433646Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T18:16:31.1484797Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T18:16:31.1554111Z Entering 'third_party/pocketfft' 2024-11-01T18:16:31.1603750Z Entering 'third_party/protobuf' 2024-11-01T18:16:31.1655766Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T18:16:31.1705092Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T18:16:31.1755062Z Entering 'third_party/psimd' 2024-11-01T18:16:31.1804783Z Entering 'third_party/pthreadpool' 2024-11-01T18:16:31.1854152Z Entering 'third_party/pybind11' 2024-11-01T18:16:31.1904339Z Entering 'third_party/python-peachpy' 2024-11-01T18:16:31.1953370Z Entering 'third_party/sleef' 2024-11-01T18:16:31.2002900Z Entering 'third_party/tensorpipe' 2024-11-01T18:16:31.2051950Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T18:16:31.2101193Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T18:16:31.2149666Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T18:16:31.2200429Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T18:16:31.2247342Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T18:16:31.2315854Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-11-01T18:16:31.2341904Z http.https://github.com/.extraheader 2024-11-01T18:16:31.2350290Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-11-01T18:16:31.2383988Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-11-01T18:16:31.2659944Z Entering 'android/libs/fbjni' 2024-11-01T18:16:31.2692475Z http.https://github.com/.extraheader 2024-11-01T18:16:31.2723322Z Entering 'third_party/FP16' 2024-11-01T18:16:31.2756675Z http.https://github.com/.extraheader 2024-11-01T18:16:31.2786661Z Entering 'third_party/FXdiv' 2024-11-01T18:16:31.2819514Z http.https://github.com/.extraheader 2024-11-01T18:16:31.2850521Z Entering 'third_party/NNPACK' 2024-11-01T18:16:31.2884118Z http.https://github.com/.extraheader 2024-11-01T18:16:31.2914754Z Entering 'third_party/NVTX' 2024-11-01T18:16:31.2947561Z http.https://github.com/.extraheader 2024-11-01T18:16:31.2978832Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T18:16:31.3012095Z http.https://github.com/.extraheader 2024-11-01T18:16:31.3042477Z Entering 'third_party/XNNPACK' 2024-11-01T18:16:31.3075242Z http.https://github.com/.extraheader 2024-11-01T18:16:31.3124663Z Entering 'third_party/benchmark' 2024-11-01T18:16:31.3157334Z http.https://github.com/.extraheader 2024-11-01T18:16:31.3187491Z Entering 'third_party/composable_kernel' 2024-11-01T18:16:31.3220462Z http.https://github.com/.extraheader 2024-11-01T18:16:31.3256160Z Entering 'third_party/cpp-httplib' 2024-11-01T18:16:31.3289177Z http.https://github.com/.extraheader 2024-11-01T18:16:31.3319601Z Entering 'third_party/cpuinfo' 2024-11-01T18:16:31.3352398Z http.https://github.com/.extraheader 2024-11-01T18:16:31.3383121Z Entering 'third_party/cudnn_frontend' 2024-11-01T18:16:31.3416114Z http.https://github.com/.extraheader 2024-11-01T18:16:31.3446699Z Entering 'third_party/cutlass' 2024-11-01T18:16:31.3479888Z http.https://github.com/.extraheader 2024-11-01T18:16:31.3517822Z Entering 'third_party/eigen' 2024-11-01T18:16:31.3550677Z http.https://github.com/.extraheader 2024-11-01T18:16:31.3583020Z Entering 'third_party/fbgemm' 2024-11-01T18:16:31.3616153Z http.https://github.com/.extraheader 2024-11-01T18:16:31.3647172Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T18:16:31.3679512Z http.https://github.com/.extraheader 2024-11-01T18:16:31.3709687Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T18:16:31.3742477Z http.https://github.com/.extraheader 2024-11-01T18:16:31.3772549Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T18:16:31.3805400Z http.https://github.com/.extraheader 2024-11-01T18:16:31.3840577Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T18:16:31.3872836Z http.https://github.com/.extraheader 2024-11-01T18:16:31.3902858Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T18:16:31.3935548Z http.https://github.com/.extraheader 2024-11-01T18:16:31.3967776Z Entering 'third_party/flatbuffers' 2024-11-01T18:16:31.4000796Z http.https://github.com/.extraheader 2024-11-01T18:16:31.4033685Z Entering 'third_party/fmt' 2024-11-01T18:16:31.4068173Z http.https://github.com/.extraheader 2024-11-01T18:16:31.4098763Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T18:16:31.4131480Z http.https://github.com/.extraheader 2024-11-01T18:16:31.4163289Z Entering 'third_party/gloo' 2024-11-01T18:16:31.4196492Z http.https://github.com/.extraheader 2024-11-01T18:16:31.4226764Z Entering 'third_party/googletest' 2024-11-01T18:16:31.4259932Z http.https://github.com/.extraheader 2024-11-01T18:16:31.4290345Z Entering 'third_party/ideep' 2024-11-01T18:16:31.4323061Z http.https://github.com/.extraheader 2024-11-01T18:16:31.4352493Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T18:16:31.4385347Z http.https://github.com/.extraheader 2024-11-01T18:16:31.4423257Z Entering 'third_party/ittapi' 2024-11-01T18:16:31.4456550Z http.https://github.com/.extraheader 2024-11-01T18:16:31.4486877Z Entering 'third_party/kineto' 2024-11-01T18:16:31.4520000Z http.https://github.com/.extraheader 2024-11-01T18:16:31.4550316Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T18:16:31.4584399Z http.https://github.com/.extraheader 2024-11-01T18:16:31.4615700Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T18:16:31.4647889Z http.https://github.com/.extraheader 2024-11-01T18:16:31.4679857Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T18:16:31.4711636Z http.https://github.com/.extraheader 2024-11-01T18:16:31.4742527Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T18:16:31.4774906Z http.https://github.com/.extraheader 2024-11-01T18:16:31.4805124Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T18:16:31.4837245Z http.https://github.com/.extraheader 2024-11-01T18:16:31.4868111Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T18:16:31.4900780Z http.https://github.com/.extraheader 2024-11-01T18:16:31.4933781Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T18:16:31.4967134Z http.https://github.com/.extraheader 2024-11-01T18:16:31.4997829Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T18:16:31.5029835Z http.https://github.com/.extraheader 2024-11-01T18:16:31.5062465Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T18:16:31.5094492Z http.https://github.com/.extraheader 2024-11-01T18:16:31.5126260Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T18:16:31.5158862Z http.https://github.com/.extraheader 2024-11-01T18:16:31.5191528Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T18:16:31.5223658Z http.https://github.com/.extraheader 2024-11-01T18:16:31.5254163Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T18:16:31.5286577Z http.https://github.com/.extraheader 2024-11-01T18:16:31.5318673Z Entering 'third_party/mimalloc' 2024-11-01T18:16:31.5351176Z http.https://github.com/.extraheader 2024-11-01T18:16:31.5382904Z Entering 'third_party/nccl/nccl' 2024-11-01T18:16:31.5416037Z http.https://github.com/.extraheader 2024-11-01T18:16:31.5446946Z Entering 'third_party/nlohmann' 2024-11-01T18:16:31.5499070Z http.https://github.com/.extraheader 2024-11-01T18:16:31.5511878Z Entering 'third_party/onnx' 2024-11-01T18:16:31.5544218Z http.https://github.com/.extraheader 2024-11-01T18:16:31.5591062Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T18:16:31.5624564Z http.https://github.com/.extraheader 2024-11-01T18:16:31.5657572Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T18:16:31.5691273Z http.https://github.com/.extraheader 2024-11-01T18:16:31.5724927Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T18:16:31.5757545Z http.https://github.com/.extraheader 2024-11-01T18:16:31.5788183Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T18:16:31.5820724Z http.https://github.com/.extraheader 2024-11-01T18:16:31.5850999Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T18:16:31.5883572Z http.https://github.com/.extraheader 2024-11-01T18:16:31.5913802Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T18:16:31.5945835Z http.https://github.com/.extraheader 2024-11-01T18:16:31.5978558Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T18:16:31.6010772Z http.https://github.com/.extraheader 2024-11-01T18:16:31.6041535Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T18:16:31.6074357Z http.https://github.com/.extraheader 2024-11-01T18:16:31.6104594Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T18:16:31.6137124Z http.https://github.com/.extraheader 2024-11-01T18:16:31.6167912Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T18:16:31.6201019Z http.https://github.com/.extraheader 2024-11-01T18:16:31.6233777Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T18:16:31.6266669Z http.https://github.com/.extraheader 2024-11-01T18:16:31.6299217Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T18:16:31.6331329Z http.https://github.com/.extraheader 2024-11-01T18:16:31.6383852Z Entering 'third_party/pocketfft' 2024-11-01T18:16:31.6417389Z http.https://github.com/.extraheader 2024-11-01T18:16:31.6447892Z Entering 'third_party/protobuf' 2024-11-01T18:16:31.6482555Z http.https://github.com/.extraheader 2024-11-01T18:16:31.6516763Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T18:16:31.6549302Z http.https://github.com/.extraheader 2024-11-01T18:16:31.6580303Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T18:16:31.6612584Z http.https://github.com/.extraheader 2024-11-01T18:16:31.6645275Z Entering 'third_party/psimd' 2024-11-01T18:16:31.6679320Z http.https://github.com/.extraheader 2024-11-01T18:16:31.6709920Z Entering 'third_party/pthreadpool' 2024-11-01T18:16:31.6743679Z http.https://github.com/.extraheader 2024-11-01T18:16:31.6775632Z Entering 'third_party/pybind11' 2024-11-01T18:16:31.6809266Z http.https://github.com/.extraheader 2024-11-01T18:16:31.6840240Z Entering 'third_party/python-peachpy' 2024-11-01T18:16:31.6874532Z http.https://github.com/.extraheader 2024-11-01T18:16:31.6904729Z Entering 'third_party/sleef' 2024-11-01T18:16:31.6937602Z http.https://github.com/.extraheader 2024-11-01T18:16:31.6968882Z Entering 'third_party/tensorpipe' 2024-11-01T18:16:31.7001715Z http.https://github.com/.extraheader 2024-11-01T18:16:31.7031973Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T18:16:31.7064742Z http.https://github.com/.extraheader 2024-11-01T18:16:31.7095039Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T18:16:31.7127549Z http.https://github.com/.extraheader 2024-11-01T18:16:31.7158067Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T18:16:31.7190678Z http.https://github.com/.extraheader 2024-11-01T18:16:31.7221839Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T18:16:31.7255105Z http.https://github.com/.extraheader 2024-11-01T18:16:31.7285572Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T18:16:31.7318278Z http.https://github.com/.extraheader 2024-11-01T18:16:31.7434424Z A job completed hook has been configured by the self-hosted runner administrator 2024-11-01T18:16:31.7455090Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-11-01T18:16:31.7461590Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T18:16:31.7462118Z ##[endgroup] 2024-11-01T18:16:38.6688166Z Cleaning up orphan processes