2024-04-03T16:00:11.6508767Z Current runner version: '2.315.0' 2024-04-03T16:00:11.6515731Z Runner name: 'i-07486a5f9ad15937b' 2024-04-03T16:00:11.6516539Z Runner group name: 'Default' 2024-04-03T16:00:11.6517419Z Machine name: 'ip-10-0-9-184' 2024-04-03T16:00:11.6521867Z ##[group]GITHUB_TOKEN Permissions 2024-04-03T16:00:11.6524117Z Actions: read 2024-04-03T16:00:11.6524642Z Checks: read 2024-04-03T16:00:11.6525192Z Contents: read 2024-04-03T16:00:11.6525649Z Deployments: read 2024-04-03T16:00:11.6526159Z Discussions: read 2024-04-03T16:00:11.6526720Z Issues: read 2024-04-03T16:00:11.6527158Z Metadata: read 2024-04-03T16:00:11.6527638Z Packages: read 2024-04-03T16:00:11.6528141Z Pages: read 2024-04-03T16:00:11.6528582Z PullRequests: read 2024-04-03T16:00:11.6529126Z RepositoryProjects: read 2024-04-03T16:00:11.6529688Z SecurityEvents: read 2024-04-03T16:00:11.6530230Z Statuses: read 2024-04-03T16:00:11.6530722Z ##[endgroup] 2024-04-03T16:00:11.6534169Z Secret source: None 2024-04-03T16:00:11.6535177Z Prepare workflow directory 2024-04-03T16:00:11.7721929Z Prepare all required actions 2024-04-03T16:00:11.8016363Z Getting action download info 2024-04-03T16:00:11.9905211Z Download action repository 'pytorch/test-infra@main' (SHA:f5543b9b2d7b7c7b8eccd6576b68fc0472b1fbde) 2024-04-03T16:00:12.3130549Z Download action repository 'pytorch/pytorch@main' (SHA:74b3a7920ef9b96315ef20adef858b00cc067af6) 2024-04-03T16:00:15.3454407Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-04-03T16:00:15.4698512Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-04-03T16:00:15.7642139Z Getting action download info 2024-04-03T16:00:15.8551072Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-04-03T16:00:16.0334054Z Getting action download info 2024-04-03T16:00:16.1596301Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-04-03T16:00:16.3031204Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/123154/merge (4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0) 2024-04-03T16:00:16.3033617Z ##[group] Inputs 2024-04-03T16:00:16.3034072Z build-environment: linux-jammy-py3.8-gcc11 2024-04-03T16:00:16.3037936Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "default", "shard": 2, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "default", "shard": 3, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "docs_test", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge", "unstable": "unstable"}, {"config": "jit_legacy", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "backwards_compat", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge"}]} 2024-04-03T16:00:16.3041619Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:00:16.3042732Z sync-tag: 2024-04-03T16:00:16.3043658Z timeout-minutes: 240 2024-04-03T16:00:16.3044039Z use-gha: 2024-04-03T16:00:16.3044361Z dashboard-tag: 2024-04-03T16:00:16.3044700Z s3-bucket: gha-artifacts 2024-04-03T16:00:16.3045086Z aws-role-to-assume: 2024-04-03T16:00:16.3045453Z ##[endgroup] 2024-04-03T16:00:16.3046257Z Complete job name: linux-jammy-py3.8-gcc11 / test (distributed, 1, 2, linux.2xlarge) 2024-04-03T16:00:16.3882099Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-04-03T16:00:16.3882802Z with: 2024-04-03T16:00:16.3883499Z github-secret: *** 2024-04-03T16:00:16.3884500Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-04-03T16:00:16.3885751Z activate-with-label: false 2024-04-03T16:00:16.3886133Z label: with-ssh 2024-04-03T16:00:16.3886487Z remove-existing-keys: true 2024-04-03T16:00:16.3886871Z env: 2024-04-03T16:00:16.3887174Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:00:16.3887595Z ##[endgroup] 2024-04-03T16:00:16.4811314Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-04-03T16:00:16.8923592Z Grabbing public ssh keys from https://github.com/caic99.keys 2024-04-03T16:00:16.9646661Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-04-03T16:00:16.9660348Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-04-03T16:00:16.9686882Z Login using: ssh ec2-user@ec2-54-205-160-136.compute-1.amazonaws.com 2024-04-03T16:00:16.9687832Z All testing is done inside the container, to start an interactive session run: 2024-04-03T16:00:16.9688652Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-04-03T16:00:16.9939659Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-04-03T16:00:16.9940205Z with: 2024-04-03T16:00:16.9940470Z submodules: recursive 2024-04-03T16:00:16.9940788Z fetch-depth: 0 2024-04-03T16:00:16.9941041Z env: 2024-04-03T16:00:16.9941293Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:00:16.9941624Z ##[endgroup] 2024-04-03T16:00:17.0160800Z ##[group]Run retry () { 2024-04-03T16:00:17.0161217Z retry () { 2024-04-03T16:00:17.0161693Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-04-03T16:00:17.0162242Z } 2024-04-03T16:00:17.0162633Z echo "${GITHUB_WORKSPACE}" 2024-04-03T16:00:17.0163036Z if [ -z "${NO_SUDO}" ]; then 2024-04-03T16:00:17.0163486Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-04-03T16:00:17.0163930Z else 2024-04-03T16:00:17.0164254Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-04-03T16:00:17.0164652Z fi 2024-04-03T16:00:17.0165003Z mkdir "${GITHUB_WORKSPACE}" 2024-04-03T16:00:17.0188473Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:00:17.0188988Z env: 2024-04-03T16:00:17.0189238Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:00:17.0189563Z NO_SUDO: 2024-04-03T16:00:17.0189822Z ##[endgroup] 2024-04-03T16:00:17.0283287Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:00:18.9265052Z ##[group]Run malfet/checkout@silent-checkout 2024-04-03T16:00:18.9265515Z with: 2024-04-03T16:00:18.9265818Z ref: d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:00:18.9266252Z fetch-depth: 0 2024-04-03T16:00:18.9266543Z submodules: recursive 2024-04-03T16:00:18.9266851Z quiet-checkout: true 2024-04-03T16:00:18.9267175Z repository: pytorch/pytorch 2024-04-03T16:00:18.9267645Z token: *** 2024-04-03T16:00:18.9267898Z ssh-strict: true 2024-04-03T16:00:18.9268202Z persist-credentials: true 2024-04-03T16:00:18.9268538Z clean: true 2024-04-03T16:00:18.9268833Z sparse-checkout-cone-mode: true 2024-04-03T16:00:18.9269213Z lfs: false 2024-04-03T16:00:18.9269490Z set-safe-directory: true 2024-04-03T16:00:18.9269789Z env: 2024-04-03T16:00:18.9270041Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:00:18.9270358Z ##[endgroup] 2024-04-03T16:00:19.0225123Z Syncing repository: pytorch/pytorch 2024-04-03T16:00:19.0226784Z ##[group]Getting Git version info 2024-04-03T16:00:19.0227737Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-04-03T16:00:19.0268035Z [command]/usr/bin/git version 2024-04-03T16:00:19.0268852Z git version 2.40.1 2024-04-03T16:00:19.0270422Z ##[endgroup] 2024-04-03T16:00:19.0282640Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/bbea9c5f-ed70-4c3c-94c3-06325a7f197e' before making global git config changes 2024-04-03T16:00:19.0283928Z Adding repository directory to the temporary git global config as a safe directory 2024-04-03T16:00:19.0285058Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:00:19.0286937Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-04-03T16:00:19.0287841Z ##[group]Initializing the repository 2024-04-03T16:00:19.0288515Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:00:19.0309515Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-04-03T16:00:19.0310402Z hint: is subject to change. To configure the initial branch name to use in all 2024-04-03T16:00:19.0311667Z hint: of your new repositories, which will suppress this warning, call: 2024-04-03T16:00:19.0312646Z hint: 2024-04-03T16:00:19.0313221Z hint: git config --global init.defaultBranch 2024-04-03T16:00:19.0313681Z hint: 2024-04-03T16:00:19.0314171Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-04-03T16:00:19.0315018Z hint: 'development'. The just-created branch can be renamed via this command: 2024-04-03T16:00:19.0316027Z hint: 2024-04-03T16:00:19.0316350Z hint: git branch -m 2024-04-03T16:00:19.0317114Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-04-03T16:00:19.0320288Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-04-03T16:00:19.0349797Z ##[endgroup] 2024-04-03T16:00:19.0350333Z ##[group]Disabling automatic garbage collection 2024-04-03T16:00:19.0353445Z [command]/usr/bin/git config --local gc.auto 0 2024-04-03T16:00:19.0381892Z ##[endgroup] 2024-04-03T16:00:19.0382366Z ##[group]Setting up auth 2024-04-03T16:00:19.0387760Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-04-03T16:00:19.0416671Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-04-03T16:00:19.0650281Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-04-03T16:00:19.0678538Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-04-03T16:00:19.0910664Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-04-03T16:00:19.0950105Z ##[endgroup] 2024-04-03T16:00:19.0950621Z ##[group]Fetching the repository 2024-04-03T16:00:19.0956574Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-04-03T16:00:24.3130547Z remote: Enumerating objects: 1123747 2024-04-03T16:00:24.3131473Z remote: Enumerating objects: 1124401, done. 2024-04-03T16:00:24.3132243Z remote: Counting objects: 0% (1/654) 2024-04-03T16:00:24.3133013Z remote: Counting objects: 1% (7/654) 2024-04-03T16:00:24.3133794Z remote: Counting objects: 2% (14/654) 2024-04-03T16:00:24.3134626Z remote: Counting objects: 3% (20/654) 2024-04-03T16:00:24.3135455Z remote: Counting objects: 4% (27/654) 2024-04-03T16:00:24.3136354Z remote: Counting objects: 5% (33/654) 2024-04-03T16:00:24.3137058Z remote: Counting objects: 6% (40/654) 2024-04-03T16:00:24.3138051Z remote: Counting objects: 7% (46/654) 2024-04-03T16:00:24.3139157Z remote: Counting objects: 8% (53/654) 2024-04-03T16:00:24.3140248Z remote: Counting objects: 9% (59/654) 2024-04-03T16:00:24.3141353Z remote: Counting objects: 10% (66/654) 2024-04-03T16:00:24.3142457Z remote: Counting objects: 11% (72/654) 2024-04-03T16:00:24.3143563Z remote: Counting objects: 12% (79/654) 2024-04-03T16:00:24.3144656Z remote: Counting objects: 13% (86/654) 2024-04-03T16:00:24.3145758Z remote: Counting objects: 14% (92/654) 2024-04-03T16:00:24.3146884Z remote: Counting objects: 15% (99/654) 2024-04-03T16:00:24.3148460Z remote: Counting objects: 16% (105/654) 2024-04-03T16:00:24.3149597Z remote: Counting objects: 17% (112/654) 2024-04-03T16:00:24.3150730Z remote: Counting objects: 18% (118/654) 2024-04-03T16:00:24.3151836Z remote: Counting objects: 19% (125/654) 2024-04-03T16:00:24.3152957Z remote: Counting objects: 20% (131/654) 2024-04-03T16:00:24.3154074Z remote: Counting objects: 21% (138/654) 2024-04-03T16:00:24.3155166Z remote: Counting objects: 22% (144/654) 2024-04-03T16:00:24.3156436Z remote: Counting objects: 23% (151/654) 2024-04-03T16:00:24.3157346Z remote: Counting objects: 24% (157/654) 2024-04-03T16:00:24.3157997Z remote: Counting objects: 25% (164/654) 2024-04-03T16:00:24.3158884Z remote: Counting objects: 26% (171/654) 2024-04-03T16:00:24.3159797Z remote: Counting objects: 27% (177/654) 2024-04-03T16:00:24.3160476Z remote: Counting objects: 28% (184/654) 2024-04-03T16:00:24.3161216Z remote: Counting objects: 29% (190/654) 2024-04-03T16:00:24.3161926Z remote: Counting objects: 30% (197/654) 2024-04-03T16:00:24.3162494Z remote: Counting objects: 31% (203/654) 2024-04-03T16:00:24.3162982Z remote: Counting objects: 32% (210/654) 2024-04-03T16:00:24.3163466Z remote: Counting objects: 33% (216/654) 2024-04-03T16:00:24.3163944Z remote: Counting objects: 34% (223/654) 2024-04-03T16:00:24.3164415Z remote: Counting objects: 35% (229/654) 2024-04-03T16:00:24.3164893Z remote: Counting objects: 36% (236/654) 2024-04-03T16:00:24.3165376Z remote: Counting objects: 37% (242/654) 2024-04-03T16:00:24.3165841Z remote: Counting objects: 38% (249/654) 2024-04-03T16:00:24.3166321Z remote: Counting objects: 39% (256/654) 2024-04-03T16:00:24.3166798Z remote: Counting objects: 40% (262/654) 2024-04-03T16:00:24.3167272Z remote: Counting objects: 41% (269/654) 2024-04-03T16:00:24.3167759Z remote: Counting objects: 42% (275/654) 2024-04-03T16:00:24.3168233Z remote: Counting objects: 43% (282/654) 2024-04-03T16:00:24.3168697Z remote: Counting objects: 44% (288/654) 2024-04-03T16:00:24.3169176Z remote: Counting objects: 45% (295/654) 2024-04-03T16:00:24.3169653Z remote: Counting objects: 46% (301/654) 2024-04-03T16:00:24.3170117Z remote: Counting objects: 47% (308/654) 2024-04-03T16:00:24.3170594Z remote: Counting objects: 48% (314/654) 2024-04-03T16:00:24.3171073Z remote: Counting objects: 49% (321/654) 2024-04-03T16:00:24.3171534Z remote: Counting objects: 50% (327/654) 2024-04-03T16:00:24.3172014Z remote: Counting objects: 51% (334/654) 2024-04-03T16:00:24.3172491Z remote: Counting objects: 52% (341/654) 2024-04-03T16:00:24.3172955Z remote: Counting objects: 53% (347/654) 2024-04-03T16:00:24.3173440Z remote: Counting objects: 54% (354/654) 2024-04-03T16:00:24.3173921Z remote: Counting objects: 55% (360/654) 2024-04-03T16:00:24.3174383Z remote: Counting objects: 56% (367/654) 2024-04-03T16:00:24.3174858Z remote: Counting objects: 57% (373/654) 2024-04-03T16:00:24.3175337Z remote: Counting objects: 58% (380/654) 2024-04-03T16:00:24.3175814Z remote: Counting objects: 59% (386/654) 2024-04-03T16:00:24.3176276Z remote: Counting objects: 60% (393/654) 2024-04-03T16:00:24.3176753Z remote: Counting objects: 61% (399/654) 2024-04-03T16:00:24.3177226Z remote: Counting objects: 62% (406/654) 2024-04-03T16:00:24.3177711Z remote: Counting objects: 63% (413/654) 2024-04-03T16:00:24.3178199Z remote: Counting objects: 64% (419/654) 2024-04-03T16:00:24.3178683Z remote: Counting objects: 65% (426/654) 2024-04-03T16:00:24.3179161Z remote: Counting objects: 66% (432/654) 2024-04-03T16:00:24.3179629Z remote: Counting objects: 67% (439/654) 2024-04-03T16:00:24.3180270Z remote: Counting objects: 68% (445/654) 2024-04-03T16:00:24.3180745Z remote: Counting objects: 69% (452/654) 2024-04-03T16:00:24.3181208Z remote: Counting objects: 70% (458/654) 2024-04-03T16:00:24.3181689Z remote: Counting objects: 71% (465/654) 2024-04-03T16:00:24.3182167Z remote: Counting objects: 72% (471/654) 2024-04-03T16:00:24.3182648Z remote: Counting objects: 73% (478/654) 2024-04-03T16:00:24.3183113Z remote: Counting objects: 74% (484/654) 2024-04-03T16:00:24.3183588Z remote: Counting objects: 75% (491/654) 2024-04-03T16:00:24.3184065Z remote: Counting objects: 76% (498/654) 2024-04-03T16:00:24.3184534Z remote: Counting objects: 77% (504/654) 2024-04-03T16:00:24.3185014Z remote: Counting objects: 78% (511/654) 2024-04-03T16:00:24.3185492Z remote: Counting objects: 79% (517/654) 2024-04-03T16:00:24.3185958Z remote: Counting objects: 80% (524/654) 2024-04-03T16:00:24.3186508Z remote: Counting objects: 81% (530/654) 2024-04-03T16:00:24.3186986Z remote: Counting objects: 82% (537/654) 2024-04-03T16:00:24.3187448Z remote: Counting objects: 83% (543/654) 2024-04-03T16:00:24.3187927Z remote: Counting objects: 84% (550/654) 2024-04-03T16:00:24.3188410Z remote: Counting objects: 85% (556/654) 2024-04-03T16:00:24.3188878Z remote: Counting objects: 86% (563/654) 2024-04-03T16:00:24.3189355Z remote: Counting objects: 87% (569/654) 2024-04-03T16:00:24.3189830Z remote: Counting objects: 88% (576/654) 2024-04-03T16:00:24.3190293Z remote: Counting objects: 89% (583/654) 2024-04-03T16:00:24.3190771Z remote: Counting objects: 90% (589/654) 2024-04-03T16:00:24.3191256Z remote: Counting objects: 91% (596/654) 2024-04-03T16:00:24.3191725Z remote: Counting objects: 92% (602/654) 2024-04-03T16:00:24.3192200Z remote: Counting objects: 93% (609/654) 2024-04-03T16:00:24.3192684Z remote: Counting objects: 94% (615/654) 2024-04-03T16:00:24.3193149Z remote: Counting objects: 95% (622/654) 2024-04-03T16:00:24.3193626Z remote: Counting objects: 96% (628/654) 2024-04-03T16:00:24.3194106Z remote: Counting objects: 97% (635/654) 2024-04-03T16:00:24.3194587Z remote: Counting objects: 98% (641/654) 2024-04-03T16:00:24.3195050Z remote: Counting objects: 99% (648/654) 2024-04-03T16:00:24.3195705Z remote: Counting objects: 100% (654/654) 2024-04-03T16:00:24.3196232Z remote: Counting objects: 100% (654/654), done. 2024-04-03T16:00:24.3196756Z remote: Compressing objects: 0% (1/364) 2024-04-03T16:00:24.3236425Z remote: Compressing objects: 1% (4/364) 2024-04-03T16:00:24.3260772Z remote: Compressing objects: 2% (8/364) 2024-04-03T16:00:24.3442964Z remote: Compressing objects: 3% (11/364) 2024-04-03T16:00:24.3517697Z remote: Compressing objects: 4% (15/364) 2024-04-03T16:00:24.3763514Z remote: Compressing objects: 5% (19/364) 2024-04-03T16:00:24.3971248Z remote: Compressing objects: 6% (22/364) 2024-04-03T16:00:24.4079325Z remote: Compressing objects: 7% (26/364) 2024-04-03T16:00:24.4140832Z remote: Compressing objects: 8% (30/364) 2024-04-03T16:00:24.4255432Z remote: Compressing objects: 9% (33/364) 2024-04-03T16:00:24.4321898Z remote: Compressing objects: 10% (37/364) 2024-04-03T16:00:24.4392257Z remote: Compressing objects: 11% (41/364) 2024-04-03T16:00:24.4413589Z remote: Compressing objects: 12% (44/364) 2024-04-03T16:00:24.4445457Z remote: Compressing objects: 13% (48/364) 2024-04-03T16:00:24.4458896Z remote: Compressing objects: 14% (51/364) 2024-04-03T16:00:24.4470472Z remote: Compressing objects: 15% (55/364) 2024-04-03T16:00:24.4491196Z remote: Compressing objects: 16% (59/364) 2024-04-03T16:00:24.4514358Z remote: Compressing objects: 17% (62/364) 2024-04-03T16:00:24.4523970Z remote: Compressing objects: 18% (66/364) 2024-04-03T16:00:24.4529430Z remote: Compressing objects: 19% (70/364) 2024-04-03T16:00:24.4533603Z remote: Compressing objects: 20% (73/364) 2024-04-03T16:00:24.4534131Z remote: Compressing objects: 21% (77/364) 2024-04-03T16:00:24.4537726Z remote: Compressing objects: 22% (81/364) 2024-04-03T16:00:24.4540639Z remote: Compressing objects: 23% (84/364) 2024-04-03T16:00:24.4541160Z remote: Compressing objects: 24% (88/364) 2024-04-03T16:00:24.4550895Z remote: Compressing objects: 25% (91/364) 2024-04-03T16:00:24.4564304Z remote: Compressing objects: 26% (95/364) 2024-04-03T16:00:24.4564829Z remote: Compressing objects: 27% (99/364) 2024-04-03T16:00:24.4565342Z remote: Compressing objects: 28% (102/364) 2024-04-03T16:00:24.4568302Z remote: Compressing objects: 29% (106/364) 2024-04-03T16:00:24.4573937Z remote: Compressing objects: 30% (110/364) 2024-04-03T16:00:24.4582671Z remote: Compressing objects: 31% (113/364) 2024-04-03T16:00:24.4586789Z remote: Compressing objects: 32% (117/364) 2024-04-03T16:00:24.4590262Z remote: Compressing objects: 33% (121/364) 2024-04-03T16:00:24.4594197Z remote: Compressing objects: 34% (124/364) 2024-04-03T16:00:24.4601001Z remote: Compressing objects: 35% (128/364) 2024-04-03T16:00:24.4604053Z remote: Compressing objects: 36% (132/364) 2024-04-03T16:00:24.4604565Z remote: Compressing objects: 37% (135/364) 2024-04-03T16:00:24.4605089Z remote: Compressing objects: 38% (139/364) 2024-04-03T16:00:24.4607271Z remote: Compressing objects: 39% (142/364) 2024-04-03T16:00:24.4612279Z remote: Compressing objects: 40% (146/364) 2024-04-03T16:00:24.4612804Z remote: Compressing objects: 41% (150/364) 2024-04-03T16:00:24.4621793Z remote: Compressing objects: 42% (153/364) 2024-04-03T16:00:24.4622334Z remote: Compressing objects: 43% (157/364) 2024-04-03T16:00:24.4622846Z remote: Compressing objects: 44% (161/364) 2024-04-03T16:00:24.4625606Z remote: Compressing objects: 45% (164/364) 2024-04-03T16:00:24.4632414Z remote: Compressing objects: 46% (168/364) 2024-04-03T16:00:24.4632921Z remote: Compressing objects: 47% (172/364) 2024-04-03T16:00:24.4637207Z remote: Compressing objects: 48% (175/364) 2024-04-03T16:00:24.4640259Z remote: Compressing objects: 49% (179/364) 2024-04-03T16:00:24.4640786Z remote: Compressing objects: 50% (182/364) 2024-04-03T16:00:24.4641286Z remote: Compressing objects: 51% (186/364) 2024-04-03T16:00:24.4643301Z remote: Compressing objects: 52% (190/364) 2024-04-03T16:00:24.4644285Z remote: Compressing objects: 53% (193/364) 2024-04-03T16:00:24.4644859Z remote: Compressing objects: 54% (197/364) 2024-04-03T16:00:24.4645369Z remote: Compressing objects: 55% (201/364) 2024-04-03T16:00:24.4645905Z remote: Compressing objects: 56% (204/364) 2024-04-03T16:00:24.4646587Z remote: Compressing objects: 57% (208/364) 2024-04-03T16:00:24.4647226Z remote: Compressing objects: 58% (212/364) 2024-04-03T16:00:24.4647741Z remote: Compressing objects: 59% (215/364) 2024-04-03T16:00:24.4648252Z remote: Compressing objects: 60% (219/364) 2024-04-03T16:00:24.4648750Z remote: Compressing objects: 61% (223/364) 2024-04-03T16:00:24.4649264Z remote: Compressing objects: 62% (226/364) 2024-04-03T16:00:24.4649775Z remote: Compressing objects: 63% (230/364) 2024-04-03T16:00:24.4650270Z remote: Compressing objects: 64% (233/364) 2024-04-03T16:00:24.4650812Z remote: Compressing objects: 65% (237/364) 2024-04-03T16:00:24.4651576Z remote: Compressing objects: 66% (241/364) 2024-04-03T16:00:24.4652075Z remote: Compressing objects: 67% (244/364) 2024-04-03T16:00:24.4652599Z remote: Compressing objects: 68% (248/364) 2024-04-03T16:00:24.4653352Z remote: Compressing objects: 69% (252/364) 2024-04-03T16:00:24.4653847Z remote: Compressing objects: 70% (255/364) 2024-04-03T16:00:24.4654364Z remote: Compressing objects: 71% (259/364) 2024-04-03T16:00:24.4659743Z remote: Compressing objects: 72% (263/364) 2024-04-03T16:00:24.4660270Z remote: Compressing objects: 73% (266/364) 2024-04-03T16:00:24.4660768Z remote: Compressing objects: 74% (270/364) 2024-04-03T16:00:24.4661283Z remote: Compressing objects: 75% (273/364) 2024-04-03T16:00:24.4661994Z remote: Compressing objects: 76% (277/364) 2024-04-03T16:00:24.4662591Z remote: Compressing objects: 77% (281/364) 2024-04-03T16:00:24.4663103Z remote: Compressing objects: 78% (284/364) 2024-04-03T16:00:24.4663611Z remote: Compressing objects: 79% (288/364) 2024-04-03T16:00:24.4666077Z remote: Compressing objects: 80% (292/364) 2024-04-03T16:00:24.4666765Z remote: Compressing objects: 81% (295/364) 2024-04-03T16:00:24.4667288Z remote: Compressing objects: 82% (299/364) 2024-04-03T16:00:24.4669255Z remote: Compressing objects: 83% (303/364) 2024-04-03T16:00:24.4669759Z remote: Compressing objects: 84% (306/364) 2024-04-03T16:00:24.4670276Z remote: Compressing objects: 85% (310/364) 2024-04-03T16:00:24.4675016Z remote: Compressing objects: 86% (314/364) 2024-04-03T16:00:24.4675729Z remote: Compressing objects: 87% (317/364) 2024-04-03T16:00:24.4676245Z remote: Compressing objects: 88% (321/364) 2024-04-03T16:00:24.4676767Z remote: Compressing objects: 89% (324/364) 2024-04-03T16:00:24.4677528Z remote: Compressing objects: 90% (328/364) 2024-04-03T16:00:24.4678043Z remote: Compressing objects: 91% (332/364) 2024-04-03T16:00:24.4678554Z remote: Compressing objects: 92% (335/364) 2024-04-03T16:00:24.4679077Z remote: Compressing objects: 93% (339/364) 2024-04-03T16:00:24.4679953Z remote: Compressing objects: 94% (343/364) 2024-04-03T16:00:24.4680467Z remote: Compressing objects: 95% (346/364) 2024-04-03T16:00:24.4683943Z remote: Compressing objects: 96% (350/364) 2024-04-03T16:00:24.4684455Z remote: Compressing objects: 97% (354/364) 2024-04-03T16:00:24.4684965Z remote: Compressing objects: 98% (357/364) 2024-04-03T16:00:24.4685481Z remote: Compressing objects: 99% (361/364) 2024-04-03T16:00:24.4685979Z remote: Compressing objects: 100% (364/364) 2024-04-03T16:00:24.4686532Z remote: Compressing objects: 100% (364/364), done. 2024-04-03T16:00:43.8957952Z remote: Total 1124401 (delta 359), reused 524 (delta 290), pack-reused 1123747 2024-04-03T16:01:14.6200974Z [command]/usr/bin/git rev-parse --verify --quiet d253bd7befc64137de913dbb8727b511f1052411^{object} 2024-04-03T16:01:14.6227155Z d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:01:14.6231943Z ##[endgroup] 2024-04-03T16:01:14.6232563Z ##[group]Determining the checkout info 2024-04-03T16:01:14.6233341Z ##[endgroup] 2024-04-03T16:01:14.6233919Z ##[group]Checking out the ref 2024-04-03T16:01:14.6237110Z [command]/usr/bin/git checkout --quiet --force d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:01:16.0460889Z ##[endgroup] 2024-04-03T16:01:16.0461467Z ##[group]Setting up auth for fetching submodules 2024-04-03T16:01:16.0465387Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-04-03T16:01:16.0510372Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-04-03T16:01:16.0538646Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-04-03T16:01:16.0565307Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-04-03T16:01:16.0588858Z ##[endgroup] 2024-04-03T16:01:16.0589364Z ##[group]Fetching submodules 2024-04-03T16:01:16.0592658Z [command]/usr/bin/git submodule sync --recursive 2024-04-03T16:01:16.0856034Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-04-03T16:01:16.1113051Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-04-03T16:01:16.1114959Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-04-03T16:01:16.1116753Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-04-03T16:01:16.1118357Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-04-03T16:01:16.1120082Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2024-04-03T16:01:16.1122688Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-04-03T16:01:16.1124815Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-04-03T16:01:16.1127354Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-04-03T16:01:16.1129965Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-04-03T16:01:16.1132684Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2024-04-03T16:01:16.1135756Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-04-03T16:01:16.1138617Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-04-03T16:01:16.1141668Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-04-03T16:01:16.1144869Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-04-03T16:01:16.1148277Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-04-03T16:01:16.1151525Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-04-03T16:01:16.1154999Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-04-03T16:01:16.1159144Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:01:16.1162702Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-04-03T16:01:16.1166709Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-04-03T16:01:16.1170429Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-04-03T16:01:16.1174622Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2024-04-03T16:01:16.1178681Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-04-03T16:01:16.1182861Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-04-03T16:01:16.1187262Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-04-03T16:01:16.1191542Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-04-03T16:01:16.1196279Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2024-04-03T16:01:16.1201176Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-04-03T16:01:16.1205739Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-04-03T16:01:16.1210670Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2024-04-03T16:01:16.1215474Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-04-03T16:01:16.1220539Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-04-03T16:01:16.1225692Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-04-03T16:01:16.1231134Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-04-03T16:01:16.1236215Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-04-03T16:01:16.1242219Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-04-03T16:01:16.1247601Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-04-03T16:01:16.1253132Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2024-04-03T16:01:16.1260478Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-04-03T16:01:16.1268192Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2024-04-03T16:01:16.1295262Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-04-03T16:01:16.3808818Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-04-03T16:01:16.5834016Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-04-03T16:01:16.7682440Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-04-03T16:01:17.0045407Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2024-04-03T16:01:17.2438443Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-04-03T16:01:19.5445798Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-04-03T16:01:31.8209643Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-04-03T16:01:32.2305036Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-04-03T16:01:32.8362532Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2024-04-03T16:01:34.4222551Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-04-03T16:01:35.6635257Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-04-03T16:01:38.0873593Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-04-03T16:01:43.4997953Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-04-03T16:01:44.7578568Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-04-03T16:01:46.5433346Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-04-03T16:01:48.1740227Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-04-03T16:01:48.3296228Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-04-03T16:01:48.7348791Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-04-03T16:01:49.0309761Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-04-03T16:01:50.2299847Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-04-03T16:01:50.6018165Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2024-04-03T16:01:50.7630960Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-04-03T16:01:51.0242722Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-04-03T16:01:52.5027568Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-04-03T16:01:53.1201280Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-04-03T16:01:53.7179371Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2024-04-03T16:01:53.9975999Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-04-03T16:02:00.5541844Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-04-03T16:02:02.9553157Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2024-04-03T16:02:03.4945002Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-04-03T16:02:03.7078211Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-04-03T16:02:12.3198549Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-04-03T16:02:12.5363077Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-04-03T16:02:12.7926649Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-04-03T16:02:13.7738842Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-04-03T16:02:14.1819326Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-04-03T16:02:14.9355876Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2024-04-03T16:02:16.8372626Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-04-03T16:02:17.3423455Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2024-04-03T16:02:19.8830758Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-04-03T16:02:19.8926772Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-04-03T16:02:19.9001402Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-04-03T16:02:19.9204641Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-04-03T16:02:19.9408505Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2024-04-03T16:02:19.9757039Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-04-03T16:02:20.8230131Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-04-03T16:02:20.8433557Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-04-03T16:02:20.9296683Z Submodule path 'third_party/cpuinfo': checked out 'd6860c477c99f1fce9e28eb206891af3c0e1a1d7' 2024-04-03T16:02:20.9615058Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2024-04-03T16:02:20.9868204Z Submodule path 'third_party/cudnn_frontend': checked out '150798fe976556078f443fdb059a1ff0361f58a2' 2024-04-03T16:02:21.4429863Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-04-03T16:02:21.6735388Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-04-03T16:02:21.7360178Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-04-03T16:02:21.7375912Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:02:21.7378201Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:02:21.7380493Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:02:21.7383167Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:02:21.7386196Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:02:21.7409171Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-04-03T16:02:22.7856547Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-04-03T16:02:23.4430085Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-04-03T16:02:26.0247862Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-04-03T16:02:27.3228567Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-04-03T16:02:27.7243144Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-04-03T16:02:27.8131962Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-04-03T16:02:28.1904139Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-04-03T16:02:28.2484291Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-04-03T16:02:28.2589237Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-04-03T16:02:28.3637918Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-04-03T16:02:28.3960647Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-04-03T16:02:28.4044543Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-04-03T16:02:28.4415706Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-04-03T16:02:28.4637611Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-04-03T16:02:28.5053307Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-04-03T16:02:28.5164381Z Submodule path 'third_party/ideep': checked out '8a6cc4e09dc509f04f83c085e38786b1fb44e14d' 2024-04-03T16:02:28.5176790Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-04-03T16:02:28.5197004Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-04-03T16:02:40.3258284Z Submodule path 'third_party/ideep/mkl-dnn': checked out '86e6af5974177e513fd3fee58425e1063e7f1361' 2024-04-03T16:02:40.3351630Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2024-04-03T16:02:40.3494812Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-04-03T16:02:40.4394828Z Submodule path 'third_party/kineto': checked out '8466a8b111b36dc725e6855d52a0b133d925a8e0' 2024-04-03T16:02:40.4410705Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:02:40.4413277Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:02:40.4415063Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:02:40.4436242Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-04-03T16:02:41.3967905Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-04-03T16:02:42.8344447Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-04-03T16:02:44.1965892Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-04-03T16:02:44.1980812Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:02:44.1984400Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:02:44.1988005Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:02:44.1991661Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:02:44.1995372Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:02:44.1999481Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:02:44.2003256Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:02:44.2005269Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:02:44.2025514Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-04-03T16:02:45.3602029Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-04-03T16:02:45.9985344Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-04-03T16:02:47.4419871Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-04-03T16:02:47.7985641Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-04-03T16:02:48.2859379Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-04-03T16:02:49.6208632Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-04-03T16:02:56.3246228Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-04-03T16:02:56.8476440Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-04-03T16:02:56.8633334Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-04-03T16:02:56.8979120Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-04-03T16:02:56.9096993Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-04-03T16:02:56.9109661Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:02:56.9132336Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-04-03T16:02:57.3878584Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-04-03T16:02:57.4040062Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-04-03T16:02:57.4405326Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-04-03T16:02:57.5358012Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-04-03T16:02:57.5500511Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-04-03T16:02:57.5846253Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-04-03T16:02:57.6384335Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-04-03T16:02:57.6713142Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-04-03T16:02:57.6933932Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-04-03T16:02:57.7063023Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2024-04-03T16:02:57.8043559Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-04-03T16:02:58.0988729Z Submodule path 'third_party/onnx': checked out 'ccde5da81388ffa770ca98b64e07f803ad089414' 2024-04-03T16:02:58.1021396Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-04-03T16:02:58.1023506Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-04-03T16:02:58.1045543Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-04-03T16:02:58.6969965Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-04-03T16:02:59.7224339Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-04-03T16:02:59.7527412Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-04-03T16:02:59.7696441Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2024-04-03T16:02:59.7716532Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:02:59.7731184Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2024-04-03T16:03:02.5430920Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2024-04-03T16:03:02.5450391Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:03:02.5452345Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:03:02.5476067Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2024-04-03T16:03:03.0842426Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2024-04-03T16:03:04.1431252Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2024-04-03T16:03:04.2104857Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2024-04-03T16:03:04.2118731Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:03:04.2141262Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2024-04-03T16:03:04.4867408Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-04-03T16:03:04.4950977Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-04-03T16:03:04.7319375Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-04-03T16:03:04.7338231Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:03:04.7340127Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-04-03T16:03:04.7363347Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-04-03T16:03:05.2090415Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-04-03T16:03:06.6614286Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-04-03T16:03:06.7273152Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-04-03T16:03:06.7350105Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-04-03T16:03:06.7458947Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-04-03T16:03:06.7753187Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-04-03T16:03:06.8012547Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-04-03T16:03:06.8384427Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-04-03T16:03:06.9369341Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2024-04-03T16:03:06.9602855Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-04-03T16:03:06.9617594Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:03:06.9619358Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:03:06.9621601Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:03:06.9623870Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:03:06.9646800Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-04-03T16:03:08.3164930Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-04-03T16:03:08.6273831Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-04-03T16:03:09.9634807Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-04-03T16:03:11.0869161Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-04-03T16:03:11.0999791Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-04-03T16:03:11.1632918Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-04-03T16:03:11.1887028Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-04-03T16:03:11.1900011Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:03:11.1921466Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-04-03T16:03:11.4441156Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-04-03T16:03:11.5803547Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2024-04-03T16:03:11.5833287Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-04-03T16:03:11.6098143Z Entering 'android/libs/fbjni' 2024-04-03T16:03:11.6132282Z Entering 'third_party/FP16' 2024-04-03T16:03:11.6166709Z Entering 'third_party/FXdiv' 2024-04-03T16:03:11.6201168Z Entering 'third_party/NNPACK' 2024-04-03T16:03:11.6234267Z Entering 'third_party/QNNPACK' 2024-04-03T16:03:11.6268744Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:03:11.6303811Z Entering 'third_party/XNNPACK' 2024-04-03T16:03:11.6356754Z Entering 'third_party/benchmark' 2024-04-03T16:03:11.6389996Z Entering 'third_party/cpuinfo' 2024-04-03T16:03:11.6424831Z Entering 'third_party/cub' 2024-04-03T16:03:11.6458891Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:03:11.6494147Z Entering 'third_party/cutlass' 2024-04-03T16:03:11.6535092Z Entering 'third_party/eigen' 2024-04-03T16:03:11.6571418Z Entering 'third_party/fbgemm' 2024-04-03T16:03:11.6604710Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:03:11.6636823Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:03:11.6669219Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:03:11.6707957Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:03:11.6740919Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:03:11.6774973Z Entering 'third_party/flatbuffers' 2024-04-03T16:03:11.6812142Z Entering 'third_party/fmt' 2024-04-03T16:03:11.6846642Z Entering 'third_party/foxi' 2024-04-03T16:03:11.6879944Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:03:11.6913610Z Entering 'third_party/gloo' 2024-04-03T16:03:11.6948554Z Entering 'third_party/googletest' 2024-04-03T16:03:11.6982301Z Entering 'third_party/ideep' 2024-04-03T16:03:11.7016533Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:03:11.7056026Z Entering 'third_party/ios-cmake' 2024-04-03T16:03:11.7089376Z Entering 'third_party/ittapi' 2024-04-03T16:03:11.7122247Z Entering 'third_party/kineto' 2024-04-03T16:03:11.7156389Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:03:11.7189224Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:03:11.7222889Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:03:11.7255497Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:03:11.7288442Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:03:11.7320243Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:03:11.7353562Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:03:11.7386199Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:03:11.7419152Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:03:11.7452884Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:03:11.7487112Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:03:11.7519841Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:03:11.7553585Z Entering 'third_party/mimalloc' 2024-04-03T16:03:11.7588319Z Entering 'third_party/nccl/nccl' 2024-04-03T16:03:11.7622249Z Entering 'third_party/neon2sse' 2024-04-03T16:03:11.7656790Z Entering 'third_party/nlohmann' 2024-04-03T16:03:11.7691826Z Entering 'third_party/onnx' 2024-04-03T16:03:11.7741356Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:03:11.7775130Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:03:11.7810588Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:03:11.7844137Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:03:11.7881708Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:03:11.7914413Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:03:11.7946133Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:03:11.7982921Z Entering 'third_party/pocketfft' 2024-04-03T16:03:11.8017395Z Entering 'third_party/protobuf' 2024-04-03T16:03:11.8055225Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:03:11.8088810Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:03:11.8123732Z Entering 'third_party/psimd' 2024-04-03T16:03:11.8158366Z Entering 'third_party/pthreadpool' 2024-04-03T16:03:11.8191941Z Entering 'third_party/pybind11' 2024-04-03T16:03:11.8225315Z Entering 'third_party/python-peachpy' 2024-04-03T16:03:11.8259403Z Entering 'third_party/sleef' 2024-04-03T16:03:11.8293194Z Entering 'third_party/tbb' 2024-04-03T16:03:11.8329472Z Entering 'third_party/tensorpipe' 2024-04-03T16:03:11.8362953Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:03:11.8395168Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:03:11.8427181Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:03:11.8460840Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:03:11.8492657Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:03:11.8528287Z Entering 'third_party/zstd' 2024-04-03T16:03:11.8573938Z ##[endgroup] 2024-04-03T16:03:11.8574994Z ##[group]Persisting credentials for submodules 2024-04-03T16:03:11.8578866Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-04-03T16:03:11.8842750Z Entering 'android/libs/fbjni' 2024-04-03T16:03:11.8888772Z Entering 'third_party/FP16' 2024-04-03T16:03:11.8929898Z Entering 'third_party/FXdiv' 2024-04-03T16:03:11.8971100Z Entering 'third_party/NNPACK' 2024-04-03T16:03:11.9012260Z Entering 'third_party/QNNPACK' 2024-04-03T16:03:11.9053887Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:03:11.9095180Z Entering 'third_party/XNNPACK' 2024-04-03T16:03:11.9153120Z Entering 'third_party/benchmark' 2024-04-03T16:03:11.9195044Z Entering 'third_party/cpuinfo' 2024-04-03T16:03:11.9236935Z Entering 'third_party/cub' 2024-04-03T16:03:11.9277772Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:03:11.9319018Z Entering 'third_party/cutlass' 2024-04-03T16:03:11.9368777Z Entering 'third_party/eigen' 2024-04-03T16:03:11.9413863Z Entering 'third_party/fbgemm' 2024-04-03T16:03:11.9455820Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:03:11.9496183Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:03:11.9537853Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:03:11.9586030Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:03:11.9627745Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:03:11.9669348Z Entering 'third_party/flatbuffers' 2024-04-03T16:03:11.9715355Z Entering 'third_party/fmt' 2024-04-03T16:03:11.9757314Z Entering 'third_party/foxi' 2024-04-03T16:03:11.9798998Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:03:11.9842434Z Entering 'third_party/gloo' 2024-04-03T16:03:11.9884149Z Entering 'third_party/googletest' 2024-04-03T16:03:11.9925967Z Entering 'third_party/ideep' 2024-04-03T16:03:11.9967418Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:03:12.0014763Z Entering 'third_party/ios-cmake' 2024-04-03T16:03:12.0057485Z Entering 'third_party/ittapi' 2024-04-03T16:03:12.0098613Z Entering 'third_party/kineto' 2024-04-03T16:03:12.0139733Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:03:12.0181205Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:03:12.0223624Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:03:12.0264167Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:03:12.0305774Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:03:12.0345355Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:03:12.0387628Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:03:12.0430456Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:03:12.0472357Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:03:12.0514414Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:03:12.0557175Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:03:12.0597310Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:03:12.0640149Z Entering 'third_party/mimalloc' 2024-04-03T16:03:12.0683146Z Entering 'third_party/nccl/nccl' 2024-04-03T16:03:12.0725119Z Entering 'third_party/neon2sse' 2024-04-03T16:03:12.0767500Z Entering 'third_party/nlohmann' 2024-04-03T16:03:12.0809528Z Entering 'third_party/onnx' 2024-04-03T16:03:12.0867343Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:03:12.0910641Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:03:12.0953245Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:03:12.0995047Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:03:12.1039786Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:03:12.1081230Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:03:12.1121430Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:03:12.1166507Z Entering 'third_party/pocketfft' 2024-04-03T16:03:12.1209529Z Entering 'third_party/protobuf' 2024-04-03T16:03:12.1255656Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:03:12.1297007Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:03:12.1340737Z Entering 'third_party/psimd' 2024-04-03T16:03:12.1383299Z Entering 'third_party/pthreadpool' 2024-04-03T16:03:12.1425620Z Entering 'third_party/pybind11' 2024-04-03T16:03:12.1468098Z Entering 'third_party/python-peachpy' 2024-04-03T16:03:12.1509846Z Entering 'third_party/sleef' 2024-04-03T16:03:12.1551498Z Entering 'third_party/tbb' 2024-04-03T16:03:12.1594964Z Entering 'third_party/tensorpipe' 2024-04-03T16:03:12.1637043Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:03:12.1678134Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:03:12.1718588Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:03:12.1759806Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:03:12.1799753Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:03:12.1843263Z Entering 'third_party/zstd' 2024-04-03T16:03:12.1898296Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-04-03T16:03:12.2163793Z Entering 'android/libs/fbjni' 2024-04-03T16:03:12.2202957Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-04-03T16:03:12.2219265Z Entering 'third_party/FP16' 2024-04-03T16:03:12.2259969Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-04-03T16:03:12.2273788Z Entering 'third_party/FXdiv' 2024-04-03T16:03:12.2313971Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-04-03T16:03:12.2329723Z Entering 'third_party/NNPACK' 2024-04-03T16:03:12.2369235Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-04-03T16:03:12.2383727Z Entering 'third_party/QNNPACK' 2024-04-03T16:03:12.2423106Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2024-04-03T16:03:12.2438938Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:03:12.2479903Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-04-03T16:03:12.2494812Z Entering 'third_party/XNNPACK' 2024-04-03T16:03:12.2535417Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-04-03T16:03:12.2566841Z Entering 'third_party/benchmark' 2024-04-03T16:03:12.2607551Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:03:12.2622881Z Entering 'third_party/cpuinfo' 2024-04-03T16:03:12.2662906Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-04-03T16:03:12.2679289Z Entering 'third_party/cub' 2024-04-03T16:03:12.2719409Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2024-04-03T16:03:12.2734734Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:03:12.2776395Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-04-03T16:03:12.2791759Z Entering 'third_party/cutlass' 2024-04-03T16:03:12.2831522Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-04-03T16:03:12.2855259Z Entering 'third_party/eigen' 2024-04-03T16:03:12.2896242Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-04-03T16:03:12.2920628Z Entering 'third_party/fbgemm' 2024-04-03T16:03:12.2960789Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-04-03T16:03:12.2975058Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:03:12.3017025Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-04-03T16:03:12.3031319Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:03:12.3071261Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-04-03T16:03:12.3086559Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:03:12.3127353Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-04-03T16:03:12.3147906Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:03:12.3186883Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:03:12.3201903Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:03:12.3241458Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-04-03T16:03:12.3256640Z Entering 'third_party/flatbuffers' 2024-04-03T16:03:12.3297373Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-04-03T16:03:12.3315210Z Entering 'third_party/fmt' 2024-04-03T16:03:12.3354700Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-04-03T16:03:12.3370495Z Entering 'third_party/foxi' 2024-04-03T16:03:12.3409967Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-04-03T16:03:12.3425359Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:03:12.3468546Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-04-03T16:03:12.3483861Z Entering 'third_party/gloo' 2024-04-03T16:03:12.3522874Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-04-03T16:03:12.3537444Z Entering 'third_party/googletest' 2024-04-03T16:03:12.3577596Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:03:12.3593199Z Entering 'third_party/ideep' 2024-04-03T16:03:12.3633001Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-04-03T16:03:12.3647324Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:03:12.3685387Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-04-03T16:03:12.3707017Z Entering 'third_party/ios-cmake' 2024-04-03T16:03:12.3746867Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2024-04-03T16:03:12.3762522Z Entering 'third_party/ittapi' 2024-04-03T16:03:12.3802467Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-04-03T16:03:12.3817182Z Entering 'third_party/kineto' 2024-04-03T16:03:12.3858005Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-04-03T16:03:12.3872974Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:03:12.3913368Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-04-03T16:03:12.3928000Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:03:12.3967684Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-04-03T16:03:12.3984171Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:03:12.4023554Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-04-03T16:03:12.4038023Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:03:12.4078445Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-04-03T16:03:12.4092397Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:03:12.4132444Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-04-03T16:03:12.4146575Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:03:12.4185680Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-04-03T16:03:12.4202181Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:03:12.4242110Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-04-03T16:03:12.4257184Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:03:12.4297287Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:03:12.4311531Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:03:12.4350768Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-04-03T16:03:12.4367319Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:03:12.4406428Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-04-03T16:03:12.4432699Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:03:12.4465049Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-04-03T16:03:12.4479557Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:03:12.4518669Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-04-03T16:03:12.4534598Z Entering 'third_party/mimalloc' 2024-04-03T16:03:12.4574458Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-04-03T16:03:12.4589461Z Entering 'third_party/nccl/nccl' 2024-04-03T16:03:12.4630290Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-04-03T16:03:12.4646002Z Entering 'third_party/neon2sse' 2024-04-03T16:03:12.4684788Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2024-04-03T16:03:12.4700602Z Entering 'third_party/nlohmann' 2024-04-03T16:03:12.4740246Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-04-03T16:03:12.4756641Z Entering 'third_party/onnx' 2024-04-03T16:03:12.4795066Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-04-03T16:03:12.4825993Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:03:12.4865138Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:03:12.4882096Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:03:12.4920028Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:03:12.4937593Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:03:12.4977055Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2024-04-03T16:03:12.4990772Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:03:12.5030764Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2024-04-03T16:03:12.5049549Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:03:12.5089998Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:03:12.5104658Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:03:12.5145712Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:03:12.5159117Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:03:12.5200311Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-04-03T16:03:12.5218191Z Entering 'third_party/pocketfft' 2024-04-03T16:03:12.5257833Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-04-03T16:03:12.5272005Z Entering 'third_party/protobuf' 2024-04-03T16:03:12.5311603Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-04-03T16:03:12.5329915Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:03:12.5368105Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-04-03T16:03:12.5382241Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:03:12.5421313Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:03:12.5437336Z Entering 'third_party/psimd' 2024-04-03T16:03:12.5477164Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-04-03T16:03:12.5491631Z Entering 'third_party/pthreadpool' 2024-04-03T16:03:12.5531199Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-04-03T16:03:12.5545905Z Entering 'third_party/pybind11' 2024-04-03T16:03:12.5585348Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:03:12.5600294Z Entering 'third_party/python-peachpy' 2024-04-03T16:03:12.5640656Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-04-03T16:03:12.5655717Z Entering 'third_party/sleef' 2024-04-03T16:03:12.5694513Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-04-03T16:03:12.5709753Z Entering 'third_party/tbb' 2024-04-03T16:03:12.5748965Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2024-04-03T16:03:12.5767056Z Entering 'third_party/tensorpipe' 2024-04-03T16:03:12.5806158Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-04-03T16:03:12.5820054Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:03:12.5859910Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-04-03T16:03:12.5874140Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:03:12.5912621Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-04-03T16:03:12.5927357Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:03:12.5966075Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-04-03T16:03:12.5980773Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:03:12.6019944Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-04-03T16:03:12.6033475Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:03:12.6072648Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-04-03T16:03:12.6088985Z Entering 'third_party/zstd' 2024-04-03T16:03:12.6128688Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2024-04-03T16:03:12.7049680Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-04-03T16:03:12.7312086Z Entering 'android/libs/fbjni' 2024-04-03T16:03:12.7345789Z Entering 'third_party/FP16' 2024-04-03T16:03:12.7380993Z Entering 'third_party/FXdiv' 2024-04-03T16:03:12.7416224Z Entering 'third_party/NNPACK' 2024-04-03T16:03:12.7450766Z Entering 'third_party/QNNPACK' 2024-04-03T16:03:12.7485990Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:03:12.7519858Z Entering 'third_party/XNNPACK' 2024-04-03T16:03:12.7571138Z Entering 'third_party/benchmark' 2024-04-03T16:03:12.7605873Z Entering 'third_party/cpuinfo' 2024-04-03T16:03:12.7640962Z Entering 'third_party/cub' 2024-04-03T16:03:12.7675367Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:03:12.7709115Z Entering 'third_party/cutlass' 2024-04-03T16:03:12.7750347Z Entering 'third_party/eigen' 2024-04-03T16:03:12.7787118Z Entering 'third_party/fbgemm' 2024-04-03T16:03:12.7821799Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:03:12.7853923Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:03:12.7887787Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:03:12.7928896Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:03:12.7961966Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:03:12.7995681Z Entering 'third_party/flatbuffers' 2024-04-03T16:03:12.8033488Z Entering 'third_party/fmt' 2024-04-03T16:03:12.8067098Z Entering 'third_party/foxi' 2024-04-03T16:03:12.8100848Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:03:12.8135786Z Entering 'third_party/gloo' 2024-04-03T16:03:12.8170127Z Entering 'third_party/googletest' 2024-04-03T16:03:12.8204502Z Entering 'third_party/ideep' 2024-04-03T16:03:12.8237896Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:03:12.8277896Z Entering 'third_party/ios-cmake' 2024-04-03T16:03:12.8311669Z Entering 'third_party/ittapi' 2024-04-03T16:03:12.8345128Z Entering 'third_party/kineto' 2024-04-03T16:03:12.8379053Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:03:12.8412079Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:03:12.8445970Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:03:12.8479949Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:03:12.8552909Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:03:12.8586310Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:03:12.8620503Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:03:12.8654237Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:03:12.8687320Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:03:12.8721316Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:03:12.8756082Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:03:12.8789503Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:03:12.8823922Z Entering 'third_party/mimalloc' 2024-04-03T16:03:12.8858869Z Entering 'third_party/nccl/nccl' 2024-04-03T16:03:12.8893908Z Entering 'third_party/neon2sse' 2024-04-03T16:03:12.8927254Z Entering 'third_party/nlohmann' 2024-04-03T16:03:12.8961987Z Entering 'third_party/onnx' 2024-04-03T16:03:12.9011356Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:03:12.9043951Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:03:12.9078929Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:03:12.9112981Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:03:12.9150413Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:03:12.9183211Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:03:12.9215149Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:03:12.9252036Z Entering 'third_party/pocketfft' 2024-04-03T16:03:12.9286675Z Entering 'third_party/protobuf' 2024-04-03T16:03:12.9324292Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:03:12.9357827Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:03:12.9392210Z Entering 'third_party/psimd' 2024-04-03T16:03:12.9426256Z Entering 'third_party/pthreadpool' 2024-04-03T16:03:12.9459968Z Entering 'third_party/pybind11' 2024-04-03T16:03:12.9494963Z Entering 'third_party/python-peachpy' 2024-04-03T16:03:12.9528956Z Entering 'third_party/sleef' 2024-04-03T16:03:12.9562862Z Entering 'third_party/tbb' 2024-04-03T16:03:12.9599701Z Entering 'third_party/tensorpipe' 2024-04-03T16:03:12.9633906Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:03:12.9666909Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:03:12.9699558Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:03:12.9732372Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:03:12.9764865Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:03:12.9800122Z Entering 'third_party/zstd' 2024-04-03T16:03:12.9848853Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-04-03T16:03:13.0108389Z Entering 'android/libs/fbjni' 2024-04-03T16:03:13.0142667Z Entering 'third_party/FP16' 2024-04-03T16:03:13.0176658Z Entering 'third_party/FXdiv' 2024-04-03T16:03:13.0209860Z Entering 'third_party/NNPACK' 2024-04-03T16:03:13.0244081Z Entering 'third_party/QNNPACK' 2024-04-03T16:03:13.0278769Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T16:03:13.0312805Z Entering 'third_party/XNNPACK' 2024-04-03T16:03:13.0363205Z Entering 'third_party/benchmark' 2024-04-03T16:03:13.0396963Z Entering 'third_party/cpuinfo' 2024-04-03T16:03:13.0431008Z Entering 'third_party/cub' 2024-04-03T16:03:13.0465384Z Entering 'third_party/cudnn_frontend' 2024-04-03T16:03:13.0499298Z Entering 'third_party/cutlass' 2024-04-03T16:03:13.0542137Z Entering 'third_party/eigen' 2024-04-03T16:03:13.0578400Z Entering 'third_party/fbgemm' 2024-04-03T16:03:13.0612258Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T16:03:13.0644863Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T16:03:13.0677947Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T16:03:13.0717391Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T16:03:13.0749125Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T16:03:13.0783620Z Entering 'third_party/flatbuffers' 2024-04-03T16:03:13.0820807Z Entering 'third_party/fmt' 2024-04-03T16:03:13.0854019Z Entering 'third_party/foxi' 2024-04-03T16:03:13.0888658Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T16:03:13.0922380Z Entering 'third_party/gloo' 2024-04-03T16:03:13.0956908Z Entering 'third_party/googletest' 2024-04-03T16:03:13.0989784Z Entering 'third_party/ideep' 2024-04-03T16:03:13.1023251Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T16:03:13.1064400Z Entering 'third_party/ios-cmake' 2024-04-03T16:03:13.1098443Z Entering 'third_party/ittapi' 2024-04-03T16:03:13.1132240Z Entering 'third_party/kineto' 2024-04-03T16:03:13.1166930Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T16:03:13.1200153Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T16:03:13.1236385Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T16:03:13.1269399Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T16:03:13.1303912Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T16:03:13.1337240Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T16:03:13.1371727Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T16:03:13.1405343Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T16:03:13.1438406Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T16:03:13.1473296Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T16:03:13.1507747Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T16:03:13.1541480Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T16:03:13.1576252Z Entering 'third_party/mimalloc' 2024-04-03T16:03:13.1610281Z Entering 'third_party/nccl/nccl' 2024-04-03T16:03:13.1644341Z Entering 'third_party/neon2sse' 2024-04-03T16:03:13.1678090Z Entering 'third_party/nlohmann' 2024-04-03T16:03:13.1713310Z Entering 'third_party/onnx' 2024-04-03T16:03:13.1762641Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T16:03:13.1796603Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T16:03:13.1832872Z Entering 'third_party/onnx-tensorrt' 2024-04-03T16:03:13.1866537Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T16:03:13.1903429Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T16:03:13.1938302Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T16:03:13.1971030Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T16:03:13.2009694Z Entering 'third_party/pocketfft' 2024-04-03T16:03:13.2044813Z Entering 'third_party/protobuf' 2024-04-03T16:03:13.2082643Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T16:03:13.2115990Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T16:03:13.2151432Z Entering 'third_party/psimd' 2024-04-03T16:03:13.2186637Z Entering 'third_party/pthreadpool' 2024-04-03T16:03:13.2221680Z Entering 'third_party/pybind11' 2024-04-03T16:03:13.2256381Z Entering 'third_party/python-peachpy' 2024-04-03T16:03:13.2291467Z Entering 'third_party/sleef' 2024-04-03T16:03:13.2326351Z Entering 'third_party/tbb' 2024-04-03T16:03:13.2363141Z Entering 'third_party/tensorpipe' 2024-04-03T16:03:13.2397323Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T16:03:13.2429656Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T16:03:13.2462797Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T16:03:13.2495824Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T16:03:13.2529650Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T16:03:13.2565469Z Entering 'third_party/zstd' 2024-04-03T16:03:13.2610247Z ##[endgroup] 2024-04-03T16:03:13.2646907Z [command]/usr/bin/git log -1 --format='%H' 2024-04-03T16:03:13.2670833Z 'd253bd7befc64137de913dbb8727b511f1052411' 2024-04-03T16:03:13.2847271Z Prepare all required actions 2024-04-03T16:03:13.2847827Z Getting action download info 2024-04-03T16:03:13.4606473Z ##[group]Run ./.github/actions/setup-linux 2024-04-03T16:03:13.4606927Z env: 2024-04-03T16:03:13.4607214Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:13.4607584Z ##[endgroup] 2024-04-03T16:03:13.4672007Z ##[group]Run set -euo pipefail 2024-04-03T16:03:13.4672472Z set -euo pipefail 2024-04-03T16:03:13.4672895Z function get_ec2_metadata() { 2024-04-03T16:03:13.4673509Z  # Pulled from instance metadata endpoint for EC2 2024-04-03T16:03:13.4674436Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-04-03T16:03:13.4675283Z  category=$1 2024-04-03T16:03:13.4676122Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-04-03T16:03:13.4676799Z  runner_name_str=i-07486a5f9ad15937b 2024-04-03T16:03:13.4677388Z  if [[ $runner_name_str != *"gcp"* ]]; then 2024-04-03T16:03:13.4678068Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-04-03T16:03:13.4678683Z  else 2024-04-03T16:03:13.4679233Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-04-03T16:03:13.4679861Z  fi 2024-04-03T16:03:13.4680175Z } 2024-04-03T16:03:13.4680559Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-04-03T16:03:13.4681168Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-04-03T16:03:13.4681862Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-04-03T16:03:13.4682471Z echo "system info $(uname -a)" 2024-04-03T16:03:13.4690749Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:13.4691272Z env: 2024-04-03T16:03:13.4691581Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:13.4691958Z ##[endgroup] 2024-04-03T16:03:13.4774001Z ami-id: ami-0ce0c36d7a00b20e2 2024-04-03T16:03:13.4828543Z instance-id: i-07486a5f9ad15937b 2024-04-03T16:03:13.4881924Z instance-type: c5.2xlarge 2024-04-03T16:03:13.4888895Z system info Linux ip-10-0-9-184.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-04-03T16:03:13.4913236Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-04-03T16:03:13.4914200Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-04-03T16:03:13.4922647Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:13.4923211Z env: 2024-04-03T16:03:13.4923512Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:13.4923873Z ##[endgroup] 2024-04-03T16:03:13.4995039Z ##[group]Run if systemctl is-active --quiet docker; then 2024-04-03T16:03:13.4995827Z if systemctl is-active --quiet docker; then 2024-04-03T16:03:13.4996402Z  echo "Docker daemon is running..."; 2024-04-03T16:03:13.4996896Z else 2024-04-03T16:03:13.4997633Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-04-03T16:03:13.4998261Z fi 2024-04-03T16:03:13.5006074Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:13.5006603Z env: 2024-04-03T16:03:13.5006890Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:13.5007264Z ##[endgroup] 2024-04-03T16:03:13.5045961Z Docker daemon is running... 2024-04-03T16:03:13.5092035Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-04-03T16:03:13.5092578Z with: 2024-04-03T16:03:13.5092820Z shell: bash 2024-04-03T16:03:13.5093255Z timeout_minutes: 5 2024-04-03T16:03:13.5093541Z max_attempts: 3 2024-04-03T16:03:13.5093832Z retry_wait_seconds: 30 2024-04-03T16:03:13.5095372Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-04-03T16:03:13.5096883Z polling_interval_seconds: 1 2024-04-03T16:03:13.5097221Z warning_on_retry: true 2024-04-03T16:03:13.5097554Z continue_on_error: false 2024-04-03T16:03:13.5097867Z env: 2024-04-03T16:03:13.5098101Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:13.5098437Z AWS_RETRY_MODE: standard 2024-04-03T16:03:13.5098766Z AWS_MAX_ATTEMPTS: 5 2024-04-03T16:03:13.5099068Z AWS_DEFAULT_REGION: us-east-1 2024-04-03T16:03:13.5099413Z ##[endgroup] 2024-04-03T16:03:14.3598980Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-04-03T16:03:14.3600329Z Configure a credential helper to remove this warning. See 2024-04-03T16:03:14.3601173Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-04-03T16:03:14.3601725Z 2024-04-03T16:03:14.3601827Z Login Succeeded 2024-04-03T16:03:14.5615972Z Command completed after 1 attempt(s). 2024-04-03T16:03:14.5661555Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-04-03T16:03:14.5662313Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-04-03T16:03:14.5663011Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-04-03T16:03:14.5670949Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:14.5671490Z env: 2024-04-03T16:03:14.5671786Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:14.5672160Z ##[endgroup] 2024-04-03T16:03:14.5736594Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T16:03:14.5737396Z # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T16:03:14.5738033Z # shellcheck disable=SC2046 2024-04-03T16:03:14.5738519Z docker stop $(docker ps -q) || true 2024-04-03T16:03:14.5739016Z # Prune all of the docker images 2024-04-03T16:03:14.5739495Z docker system prune -af 2024-04-03T16:03:14.5746954Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:14.5747508Z env: 2024-04-03T16:03:14.5747800Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:14.5748169Z ##[endgroup] 2024-04-03T16:03:14.6063167Z "docker stop" requires at least 1 argument. 2024-04-03T16:03:14.6064065Z See 'docker stop --help'. 2024-04-03T16:03:14.6064317Z 2024-04-03T16:03:14.6064568Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-04-03T16:03:14.6064959Z 2024-04-03T16:03:14.6065116Z Stop one or more running containers 2024-04-03T16:03:14.6247527Z Total reclaimed space: 0B 2024-04-03T16:03:14.6280582Z ##[group]Run set +e 2024-04-03T16:03:14.6280919Z set +e 2024-04-03T16:03:14.6281204Z set -x 2024-04-03T16:03:14.6281475Z  2024-04-03T16:03:14.6281773Z PT_DOMAIN=download.pytorch.org 2024-04-03T16:03:14.6282626Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-04-03T16:03:14.6283731Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-04-03T16:03:14.6284478Z # one is returned at random 2024-04-03T16:03:14.6285001Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-04-03T16:03:14.6285481Z  2024-04-03T16:03:14.6285772Z if [ -z "${RESOLVED_IP}" ]; then 2024-04-03T16:03:14.6286372Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-04-03T16:03:14.6287102Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-04-03T16:03:14.6287641Z  2024-04-03T16:03:14.6287943Z  if [ -z "${RESOLVED_IP}" ]; then 2024-04-03T16:03:14.6288582Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-04-03T16:03:14.6289080Z  exit 1 2024-04-03T16:03:14.6289375Z  fi 2024-04-03T16:03:14.6289624Z fi 2024-04-03T16:03:14.6289875Z  2024-04-03T16:03:14.6290208Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-04-03T16:03:14.6290789Z  # Clean up any old records first 2024-04-03T16:03:14.6291278Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-04-03T16:03:14.6291711Z fi 2024-04-03T16:03:14.6291946Z  2024-04-03T16:03:14.6292349Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-04-03T16:03:14.6292885Z cat /etc/hosts 2024-04-03T16:03:14.6300659Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:14.6301130Z env: 2024-04-03T16:03:14.6301389Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:14.6301713Z ##[endgroup] 2024-04-03T16:03:14.6322740Z + PT_DOMAIN=download.pytorch.org 2024-04-03T16:03:14.6327603Z ++ dig -4 +short download.pytorch.org 2024-04-03T16:03:14.6328266Z ++ tail -n1 2024-04-03T16:03:14.6591974Z + RESOLVED_IP=18.160.41.124 2024-04-03T16:03:14.6592548Z + '[' -z 18.160.41.124 ']' 2024-04-03T16:03:14.6592972Z + grep -r download.pytorch.org /etc/hosts 2024-04-03T16:03:14.6599552Z 3.162.125.66 download.pytorch.org 2024-04-03T16:03:14.6600392Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-04-03T16:03:14.6708238Z + echo '18.160.41.124 download.pytorch.org' 2024-04-03T16:03:14.6709137Z + sudo tee -a /etc/hosts 2024-04-03T16:03:14.7157085Z 18.160.41.124 download.pytorch.org 2024-04-03T16:03:14.7170486Z + cat /etc/hosts 2024-04-03T16:03:14.7176984Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-04-03T16:03:14.7189625Z ::1 localhost6 localhost6.localdomain6 2024-04-03T16:03:14.7190126Z 18.160.41.124 download.pytorch.org 2024-04-03T16:03:14.7345356Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-04-03T16:03:14.7345967Z with: 2024-04-03T16:03:14.7346825Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:14.7347848Z docker-build-dir: .ci/docker 2024-04-03T16:03:14.7348210Z working-directory: . 2024-04-03T16:03:14.7348653Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:14.7349170Z force-push: false 2024-04-03T16:03:14.7349454Z env: 2024-04-03T16:03:14.7349693Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:14.7350016Z ##[endgroup] 2024-04-03T16:03:14.7369272Z ##[group]Run set -ex 2024-04-03T16:03:14.7369625Z set -ex 2024-04-03T16:03:14.7369886Z  2024-04-03T16:03:14.7370428Z # If the docker build directory or the build script doesn't exist, the action will 2024-04-03T16:03:14.7371425Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-04-03T16:03:14.7372210Z # job could then download the pre-built image as usual 2024-04-03T16:03:14.7372950Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-04-03T16:03:14.7373610Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-04-03T16:03:14.7374227Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:03:14.7374755Z  2024-04-03T16:03:14.7375243Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-04-03T16:03:14.7375853Z  exit 0 2024-04-03T16:03:14.7376115Z else 2024-04-03T16:03:14.7376454Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-04-03T16:03:14.7377038Z fi 2024-04-03T16:03:14.7377273Z  2024-04-03T16:03:14.7377723Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-04-03T16:03:14.7378577Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-04-03T16:03:14.7379480Z  # use it as it is, but first let's extract the tag 2024-04-03T16:03:14.7380171Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-04-03T16:03:14.7380881Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:03:14.7381568Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:03:14.7382100Z else 2024-04-03T16:03:14.7382512Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-04-03T16:03:14.7383162Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:03:14.7384036Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:03:14.7384785Z fi 2024-04-03T16:03:14.7392193Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:14.7392674Z env: 2024-04-03T16:03:14.7392914Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:14.7393252Z REPO_NAME: pytorch 2024-04-03T16:03:14.7394155Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:14.7395135Z DOCKER_BUILD_DIR: .ci/docker 2024-04-03T16:03:14.7395786Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:14.7396288Z ##[endgroup] 2024-04-03T16:03:14.7418288Z + [[ ! -d .ci/docker ]] 2024-04-03T16:03:14.7418676Z + [[ ! -f .ci/docker/build.sh ]] 2024-04-03T16:03:14.7419038Z + echo skip=false 2024-04-03T16:03:14.7420727Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-04-03T16:03:14.7424865Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:14.7426204Z ++ awk -F '[:,]' '{print $2}' 2024-04-03T16:03:14.7434409Z + DOCKER_TAG=16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:14.7435103Z + echo docker-tag=16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:14.7436653Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:14.7463013Z ##[group]Run set +e 2024-04-03T16:03:14.7463353Z set +e 2024-04-03T16:03:14.7463633Z set -x 2024-04-03T16:03:14.7463907Z  2024-04-03T16:03:14.7464147Z login() { 2024-04-03T16:03:14.7464794Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-04-03T16:03:14.7465510Z } 2024-04-03T16:03:14.7465748Z  2024-04-03T16:03:14.7466001Z retry () { 2024-04-03T16:03:14.7466371Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-04-03T16:03:14.7466784Z } 2024-04-03T16:03:14.7467046Z  2024-04-03T16:03:14.7467337Z retry login "${DOCKER_REGISTRY}" 2024-04-03T16:03:14.7467719Z  2024-04-03T16:03:14.7468165Z # Check if image already exists, if it does then skip building it 2024-04-03T16:03:14.7468859Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-04-03T16:03:14.7469345Z  exit 0 2024-04-03T16:03:14.7469609Z fi 2024-04-03T16:03:14.7469860Z  2024-04-03T16:03:14.7470339Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-04-03T16:03:14.7471146Z # be empty. The default action would be to continue rebuild the image 2024-04-03T16:03:14.7471869Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-04-03T16:03:14.7472526Z  # if we're on the base branch then use the parent commit 2024-04-03T16:03:14.7473089Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-04-03T16:03:14.7473498Z else 2024-04-03T16:03:14.7474066Z  # otherwise we're on a PR, so use the most recent base commit 2024-04-03T16:03:14.7474729Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-04-03T16:03:14.7475219Z fi 2024-04-03T16:03:14.7475646Z  2024-04-03T16:03:14.7476054Z if [[ -z "${MERGE_BASE}" ]]; then 2024-04-03T16:03:14.7476584Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-04-03T16:03:14.7477027Z  2024-04-03T16:03:14.7477671Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-04-03T16:03:14.7478427Z  exit 0 2024-04-03T16:03:14.7478703Z fi 2024-04-03T16:03:14.7478954Z  2024-04-03T16:03:14.7479353Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-04-03T16:03:14.7480332Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-04-03T16:03:14.7481143Z  exit 1 2024-04-03T16:03:14.7481405Z fi 2024-04-03T16:03:14.7481657Z  2024-04-03T16:03:14.7482129Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-04-03T16:03:14.7483147Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-04-03T16:03:14.7484011Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-04-03T16:03:14.7484980Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-04-03T16:03:14.7486080Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-04-03T16:03:14.7486706Z fi 2024-04-03T16:03:14.7486960Z  2024-04-03T16:03:14.7487288Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-04-03T16:03:14.7494684Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:14.7495162Z env: 2024-04-03T16:03:14.7495420Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:14.7495762Z DOCKER_BUILD_DIR: .ci/docker 2024-04-03T16:03:14.7496187Z BASE_REVISION: 1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:03:14.7497234Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:14.7498259Z DOCKER_TAG: 16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:14.7498826Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:14.7499327Z ##[endgroup] 2024-04-03T16:03:14.7519737Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:14.7520373Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:14.7522240Z + aws ecr get-login-password --region us-east-1 2024-04-03T16:03:14.7523410Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:15.1611026Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-04-03T16:03:15.1612266Z Configure a credential helper to remove this warning. See 2024-04-03T16:03:15.1613572Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-04-03T16:03:15.1614379Z 2024-04-03T16:03:15.1614558Z Login Succeeded 2024-04-03T16:03:15.1626749Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:15.3699428Z { 2024-04-03T16:03:15.3699994Z "schemaVersion": 2, 2024-04-03T16:03:15.3700784Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-04-03T16:03:15.3701641Z "config": { 2024-04-03T16:03:15.3702291Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-04-03T16:03:15.3703105Z "size": 40203, 2024-04-03T16:03:15.3704014Z "digest": "sha256:edab37a6fecd099844f4888cdb58ab57de8cc1ffb0964933664a10a6b9fa73a8" 2024-04-03T16:03:15.3704958Z }, 2024-04-03T16:03:15.3705685Z "layers": [ 2024-04-03T16:03:15.3706115Z { 2024-04-03T16:03:15.3706807Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3707713Z "size": 30451302, 2024-04-03T16:03:15.3708620Z "digest": "sha256:23828d760c7b04df02891af556c40ca44c2dd79d6837ea6f18fac24f4108448c" 2024-04-03T16:03:15.3709631Z }, 2024-04-03T16:03:15.3710019Z { 2024-04-03T16:03:15.3716460Z + exit 0 2024-04-03T16:03:15.3717590Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3718889Z "size": 1828, 2024-04-03T16:03:15.3720113Z "digest": "sha256:4c71b2109c4fce41644c534918307f4ec9dd5d74b2d1491181d8b896339253e9" 2024-04-03T16:03:15.3721535Z }, 2024-04-03T16:03:15.3722056Z { 2024-04-03T16:03:15.3723084Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3724366Z "size": 313213453, 2024-04-03T16:03:15.3725687Z "digest": "sha256:42a766fc09097cc12b6898fbc22c4b5afc61dc6e8cdca26fda1af54cd5808163" 2024-04-03T16:03:15.3727176Z }, 2024-04-03T16:03:15.3727676Z { 2024-04-03T16:03:15.3728637Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3732628Z "size": 803, 2024-04-03T16:03:15.3733553Z "digest": "sha256:0e2f2458798f790b54c7f9276d0e1a9a69ec59f7271c51c3fcae9227ad87e7eb" 2024-04-03T16:03:15.3734603Z }, 2024-04-03T16:03:15.3734982Z { 2024-04-03T16:03:15.3735718Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3736651Z "size": 106, 2024-04-03T16:03:15.3737546Z "digest": "sha256:52475bb51e22a833df2d915ac317fb98bfd19fee4ca169046b28174a752de665" 2024-04-03T16:03:15.3738560Z }, 2024-04-03T16:03:15.3738931Z { 2024-04-03T16:03:15.3739662Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3740633Z "size": 545, 2024-04-03T16:03:15.3741854Z "digest": "sha256:928ebba6ddfb9f4b945e7b6b29221dfe7b1831becab9dc384c852b9449db7229" 2024-04-03T16:03:15.3743018Z }, 2024-04-03T16:03:15.3743440Z { 2024-04-03T16:03:15.3744176Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3745219Z "size": 1240, 2024-04-03T16:03:15.3746095Z "digest": "sha256:71bfb06f00796d9766e8078ce0d22c3504c748cdda403b229e08b9f66c90bce7" 2024-04-03T16:03:15.3747007Z }, 2024-04-03T16:03:15.3747222Z { 2024-04-03T16:03:15.3747645Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3748199Z "size": 483, 2024-04-03T16:03:15.3748875Z "digest": "sha256:56090b14b977c9226e4e1d467688ba76f54434a7f44807fcfed50926a75aa080" 2024-04-03T16:03:15.3749764Z }, 2024-04-03T16:03:15.3750114Z { 2024-04-03T16:03:15.3750752Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3751602Z "size": 110313371, 2024-04-03T16:03:15.3752260Z "digest": "sha256:22343a243e5681e355fd8661325b3eedb736d5da7e950ccea5d37c9ee3bb3a52" 2024-04-03T16:03:15.3753270Z }, 2024-04-03T16:03:15.3753500Z { 2024-04-03T16:03:15.3754115Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3755001Z "size": 3706, 2024-04-03T16:03:15.3756078Z "digest": "sha256:c29bb23a799916209b3b5f78924aad9644913961489623775658d5e3786c1e03" 2024-04-03T16:03:15.3757276Z }, 2024-04-03T16:03:15.3757578Z { 2024-04-03T16:03:15.3757998Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3758547Z "size": 1904, 2024-04-03T16:03:15.3759067Z "digest": "sha256:84aa8c2bf621b344af8103f37e465c7c28753ecee0bb9418eeefb32f112a07b0" 2024-04-03T16:03:15.3759687Z }, 2024-04-03T16:03:15.3759912Z { 2024-04-03T16:03:15.3760312Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3760858Z "size": 700, 2024-04-03T16:03:15.3761385Z "digest": "sha256:869f7e29fe434c4e7c345a34b7dda28235a1e275755824a3b2faf073a4f64cdf" 2024-04-03T16:03:15.3762003Z }, 2024-04-03T16:03:15.3762214Z { 2024-04-03T16:03:15.3762704Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3763414Z "size": 2590646918, 2024-04-03T16:03:15.3763955Z "digest": "sha256:a35b44e43b45a2801de628c066941679429d266b5d0bcd155ade98198da98461" 2024-04-03T16:03:15.3764550Z }, 2024-04-03T16:03:15.3764759Z { 2024-04-03T16:03:15.3765181Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3765732Z "size": 32, 2024-04-03T16:03:15.3766264Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:15.3766869Z }, 2024-04-03T16:03:15.3767092Z { 2024-04-03T16:03:15.3767495Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3768042Z "size": 380, 2024-04-03T16:03:15.3768578Z "digest": "sha256:f7cdcec5e6209d6102c3247b2bddf556339da28d039352be239d287af12c3da7" 2024-04-03T16:03:15.3769202Z }, 2024-04-03T16:03:15.3769419Z { 2024-04-03T16:03:15.3769838Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3770387Z "size": 220395, 2024-04-03T16:03:15.3770921Z "digest": "sha256:76fd1df065af26b8c5cbc6b570fa6b05472bc0639b5a8546ac1a70fddba60ad1" 2024-04-03T16:03:15.3771542Z }, 2024-04-03T16:03:15.3771769Z { 2024-04-03T16:03:15.3772168Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3772715Z "size": 230, 2024-04-03T16:03:15.3773228Z "digest": "sha256:83315c4c7ef0a0b8556648bfcb8072517770229c65bd5374b453d5741fcd9e4f" 2024-04-03T16:03:15.3773816Z }, 2024-04-03T16:03:15.3774037Z { 2024-04-03T16:03:15.3774451Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3774980Z "size": 2691615, 2024-04-03T16:03:15.3775511Z "digest": "sha256:6460360cb1298811ed09c68381447bbab44626003d5320a65264e700538b0b0d" 2024-04-03T16:03:15.3776104Z }, 2024-04-03T16:03:15.3776313Z { 2024-04-03T16:03:15.3776836Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3777385Z "size": 1994, 2024-04-03T16:03:15.3777903Z "digest": "sha256:f5187f0f8da243ed7751f5eefa2cec05f1393803726eeee84e330057805ca678" 2024-04-03T16:03:15.3778517Z }, 2024-04-03T16:03:15.3778737Z { 2024-04-03T16:03:15.3779139Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3779685Z "size": 105, 2024-04-03T16:03:15.3780208Z "digest": "sha256:b1fe483228e92afe41514cbd9402f338907bd8f46304587c793f7d1606bb1a39" 2024-04-03T16:03:15.3780801Z }, 2024-04-03T16:03:15.3781024Z { 2024-04-03T16:03:15.3781441Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3781972Z "size": 563, 2024-04-03T16:03:15.3782500Z "digest": "sha256:f148109b25c63db922ba0015afc51f9b12bd576ad392435463f6c2ee325f24de" 2024-04-03T16:03:15.3783109Z }, 2024-04-03T16:03:15.3783322Z { 2024-04-03T16:03:15.3783744Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3784287Z "size": 32, 2024-04-03T16:03:15.3784806Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:15.3785431Z }, 2024-04-03T16:03:15.3785655Z { 2024-04-03T16:03:15.3786057Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3786607Z "size": 104, 2024-04-03T16:03:15.3787132Z "digest": "sha256:6e94329e7d9bf7082fc26cec2348784557d985302de815ad85bf204ae9fe67e3" 2024-04-03T16:03:15.3787724Z }, 2024-04-03T16:03:15.3787954Z { 2024-04-03T16:03:15.3788493Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3789022Z "size": 503, 2024-04-03T16:03:15.3789539Z "digest": "sha256:e418f79cac9b43921f1702331a3d93926e2e664608005326054dabcdae1ad4d4" 2024-04-03T16:03:15.3790136Z }, 2024-04-03T16:03:15.3790349Z { 2024-04-03T16:03:15.3790761Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3791313Z "size": 101109813, 2024-04-03T16:03:15.3791830Z "digest": "sha256:77938992fe7d6f9a688194fb8672562ebc57c33a75795b9e880e215974705c68" 2024-04-03T16:03:15.3792522Z }, 2024-04-03T16:03:15.3792743Z { 2024-04-03T16:03:15.3793144Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3793687Z "size": 109, 2024-04-03T16:03:15.3794224Z "digest": "sha256:dd295d40e381bdd36f41fe90b77efba1a050acf7f813197058bb3b925e774bd1" 2024-04-03T16:03:15.3794831Z }, 2024-04-03T16:03:15.3812704Z { 2024-04-03T16:03:15.3813302Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3814179Z "size": 557, 2024-04-03T16:03:15.3815058Z "digest": "sha256:6c196b06c1db85ebaab271581b67c41bf5f6beeab678f989ac6366ff5ee37649" 2024-04-03T16:03:15.3816150Z }, 2024-04-03T16:03:15.3816528Z { 2024-04-03T16:03:15.3817252Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3818151Z "size": 1139068, 2024-04-03T16:03:15.3819039Z "digest": "sha256:d2a4f8c972bba57f9ea2255f6631e9054cea5efdbf91f82a56ab7b897fa5c1d1" 2024-04-03T16:03:15.3819808Z }, 2024-04-03T16:03:15.3820034Z { 2024-04-03T16:03:15.3820450Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3820985Z "size": 103, 2024-04-03T16:03:15.3821500Z "digest": "sha256:e17fa7c51474c7c233f32368ce4a8a33e8ac0992e26c2d76479767026b44ceb3" 2024-04-03T16:03:15.3822104Z }, 2024-04-03T16:03:15.3822311Z { 2024-04-03T16:03:15.3822722Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3823264Z "size": 1489, 2024-04-03T16:03:15.3823787Z "digest": "sha256:cff02b807961d4ba1cd0afa55af83f4c5eecd17857db2049d7b8163477197a13" 2024-04-03T16:03:15.3824407Z }, 2024-04-03T16:03:15.3824626Z { 2024-04-03T16:03:15.3825022Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3825564Z "size": 461458425, 2024-04-03T16:03:15.3826270Z "digest": "sha256:e96da33903b7fc6a3bd99919e3008604c7b7912425f0821837508056cc42ac40" 2024-04-03T16:03:15.3826854Z }, 2024-04-03T16:03:15.3827078Z { 2024-04-03T16:03:15.3827490Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3828022Z "size": 160, 2024-04-03T16:03:15.3828541Z "digest": "sha256:42bd346d5d1c5a410024ee9ecf597a86fc80f47363b6170c7008a5bfde7a1935" 2024-04-03T16:03:15.3829154Z }, 2024-04-03T16:03:15.3829360Z { 2024-04-03T16:03:15.3829772Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3830314Z "size": 2210, 2024-04-03T16:03:15.3830831Z "digest": "sha256:f688bbce9791905b8e853b07768c0b16b689aff2fb5adf40b76836a55e856da8" 2024-04-03T16:03:15.3831449Z }, 2024-04-03T16:03:15.3831669Z { 2024-04-03T16:03:15.3832070Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3832614Z "size": 151, 2024-04-03T16:03:15.3833139Z "digest": "sha256:8fbc77b9694b8d91b1f39276723dece503921f6578be17b0b2dfaf43e04f316b" 2024-04-03T16:03:15.3833738Z }, 2024-04-03T16:03:15.3833957Z { 2024-04-03T16:03:15.3834375Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3834905Z "size": 648, 2024-04-03T16:03:15.3835591Z "digest": "sha256:e2a3bc47943f6d4122b5c4adbbfff57e0d9d734ba8fce407089307ecd51be02b" 2024-04-03T16:03:15.3836231Z }, 2024-04-03T16:03:15.3836433Z { 2024-04-03T16:03:15.3836841Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3837379Z "size": 32, 2024-04-03T16:03:15.3837892Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:15.3838508Z }, 2024-04-03T16:03:15.3838728Z { 2024-04-03T16:03:15.3839124Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3839670Z "size": 163, 2024-04-03T16:03:15.3840195Z "digest": "sha256:8237da8eed865d197e234faec5be6043bd60f07d7409b664bb649ed2cab44fef" 2024-04-03T16:03:15.3840813Z }, 2024-04-03T16:03:15.3841019Z { 2024-04-03T16:03:15.3841428Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3842065Z "size": 109, 2024-04-03T16:03:15.3842680Z "digest": "sha256:71bb79f7fa9caaf8e330a2d1f846acae32a8c4be0e87e1a7e1aab1c01153ddcd" 2024-04-03T16:03:15.3843310Z }, 2024-04-03T16:03:15.3843516Z { 2024-04-03T16:03:15.3843928Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3844471Z "size": 102, 2024-04-03T16:03:15.3844985Z "digest": "sha256:9688271757d9d394c509517c6cb13ec1cfcd741bd1ad31529b69aff129a3f553" 2024-04-03T16:03:15.3845568Z }, 2024-04-03T16:03:15.3845785Z { 2024-04-03T16:03:15.3846180Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3846718Z "size": 421, 2024-04-03T16:03:15.3847253Z "digest": "sha256:8fba31fce28cff8c2ff69af296d90a4bf77fa61d79ed329e345544d12a70f9f0" 2024-04-03T16:03:15.3847881Z }, 2024-04-03T16:03:15.3848089Z { 2024-04-03T16:03:15.3848501Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3849045Z "size": 32, 2024-04-03T16:03:15.3849555Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:15.3850171Z }, 2024-04-03T16:03:15.3850389Z { 2024-04-03T16:03:15.3850789Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3851330Z "size": 111, 2024-04-03T16:03:15.3851851Z "digest": "sha256:d2f8ea8e2cd5a274189517f6e9212602e45c45b6ad24d3e6efca36b641c3c711" 2024-04-03T16:03:15.3852441Z }, 2024-04-03T16:03:15.3852659Z { 2024-04-03T16:03:15.3853070Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3853597Z "size": 474, 2024-04-03T16:03:15.3854249Z "digest": "sha256:a337c4be3a83b6eb74f2d656fc330bf670263c5a4d580727840df8478a761897" 2024-04-03T16:03:15.3854857Z }, 2024-04-03T16:03:15.3855062Z { 2024-04-03T16:03:15.3855583Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3856131Z "size": 32, 2024-04-03T16:03:15.3856648Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:15.3857263Z }, 2024-04-03T16:03:15.3857484Z { 2024-04-03T16:03:15.3857879Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3858417Z "size": 112, 2024-04-03T16:03:15.3858938Z "digest": "sha256:e2be7086a133e82cb0b446c958db2bb6279d7c81594e240dd90bd96ca3f8d337" 2024-04-03T16:03:15.3859534Z }, 2024-04-03T16:03:15.3859755Z { 2024-04-03T16:03:15.3860163Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3860693Z "size": 564, 2024-04-03T16:03:15.3861224Z "digest": "sha256:e8b6b86420ea9bb0eb2bf0e269ae6323b5c83f3b9236e97dfb5dadfda69a8160" 2024-04-03T16:03:15.3861840Z }, 2024-04-03T16:03:15.3862046Z { 2024-04-03T16:03:15.3862454Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3863001Z "size": 43166471, 2024-04-03T16:03:15.3863509Z "digest": "sha256:4673681701f97c1326775e836dfad15804d325f29fa24d2144d0d32deb5bd456" 2024-04-03T16:03:15.3864105Z }, 2024-04-03T16:03:15.3864324Z { 2024-04-03T16:03:15.3864727Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3865270Z "size": 106, 2024-04-03T16:03:15.3865795Z "digest": "sha256:402352c3f6525cefd02a2ffb8975241b7f1ea3fef56fed20d1fb3ee2b562fa2b" 2024-04-03T16:03:15.3866393Z }, 2024-04-03T16:03:15.3866615Z { 2024-04-03T16:03:15.3867021Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3867554Z "size": 295, 2024-04-03T16:03:15.3868063Z "digest": "sha256:d6271b2f16b74d1b43a4b796fb74885f0065b4045389a65007b5e5ca6517fa87" 2024-04-03T16:03:15.3868659Z }, 2024-04-03T16:03:15.3868865Z { 2024-04-03T16:03:15.3869278Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3869858Z "size": 32, 2024-04-03T16:03:15.3870373Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:15.3871074Z }, 2024-04-03T16:03:15.3871297Z { 2024-04-03T16:03:15.3871695Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3872238Z "size": 106, 2024-04-03T16:03:15.3872762Z "digest": "sha256:0a60eb3db52708e5fe6dc16a141ef3fec47c615c96b1311298120b3e2fa9ead6" 2024-04-03T16:03:15.3873358Z }, 2024-04-03T16:03:15.3873578Z { 2024-04-03T16:03:15.3873986Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3874512Z "size": 425, 2024-04-03T16:03:15.3875025Z "digest": "sha256:5b9f4e04399989649963f72ecbcd3257433a55da6db030f52b4d1d1f3bffc1b4" 2024-04-03T16:03:15.3875791Z }, 2024-04-03T16:03:15.3876003Z { 2024-04-03T16:03:15.3876417Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3876962Z "size": 19308750, 2024-04-03T16:03:15.3877498Z "digest": "sha256:3316bc233a5488f32dcd31964f191c1844dade9b6dc226b8da9db9057c5f9f63" 2024-04-03T16:03:15.3878108Z }, 2024-04-03T16:03:15.3878331Z { 2024-04-03T16:03:15.3878730Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3879277Z "size": 108, 2024-04-03T16:03:15.3879819Z "digest": "sha256:ee5d0a9a6a1eedcd98bf22747c17bdea3ca3e6e8979be9f6f0d6dca8655d3d39" 2024-04-03T16:03:15.3880438Z }, 2024-04-03T16:03:15.3880659Z { 2024-04-03T16:03:15.3881070Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3881601Z "size": 438, 2024-04-03T16:03:15.3882122Z "digest": "sha256:7396a8060fceee68a7f0d96ab5f17e2466f05430b1c4fab23241b780248b0e2e" 2024-04-03T16:03:15.3882821Z }, 2024-04-03T16:03:15.3883029Z { 2024-04-03T16:03:15.3883441Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3883982Z "size": 700, 2024-04-03T16:03:15.3884608Z "digest": "sha256:869f7e29fe434c4e7c345a34b7dda28235a1e275755824a3b2faf073a4f64cdf" 2024-04-03T16:03:15.3885215Z }, 2024-04-03T16:03:15.3885435Z { 2024-04-03T16:03:15.3885839Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3886376Z "size": 141, 2024-04-03T16:03:15.3886903Z "digest": "sha256:8f2884fd6deca001da0a08315864318acd90ef8f9bbd2f79b821b2e121ac04b4" 2024-04-03T16:03:15.3887505Z }, 2024-04-03T16:03:15.3887725Z { 2024-04-03T16:03:15.3888135Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3888667Z "size": 136, 2024-04-03T16:03:15.3889212Z "digest": "sha256:9b6932bd4bc842c9bafcdad19b3a9a796c16cbdad4d20caaf47002b8b74f148a" 2024-04-03T16:03:15.3889836Z }, 2024-04-03T16:03:15.3890044Z { 2024-04-03T16:03:15.3890461Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3891002Z "size": 32, 2024-04-03T16:03:15.3891518Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:15.3892136Z }, 2024-04-03T16:03:15.3892354Z { 2024-04-03T16:03:15.3892751Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3893297Z "size": 189, 2024-04-03T16:03:15.3893819Z "digest": "sha256:7066d7f0b725d30bf85e388a4e99fb73655e6ba43b948cf1be6aed407249e357" 2024-04-03T16:03:15.3894412Z }, 2024-04-03T16:03:15.3894632Z { 2024-04-03T16:03:15.3895044Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3895571Z "size": 1193, 2024-04-03T16:03:15.3896088Z "digest": "sha256:8a03304f5dc79086302ddbe9a18669358304a92ed8ade623494e72eeca414bd9" 2024-04-03T16:03:15.3896689Z }, 2024-04-03T16:03:15.3896896Z { 2024-04-03T16:03:15.3897305Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3897847Z "size": 700, 2024-04-03T16:03:15.3898349Z "digest": "sha256:869f7e29fe434c4e7c345a34b7dda28235a1e275755824a3b2faf073a4f64cdf" 2024-04-03T16:03:15.3898951Z }, 2024-04-03T16:03:15.3899176Z { 2024-04-03T16:03:15.3899576Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3900241Z "size": 137, 2024-04-03T16:03:15.3900766Z "digest": "sha256:63f01b7de7c9c0753c106e467eeac8c023357ae6dc402ca14015ef5d99de3da5" 2024-04-03T16:03:15.3901361Z }, 2024-04-03T16:03:15.3901581Z { 2024-04-03T16:03:15.3901991Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3902527Z "size": 1645357786, 2024-04-03T16:03:15.3903065Z "digest": "sha256:9c7bc24d85e16389f450276d031b09c515961f9f71baf9838a32ba99c64571ab" 2024-04-03T16:03:15.3903669Z }, 2024-04-03T16:03:15.3903879Z { 2024-04-03T16:03:15.3904290Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3904834Z "size": 154, 2024-04-03T16:03:15.3905345Z "digest": "sha256:0829c9fd377b7a48911dd53ddc38bf4a29c0d94c592184923f4195efa47a487f" 2024-04-03T16:03:15.3905945Z }, 2024-04-03T16:03:15.3906169Z { 2024-04-03T16:03:15.3906575Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3907121Z "size": 628, 2024-04-03T16:03:15.3907635Z "digest": "sha256:c0d204447609e55f6d096e704a166a553d3d0b9f6a2f2b1369389ea7856db263" 2024-04-03T16:03:15.3908217Z }, 2024-04-03T16:03:15.3908437Z { 2024-04-03T16:03:15.3908848Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3909377Z "size": 700, 2024-04-03T16:03:15.3909892Z "digest": "sha256:869f7e29fe434c4e7c345a34b7dda28235a1e275755824a3b2faf073a4f64cdf" 2024-04-03T16:03:15.3910499Z }, 2024-04-03T16:03:15.3910705Z { 2024-04-03T16:03:15.3911120Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3911663Z "size": 142, 2024-04-03T16:03:15.3912153Z "digest": "sha256:779e99646108784b34aa56e42f18349cdd830012c4bd3233e0e4d929ce272f35" 2024-04-03T16:03:15.3912747Z }, 2024-04-03T16:03:15.3912970Z { 2024-04-03T16:03:15.3913447Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3913996Z "size": 32, 2024-04-03T16:03:15.3914523Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:15.3915129Z }, 2024-04-03T16:03:15.3915348Z { 2024-04-03T16:03:15.3915947Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3916478Z "size": 158, 2024-04-03T16:03:15.3917013Z "digest": "sha256:80f4cc22eacdc87b398bbd2d5c658a78fa998805084868fddebfbfb46b1e0507" 2024-04-03T16:03:15.3917649Z }, 2024-04-03T16:03:15.3917855Z { 2024-04-03T16:03:15.3918265Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3918808Z "size": 1562, 2024-04-03T16:03:15.3919350Z "digest": "sha256:e0aa2cd02b55bb1e67ef7e0bfba2f8bf629e7b8f1b8a1cd7a63bc562805e4450" 2024-04-03T16:03:15.3919983Z }, 2024-04-03T16:03:15.3920201Z { 2024-04-03T16:03:15.3920601Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3921150Z "size": 32, 2024-04-03T16:03:15.3921678Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:15.3922290Z }, 2024-04-03T16:03:15.3922516Z { 2024-04-03T16:03:15.3923013Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3923544Z "size": 128, 2024-04-03T16:03:15.3924073Z "digest": "sha256:f064f2e8f1a86f5462546997b860cdff57ddcd2a21a4b3c86b0ab413eb173e22" 2024-04-03T16:03:15.3924688Z }, 2024-04-03T16:03:15.3924895Z { 2024-04-03T16:03:15.3925310Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3925849Z "size": 380, 2024-04-03T16:03:15.3926342Z "digest": "sha256:7874003851e3610104b2deaca398908082b276deac1baed8640e611c039721c7" 2024-04-03T16:03:15.3926942Z }, 2024-04-03T16:03:15.3927163Z { 2024-04-03T16:03:15.3927562Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3928106Z "size": 32, 2024-04-03T16:03:15.3928634Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:15.3929361Z }, 2024-04-03T16:03:15.3929584Z { 2024-04-03T16:03:15.3929997Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3930526Z "size": 104, 2024-04-03T16:03:15.3931045Z "digest": "sha256:57eb0af0150176410de409eebc484512635f93bd6d90e515dfe0705a6b59941e" 2024-04-03T16:03:15.3931650Z }, 2024-04-03T16:03:15.3931857Z { 2024-04-03T16:03:15.3932276Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3932827Z "size": 1840, 2024-04-03T16:03:15.3933340Z "digest": "sha256:111b525fbe5a364938574520a5fa49d6a3e3f53fcaf3cf5e415d93c9a1655b4d" 2024-04-03T16:03:15.3933950Z }, 2024-04-03T16:03:15.3934169Z { 2024-04-03T16:03:15.3934571Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3935115Z "size": 7529636, 2024-04-03T16:03:15.3935656Z "digest": "sha256:06eb833ae33bb5b9ac945f6d4588015261b4f9d6ce63809be582319c4d2ad0c6" 2024-04-03T16:03:15.3936250Z }, 2024-04-03T16:03:15.3936473Z { 2024-04-03T16:03:15.3936885Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3937410Z "size": 164, 2024-04-03T16:03:15.3937933Z "digest": "sha256:589a8ceece91b6ecad93fc270011293465962dde63d4e4380bc49901c962f636" 2024-04-03T16:03:15.3938679Z }, 2024-04-03T16:03:15.3938885Z { 2024-04-03T16:03:15.3939295Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3939838Z "size": 7938, 2024-04-03T16:03:15.3940345Z "digest": "sha256:a4814dd3f69b9429489be0ce78788fd65239f710ca52ddc34027145ea26a7545" 2024-04-03T16:03:15.3940946Z }, 2024-04-03T16:03:15.3941164Z { 2024-04-03T16:03:15.3941563Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3942102Z "size": 8068, 2024-04-03T16:03:15.3942738Z "digest": "sha256:1f18671b7fca523a587f03c066f4a857df8fceb3485bb8133be4f8ba46b113f4" 2024-04-03T16:03:15.3943350Z }, 2024-04-03T16:03:15.3943558Z { 2024-04-03T16:03:15.3943971Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3944502Z "size": 302, 2024-04-03T16:03:15.3945026Z "digest": "sha256:4002c8a87e7ae430b8882d8bb58e0c96616d7e574f1e4bffa35c05b4894e98ef" 2024-04-03T16:03:15.3945632Z }, 2024-04-03T16:03:15.3945841Z { 2024-04-03T16:03:15.3946253Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3946796Z "size": 32, 2024-04-03T16:03:15.3947327Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-03T16:03:15.3947925Z }, 2024-04-03T16:03:15.3948144Z { 2024-04-03T16:03:15.3948544Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3949085Z "size": 108, 2024-04-03T16:03:15.3949618Z "digest": "sha256:cc9527ae45fc240b262a1e61a4955e451b4680ae30c5bcb08d5bb3ea1b93c11e" 2024-04-03T16:03:15.3950230Z }, 2024-04-03T16:03:15.3950442Z { 2024-04-03T16:03:15.3950857Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-03T16:03:15.3951412Z "size": 54145478, 2024-04-03T16:03:15.3951964Z "digest": "sha256:b848d28edf10dd6dc82ce8b6a32dd20d6ff3c8662ba5932bc0d2d7e7bea131e6" 2024-04-03T16:03:15.3952591Z } 2024-04-03T16:03:15.3952816Z ] 2024-04-03T16:03:15.3953021Z } 2024-04-03T16:03:15.4059912Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-04-03T16:03:15.4060384Z tag=${ECR_DOCKER_IMAGE##*/} 2024-04-03T16:03:15.4060912Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-04-03T16:03:15.4068930Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:15.4069422Z env: 2024-04-03T16:03:15.4069680Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:15.4070608Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:15.4071589Z ##[endgroup] 2024-04-03T16:03:15.4095557Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-jammy-py3.8-gcc11-16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:15.4147627Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-04-03T16:03:15.4148183Z with: 2024-04-03T16:03:15.4149025Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:15.4150130Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:15.4150613Z env: 2024-04-03T16:03:15.4150863Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:15.4151181Z ##[endgroup] 2024-04-03T16:03:15.4168797Z ##[group]Run set -x 2024-04-03T16:03:15.4169109Z set -x 2024-04-03T16:03:15.4169382Z set +e 2024-04-03T16:03:15.4169645Z  2024-04-03T16:03:15.4169880Z login() { 2024-04-03T16:03:15.4170520Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-04-03T16:03:15.4171227Z } 2024-04-03T16:03:15.4171462Z  2024-04-03T16:03:15.4171766Z retry () { 2024-04-03T16:03:15.4172117Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-04-03T16:03:15.4172540Z } 2024-04-03T16:03:15.4172785Z  2024-04-03T16:03:15.4173056Z retry login "${DOCKER_REGISTRY}" 2024-04-03T16:03:15.4173451Z  2024-04-03T16:03:15.4173694Z set -e 2024-04-03T16:03:15.4174129Z # ignore output since only exit code is used for conditional 2024-04-03T16:03:15.4174818Z # only pull docker image if it's not available locally 2024-04-03T16:03:15.4175572Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-04-03T16:03:15.4176250Z  retry docker pull "${DOCKER_IMAGE}" 2024-04-03T16:03:15.4176652Z fi 2024-04-03T16:03:15.4183903Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:03:15.4184383Z env: 2024-04-03T16:03:15.4184637Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:03:15.4185546Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:15.4186649Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:15.4187153Z ##[endgroup] 2024-04-03T16:03:15.4207597Z + set +e 2024-04-03T16:03:15.4208307Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:15.4208936Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:15.4211645Z + aws ecr get-login-password --region us-east-1 2024-04-03T16:03:15.4212506Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-03T16:03:15.8249774Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-04-03T16:03:15.8250654Z Configure a credential helper to remove this warning. See 2024-04-03T16:03:15.8252132Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-04-03T16:03:15.8252905Z 2024-04-03T16:03:15.8253035Z Login Succeeded 2024-04-03T16:03:15.8262282Z + set -e 2024-04-03T16:03:15.8263428Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:15.8418971Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:15.8420717Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:03:16.1238351Z 16884143090f148aa56809ceba26eeb699b5dc49: Pulling from pytorch/pytorch-linux-jammy-py3.8-gcc11 2024-04-03T16:03:16.1244303Z 23828d760c7b: Pulling fs layer 2024-04-03T16:03:16.1249152Z 4c71b2109c4f: Pulling fs layer 2024-04-03T16:03:16.1250360Z 42a766fc0909: Pulling fs layer 2024-04-03T16:03:16.1251306Z 0e2f2458798f: Pulling fs layer 2024-04-03T16:03:16.1251945Z 52475bb51e22: Pulling fs layer 2024-04-03T16:03:16.1254140Z 928ebba6ddfb: Pulling fs layer 2024-04-03T16:03:16.1254828Z 71bfb06f0079: Pulling fs layer 2024-04-03T16:03:16.1255499Z 56090b14b977: Pulling fs layer 2024-04-03T16:03:16.1256123Z 22343a243e56: Pulling fs layer 2024-04-03T16:03:16.1260204Z c29bb23a7999: Pulling fs layer 2024-04-03T16:03:16.1260852Z 0e2f2458798f: Waiting 2024-04-03T16:03:16.1261441Z 84aa8c2bf621: Pulling fs layer 2024-04-03T16:03:16.1261959Z 52475bb51e22: Waiting 2024-04-03T16:03:16.1262267Z 869f7e29fe43: Pulling fs layer 2024-04-03T16:03:16.1262619Z a35b44e43b45: Pulling fs layer 2024-04-03T16:03:16.1262958Z 4f4fb700ef54: Pulling fs layer 2024-04-03T16:03:16.1263311Z f7cdcec5e620: Pulling fs layer 2024-04-03T16:03:16.1263667Z 76fd1df065af: Pulling fs layer 2024-04-03T16:03:16.1264003Z 83315c4c7ef0: Pulling fs layer 2024-04-03T16:03:16.1264351Z 6460360cb129: Pulling fs layer 2024-04-03T16:03:16.1264698Z f5187f0f8da2: Pulling fs layer 2024-04-03T16:03:16.1265033Z b1fe483228e9: Pulling fs layer 2024-04-03T16:03:16.1265384Z f148109b25c6: Pulling fs layer 2024-04-03T16:03:16.1265797Z 6e94329e7d9b: Pulling fs layer 2024-04-03T16:03:16.1266133Z e418f79cac9b: Pulling fs layer 2024-04-03T16:03:16.1266474Z 928ebba6ddfb: Waiting 2024-04-03T16:03:16.1266780Z 77938992fe7d: Pulling fs layer 2024-04-03T16:03:16.1267103Z 71bfb06f0079: Waiting 2024-04-03T16:03:16.1267410Z dd295d40e381: Pulling fs layer 2024-04-03T16:03:16.1267744Z 56090b14b977: Waiting 2024-04-03T16:03:16.1268035Z 6c196b06c1db: Pulling fs layer 2024-04-03T16:03:16.1268365Z 22343a243e56: Waiting 2024-04-03T16:03:16.1268672Z d2a4f8c972bb: Pulling fs layer 2024-04-03T16:03:16.1269003Z c29bb23a7999: Waiting 2024-04-03T16:03:16.1269307Z e17fa7c51474: Pulling fs layer 2024-04-03T16:03:16.1269657Z cff02b807961: Pulling fs layer 2024-04-03T16:03:16.1270167Z e96da33903b7: Pulling fs layer 2024-04-03T16:03:16.1270491Z 84aa8c2bf621: Waiting 2024-04-03T16:03:16.1270902Z 42bd346d5d1c: Pulling fs layer 2024-04-03T16:03:16.1271241Z 4f4fb700ef54: Waiting 2024-04-03T16:03:16.1271535Z f688bbce9791: Pulling fs layer 2024-04-03T16:03:16.1271881Z 869f7e29fe43: Waiting 2024-04-03T16:03:16.1272173Z f7cdcec5e620: Waiting 2024-04-03T16:03:16.1272473Z 8fbc77b9694b: Pulling fs layer 2024-04-03T16:03:16.1272817Z 76fd1df065af: Waiting 2024-04-03T16:03:16.1273124Z e2a3bc47943f: Pulling fs layer 2024-04-03T16:03:16.1273450Z a35b44e43b45: Waiting 2024-04-03T16:03:16.1273738Z 83315c4c7ef0: Waiting 2024-04-03T16:03:16.1274025Z f148109b25c6: Waiting 2024-04-03T16:03:16.1274319Z 8237da8eed86: Pulling fs layer 2024-04-03T16:03:16.1274652Z 6460360cb129: Waiting 2024-04-03T16:03:16.1274943Z 71bb79f7fa9c: Pulling fs layer 2024-04-03T16:03:16.1275279Z f5187f0f8da2: Waiting 2024-04-03T16:03:16.1275809Z 9688271757d9: Pulling fs layer 2024-04-03T16:03:16.1276152Z 8fba31fce28c: Pulling fs layer 2024-04-03T16:03:16.1276494Z 6e94329e7d9b: Waiting 2024-04-03T16:03:16.1276788Z cff02b807961: Waiting 2024-04-03T16:03:16.1277088Z d2f8ea8e2cd5: Pulling fs layer 2024-04-03T16:03:16.1277433Z e418f79cac9b: Waiting 2024-04-03T16:03:16.1277754Z a337c4be3a83: Pulling fs layer 2024-04-03T16:03:16.1278078Z e96da33903b7: Waiting 2024-04-03T16:03:16.1278363Z 77938992fe7d: Waiting 2024-04-03T16:03:16.1278667Z e2be7086a133: Pulling fs layer 2024-04-03T16:03:16.1279005Z e8b6b86420ea: Pulling fs layer 2024-04-03T16:03:16.1279339Z 42bd346d5d1c: Waiting 2024-04-03T16:03:16.1279633Z dd295d40e381: Waiting 2024-04-03T16:03:16.1279918Z 4673681701f9: Pulling fs layer 2024-04-03T16:03:16.1280247Z d2a4f8c972bb: Waiting 2024-04-03T16:03:16.1280539Z 8fbc77b9694b: Waiting 2024-04-03T16:03:16.1280815Z 6c196b06c1db: Waiting 2024-04-03T16:03:16.1281112Z 402352c3f652: Pulling fs layer 2024-04-03T16:03:16.1281448Z f688bbce9791: Waiting 2024-04-03T16:03:16.1281740Z d6271b2f16b7: Pulling fs layer 2024-04-03T16:03:16.1282074Z e2a3bc47943f: Waiting 2024-04-03T16:03:16.1282376Z 0a60eb3db527: Pulling fs layer 2024-04-03T16:03:16.1282770Z 8237da8eed86: Waiting 2024-04-03T16:03:16.1283062Z d2f8ea8e2cd5: Waiting 2024-04-03T16:03:16.1283369Z 5b9f4e043999: Pulling fs layer 2024-04-03T16:03:16.1284044Z 71bb79f7fa9c: Waiting 2024-04-03T16:03:16.1284339Z e2be7086a133: Waiting 2024-04-03T16:03:16.1284644Z 3316bc233a54: Pulling fs layer 2024-04-03T16:03:16.1284970Z 9688271757d9: Waiting 2024-04-03T16:03:16.1285256Z e8b6b86420ea: Waiting 2024-04-03T16:03:16.1285547Z a337c4be3a83: Waiting 2024-04-03T16:03:16.1285840Z ee5d0a9a6a1e: Pulling fs layer 2024-04-03T16:03:16.1286185Z 8fba31fce28c: Waiting 2024-04-03T16:03:16.1286490Z 7396a8060fce: Pulling fs layer 2024-04-03T16:03:16.1286811Z 402352c3f652: Waiting 2024-04-03T16:03:16.1287159Z d6271b2f16b7: Waiting 2024-04-03T16:03:16.1287450Z e17fa7c51474: Waiting 2024-04-03T16:03:16.1287844Z 4673681701f9: Waiting 2024-04-03T16:03:16.1288149Z 8f2884fd6dec: Pulling fs layer 2024-04-03T16:03:16.1288541Z 9b6932bd4bc8: Pulling fs layer 2024-04-03T16:03:16.1288869Z ee5d0a9a6a1e: Waiting 2024-04-03T16:03:16.1289178Z 7066d7f0b725: Pulling fs layer 2024-04-03T16:03:16.1289524Z 8a03304f5dc7: Pulling fs layer 2024-04-03T16:03:16.1289846Z 9b6932bd4bc8: Waiting 2024-04-03T16:03:16.1290145Z 7396a8060fce: Waiting 2024-04-03T16:03:16.1290437Z 8f2884fd6dec: Waiting 2024-04-03T16:03:16.1290728Z 63f01b7de7c9: Pulling fs layer 2024-04-03T16:03:16.1291080Z 9c7bc24d85e1: Pulling fs layer 2024-04-03T16:03:16.1291430Z 0829c9fd377b: Pulling fs layer 2024-04-03T16:03:16.1291751Z 7066d7f0b725: Waiting 2024-04-03T16:03:16.1292037Z 8a03304f5dc7: Waiting 2024-04-03T16:03:16.1292337Z c0d204447609: Pulling fs layer 2024-04-03T16:03:16.1292658Z 63f01b7de7c9: Waiting 2024-04-03T16:03:16.1292950Z 9c7bc24d85e1: Waiting 2024-04-03T16:03:16.1293255Z 779e99646108: Pulling fs layer 2024-04-03T16:03:16.1293578Z 3316bc233a54: Waiting 2024-04-03T16:03:16.1293864Z c0d204447609: Waiting 2024-04-03T16:03:16.1294155Z 80f4cc22eacd: Pulling fs layer 2024-04-03T16:03:16.1294492Z 779e99646108: Waiting 2024-04-03T16:03:16.1294793Z e0aa2cd02b55: Pulling fs layer 2024-04-03T16:03:16.1295141Z f064f2e8f1a8: Pulling fs layer 2024-04-03T16:03:16.1295488Z 7874003851e3: Pulling fs layer 2024-04-03T16:03:16.1295849Z 57eb0af01501: Pulling fs layer 2024-04-03T16:03:16.1296179Z e0aa2cd02b55: Waiting 2024-04-03T16:03:16.1296486Z 111b525fbe5a: Pulling fs layer 2024-04-03T16:03:16.1296839Z 06eb833ae33b: Pulling fs layer 2024-04-03T16:03:16.1297166Z f064f2e8f1a8: Waiting 2024-04-03T16:03:16.1297471Z 589a8ceece91: Pulling fs layer 2024-04-03T16:03:16.1297800Z 7874003851e3: Waiting 2024-04-03T16:03:16.1298090Z a4814dd3f69b: Pulling fs layer 2024-04-03T16:03:16.1298440Z 1f18671b7fca: Pulling fs layer 2024-04-03T16:03:16.1298776Z 57eb0af01501: Waiting 2024-04-03T16:03:16.1299067Z 4002c8a87e7a: Pulling fs layer 2024-04-03T16:03:16.1299420Z cc9527ae45fc: Pulling fs layer 2024-04-03T16:03:16.1299767Z b848d28edf10: Pulling fs layer 2024-04-03T16:03:16.1300086Z 111b525fbe5a: Waiting 2024-04-03T16:03:16.1300378Z 4002c8a87e7a: Waiting 2024-04-03T16:03:16.1300664Z 1f18671b7fca: Waiting 2024-04-03T16:03:16.1300942Z 06eb833ae33b: Waiting 2024-04-03T16:03:16.1301229Z b848d28edf10: Waiting 2024-04-03T16:03:16.1301522Z cc9527ae45fc: Waiting 2024-04-03T16:03:16.1301814Z 589a8ceece91: Waiting 2024-04-03T16:03:16.1302105Z a4814dd3f69b: Waiting 2024-04-03T16:03:16.1302393Z 0829c9fd377b: Waiting 2024-04-03T16:03:16.2117646Z 4c71b2109c4f: Verifying Checksum 2024-04-03T16:03:16.2118116Z 4c71b2109c4f: Download complete 2024-04-03T16:03:16.2868556Z 0e2f2458798f: Verifying Checksum 2024-04-03T16:03:16.2869168Z 0e2f2458798f: Download complete 2024-04-03T16:03:16.3604184Z 52475bb51e22: Verifying Checksum 2024-04-03T16:03:16.3604884Z 52475bb51e22: Download complete 2024-04-03T16:03:16.4367759Z 928ebba6ddfb: Download complete 2024-04-03T16:03:16.4920728Z 23828d760c7b: Download complete 2024-04-03T16:03:16.5236465Z 71bfb06f0079: Verifying Checksum 2024-04-03T16:03:16.5236945Z 71bfb06f0079: Download complete 2024-04-03T16:03:16.5644087Z 56090b14b977: Verifying Checksum 2024-04-03T16:03:16.5644585Z 56090b14b977: Download complete 2024-04-03T16:03:16.6560874Z c29bb23a7999: Verifying Checksum 2024-04-03T16:03:16.6561719Z c29bb23a7999: Download complete 2024-04-03T16:03:16.7471471Z 84aa8c2bf621: Verifying Checksum 2024-04-03T16:03:16.7472223Z 84aa8c2bf621: Download complete 2024-04-03T16:03:16.8365986Z 869f7e29fe43: Download complete 2024-04-03T16:03:17.2281531Z 23828d760c7b: Pull complete 2024-04-03T16:03:17.3500337Z 4c71b2109c4f: Pull complete 2024-04-03T16:03:17.6977896Z 22343a243e56: Verifying Checksum 2024-04-03T16:03:17.6978363Z 22343a243e56: Download complete 2024-04-03T16:03:17.7097916Z 4f4fb700ef54: Download complete 2024-04-03T16:03:17.8470307Z f7cdcec5e620: Verifying Checksum 2024-04-03T16:03:17.8470823Z f7cdcec5e620: Download complete 2024-04-03T16:03:17.9308350Z 76fd1df065af: Download complete 2024-04-03T16:03:18.0117873Z 83315c4c7ef0: Verifying Checksum 2024-04-03T16:03:18.0118467Z 83315c4c7ef0: Download complete 2024-04-03T16:03:18.1258854Z 6460360cb129: Verifying Checksum 2024-04-03T16:03:18.1259575Z 6460360cb129: Download complete 2024-04-03T16:03:18.2128301Z f5187f0f8da2: Verifying Checksum 2024-04-03T16:03:18.2128957Z f5187f0f8da2: Download complete 2024-04-03T16:03:18.2966708Z b1fe483228e9: Verifying Checksum 2024-04-03T16:03:18.2967290Z b1fe483228e9: Download complete 2024-04-03T16:03:18.3786995Z f148109b25c6: Download complete 2024-04-03T16:03:18.4653201Z 6e94329e7d9b: Download complete 2024-04-03T16:03:18.5430655Z e418f79cac9b: Download complete 2024-04-03T16:03:19.3249428Z 42a766fc0909: Verifying Checksum 2024-04-03T16:03:19.3250104Z 42a766fc0909: Download complete 2024-04-03T16:03:19.4011332Z dd295d40e381: Verifying Checksum 2024-04-03T16:03:19.4011947Z dd295d40e381: Download complete 2024-04-03T16:03:19.4819549Z 6c196b06c1db: Verifying Checksum 2024-04-03T16:03:19.4820270Z 6c196b06c1db: Download complete 2024-04-03T16:03:19.5710214Z d2a4f8c972bb: Verifying Checksum 2024-04-03T16:03:19.5710959Z d2a4f8c972bb: Download complete 2024-04-03T16:03:19.6110289Z 77938992fe7d: Verifying Checksum 2024-04-03T16:03:19.6110748Z 77938992fe7d: Download complete 2024-04-03T16:03:19.6538117Z e17fa7c51474: Download complete 2024-04-03T16:03:19.7109770Z cff02b807961: Verifying Checksum 2024-04-03T16:03:19.7110652Z cff02b807961: Download complete 2024-04-03T16:03:19.7983012Z 42bd346d5d1c: Verifying Checksum 2024-04-03T16:03:19.7983672Z 42bd346d5d1c: Download complete 2024-04-03T16:03:19.8847711Z f688bbce9791: Verifying Checksum 2024-04-03T16:03:19.8848433Z f688bbce9791: Download complete 2024-04-03T16:03:19.9754216Z 8fbc77b9694b: Verifying Checksum 2024-04-03T16:03:19.9755403Z 8fbc77b9694b: Download complete 2024-04-03T16:03:20.0604296Z e2a3bc47943f: Verifying Checksum 2024-04-03T16:03:20.0623615Z e2a3bc47943f: Download complete 2024-04-03T16:03:20.1332235Z 8237da8eed86: Download complete 2024-04-03T16:03:20.2052130Z 71bb79f7fa9c: Verifying Checksum 2024-04-03T16:03:20.2052853Z 71bb79f7fa9c: Download complete 2024-04-03T16:03:20.3129321Z 9688271757d9: Download complete 2024-04-03T16:03:20.4340257Z 8fba31fce28c: Verifying Checksum 2024-04-03T16:03:20.4341965Z 8fba31fce28c: Download complete 2024-04-03T16:03:20.5245183Z d2f8ea8e2cd5: Verifying Checksum 2024-04-03T16:03:20.5246027Z d2f8ea8e2cd5: Download complete 2024-04-03T16:03:20.6181220Z a337c4be3a83: Download complete 2024-04-03T16:03:20.7120292Z e2be7086a133: Verifying Checksum 2024-04-03T16:03:20.7120934Z e2be7086a133: Download complete 2024-04-03T16:03:20.7972967Z e8b6b86420ea: Verifying Checksum 2024-04-03T16:03:20.7973683Z e8b6b86420ea: Download complete 2024-04-03T16:03:21.2859651Z 4673681701f9: Verifying Checksum 2024-04-03T16:03:21.2862461Z 4673681701f9: Download complete 2024-04-03T16:03:21.3746951Z 402352c3f652: Verifying Checksum 2024-04-03T16:03:21.3747554Z 402352c3f652: Download complete 2024-04-03T16:03:21.4771573Z d6271b2f16b7: Download complete 2024-04-03T16:03:21.5703658Z 0a60eb3db527: Verifying Checksum 2024-04-03T16:03:21.5706678Z 0a60eb3db527: Download complete 2024-04-03T16:03:21.7124035Z 5b9f4e043999: Download complete 2024-04-03T16:03:21.9724049Z 3316bc233a54: Verifying Checksum 2024-04-03T16:03:21.9724770Z 3316bc233a54: Download complete 2024-04-03T16:03:22.0601121Z ee5d0a9a6a1e: Download complete 2024-04-03T16:03:22.1422278Z 7396a8060fce: Verifying Checksum 2024-04-03T16:03:22.1422883Z 7396a8060fce: Download complete 2024-04-03T16:03:22.2228130Z 8f2884fd6dec: Download complete 2024-04-03T16:03:22.3178808Z 9b6932bd4bc8: Verifying Checksum 2024-04-03T16:03:22.3179479Z 9b6932bd4bc8: Download complete 2024-04-03T16:03:22.4055670Z 7066d7f0b725: Verifying Checksum 2024-04-03T16:03:22.4056325Z 7066d7f0b725: Download complete 2024-04-03T16:03:22.4926529Z 8a03304f5dc7: Verifying Checksum 2024-04-03T16:03:22.5953947Z 8a03304f5dc7: Download complete 2024-04-03T16:03:22.5954603Z 63f01b7de7c9: Verifying Checksum 2024-04-03T16:03:22.5955046Z 63f01b7de7c9: Download complete 2024-04-03T16:03:24.3382163Z e96da33903b7: Verifying Checksum 2024-04-03T16:03:24.3382911Z e96da33903b7: Download complete 2024-04-03T16:03:24.4166463Z 0829c9fd377b: Verifying Checksum 2024-04-03T16:03:24.4167220Z 0829c9fd377b: Download complete 2024-04-03T16:03:24.4990715Z c0d204447609: Download complete 2024-04-03T16:03:24.6137253Z 779e99646108: Verifying Checksum 2024-04-03T16:03:24.6138297Z 779e99646108: Download complete 2024-04-03T16:03:24.7015870Z 80f4cc22eacd: Download complete 2024-04-03T16:03:24.8027981Z e0aa2cd02b55: Download complete 2024-04-03T16:03:24.8929030Z f064f2e8f1a8: Verifying Checksum 2024-04-03T16:03:24.8929784Z f064f2e8f1a8: Download complete 2024-04-03T16:03:25.0224710Z 7874003851e3: Verifying Checksum 2024-04-03T16:03:25.0225172Z 7874003851e3: Download complete 2024-04-03T16:03:25.1131529Z 57eb0af01501: Download complete 2024-04-03T16:03:25.2036320Z 111b525fbe5a: Download complete 2024-04-03T16:03:25.3473027Z 06eb833ae33b: Verifying Checksum 2024-04-03T16:03:25.3473680Z 06eb833ae33b: Download complete 2024-04-03T16:03:25.4206646Z 589a8ceece91: Verifying Checksum 2024-04-03T16:03:25.4207230Z 589a8ceece91: Download complete 2024-04-03T16:03:25.5050505Z a4814dd3f69b: Download complete 2024-04-03T16:03:25.5928912Z 1f18671b7fca: Download complete 2024-04-03T16:03:25.6148917Z 42a766fc0909: Pull complete 2024-04-03T16:03:25.6927514Z 4002c8a87e7a: Verifying Checksum 2024-04-03T16:03:25.6928303Z 4002c8a87e7a: Download complete 2024-04-03T16:03:25.7768505Z cc9527ae45fc: Verifying Checksum 2024-04-03T16:03:25.7769188Z cc9527ae45fc: Download complete 2024-04-03T16:03:25.8672878Z 0e2f2458798f: Pull complete 2024-04-03T16:03:26.0508514Z 52475bb51e22: Pull complete 2024-04-03T16:03:26.1991654Z 928ebba6ddfb: Pull complete 2024-04-03T16:03:26.3647325Z 71bfb06f0079: Pull complete 2024-04-03T16:03:26.4429460Z b848d28edf10: Verifying Checksum 2024-04-03T16:03:26.4429938Z b848d28edf10: Download complete 2024-04-03T16:03:26.5198262Z 56090b14b977: Pull complete 2024-04-03T16:03:42.0643468Z 22343a243e56: Pull complete 2024-04-03T16:03:42.4734986Z c29bb23a7999: Pull complete 2024-04-03T16:03:42.9468245Z 84aa8c2bf621: Pull complete 2024-04-03T16:03:43.4266034Z 869f7e29fe43: Pull complete 2024-04-03T16:03:48.6460895Z 9c7bc24d85e1: Verifying Checksum 2024-04-03T16:03:48.6462578Z 9c7bc24d85e1: Download complete 2024-04-03T16:03:52.9457754Z a35b44e43b45: Download complete 2024-04-03T16:04:39.5289655Z a35b44e43b45: Pull complete 2024-04-03T16:04:39.6476510Z 4f4fb700ef54: Pull complete 2024-04-03T16:04:39.7370900Z f7cdcec5e620: Pull complete 2024-04-03T16:04:39.9282969Z 76fd1df065af: Pull complete 2024-04-03T16:04:40.1570710Z 83315c4c7ef0: Pull complete 2024-04-03T16:04:40.4460543Z 6460360cb129: Pull complete 2024-04-03T16:04:40.6765803Z f5187f0f8da2: Pull complete 2024-04-03T16:04:40.8987889Z b1fe483228e9: Pull complete 2024-04-03T16:04:41.1334835Z f148109b25c6: Pull complete 2024-04-03T16:04:41.6064541Z 6e94329e7d9b: Pull complete 2024-04-03T16:04:41.8203709Z e418f79cac9b: Pull complete 2024-04-03T16:04:43.8101900Z 77938992fe7d: Pull complete 2024-04-03T16:04:43.9330885Z dd295d40e381: Pull complete 2024-04-03T16:04:44.0455430Z 6c196b06c1db: Pull complete 2024-04-03T16:04:44.1583979Z d2a4f8c972bb: Pull complete 2024-04-03T16:04:44.2503831Z e17fa7c51474: Pull complete 2024-04-03T16:04:44.3407085Z cff02b807961: Pull complete 2024-04-03T16:04:50.2292500Z e96da33903b7: Pull complete 2024-04-03T16:04:50.4297381Z 42bd346d5d1c: Pull complete 2024-04-03T16:04:50.5890437Z f688bbce9791: Pull complete 2024-04-03T16:04:50.8032646Z 8fbc77b9694b: Pull complete 2024-04-03T16:04:51.0435387Z e2a3bc47943f: Pull complete 2024-04-03T16:04:51.4594202Z 8237da8eed86: Pull complete 2024-04-03T16:04:51.7036006Z 71bb79f7fa9c: Pull complete 2024-04-03T16:04:51.9352020Z 9688271757d9: Pull complete 2024-04-03T16:04:52.1633520Z 8fba31fce28c: Pull complete 2024-04-03T16:04:52.6142825Z d2f8ea8e2cd5: Pull complete 2024-04-03T16:04:52.7646783Z a337c4be3a83: Pull complete 2024-04-03T16:04:53.1947818Z e2be7086a133: Pull complete 2024-04-03T16:04:53.4391559Z e8b6b86420ea: Pull complete 2024-04-03T16:04:54.6855599Z 4673681701f9: Pull complete 2024-04-03T16:04:54.8303259Z 402352c3f652: Pull complete 2024-04-03T16:04:55.0281419Z d6271b2f16b7: Pull complete 2024-04-03T16:04:55.4315160Z 0a60eb3db527: Pull complete 2024-04-03T16:04:55.5143635Z 5b9f4e043999: Pull complete 2024-04-03T16:04:55.7847397Z 3316bc233a54: Pull complete 2024-04-03T16:04:55.8735113Z ee5d0a9a6a1e: Pull complete 2024-04-03T16:04:55.9589022Z 7396a8060fce: Pull complete 2024-04-03T16:04:56.1283193Z 8f2884fd6dec: Pull complete 2024-04-03T16:04:56.2158080Z 9b6932bd4bc8: Pull complete 2024-04-03T16:04:56.3946192Z 7066d7f0b725: Pull complete 2024-04-03T16:04:56.4913294Z 8a03304f5dc7: Pull complete 2024-04-03T16:04:56.7074501Z 63f01b7de7c9: Pull complete 2024-04-03T16:05:23.8875681Z 9c7bc24d85e1: Pull complete 2024-04-03T16:05:24.3338859Z 0829c9fd377b: Pull complete 2024-04-03T16:05:24.6141257Z c0d204447609: Pull complete 2024-04-03T16:05:25.0727460Z 779e99646108: Pull complete 2024-04-03T16:05:25.4968134Z 80f4cc22eacd: Pull complete 2024-04-03T16:05:25.6833942Z e0aa2cd02b55: Pull complete 2024-04-03T16:05:26.1469097Z f064f2e8f1a8: Pull complete 2024-04-03T16:05:26.3708115Z 7874003851e3: Pull complete 2024-04-03T16:05:26.8212312Z 57eb0af01501: Pull complete 2024-04-03T16:05:27.0555743Z 111b525fbe5a: Pull complete 2024-04-03T16:05:27.3888434Z 06eb833ae33b: Pull complete 2024-04-03T16:05:27.5811020Z 589a8ceece91: Pull complete 2024-04-03T16:05:27.7408371Z a4814dd3f69b: Pull complete 2024-04-03T16:05:27.9219760Z 1f18671b7fca: Pull complete 2024-04-03T16:05:28.1448662Z 4002c8a87e7a: Pull complete 2024-04-03T16:05:28.6052759Z cc9527ae45fc: Pull complete 2024-04-03T16:05:30.2306520Z b848d28edf10: Pull complete 2024-04-03T16:05:30.3419214Z Digest: sha256:66934e8a2aa7cad8e06a1fbc5a6f0e239ee3d124c7929e4418f76d11beb80c7b 2024-04-03T16:05:30.3895669Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:05:30.4137841Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:05:30.4194100Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-04-03T16:05:30.4194862Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-04-03T16:05:30.4195753Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-04-03T16:05:30.4196403Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:05:30.4205348Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:05:30.4205841Z env: 2024-04-03T16:05:30.4206083Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:30.4206404Z ##[endgroup] 2024-04-03T16:05:32.8668460Z Defaulting to user installation because normal site-packages is not writeable 2024-04-03T16:05:32.9315217Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-04-03T16:05:32.9499679Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-04-03T16:05:33.1975200Z Prepare all required actions 2024-04-03T16:05:33.1975707Z Getting action download info 2024-04-03T16:05:33.3099629Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-04-03T16:05:33.4860672Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-04-03T16:05:33.6555391Z ##[group]Run ./.github/actions/download-build-artifacts 2024-04-03T16:05:33.6556203Z with: 2024-04-03T16:05:33.6556484Z name: linux-jammy-py3.8-gcc11 2024-04-03T16:05:33.6556967Z s3-bucket: gha-artifacts 2024-04-03T16:05:33.6557285Z env: 2024-04-03T16:05:33.6557533Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:33.6557857Z ##[endgroup] 2024-04-03T16:05:33.6676094Z ##[group]Run seemethere/download-artifact-s3@v4 2024-04-03T16:05:33.6676545Z with: 2024-04-03T16:05:33.6676822Z name: linux-jammy-py3.8-gcc11 2024-04-03T16:05:33.6677346Z s3-bucket: gha-artifacts 2024-04-03T16:05:33.6677728Z region: us-east-1 2024-04-03T16:05:33.6678002Z env: 2024-04-03T16:05:33.6678253Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:33.6678585Z ##[endgroup] 2024-04-03T16:05:34.1450035Z (node:14397) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-04-03T16:05:34.1450786Z 2024-04-03T16:05:34.1451036Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-04-03T16:05:34.1451848Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-04-03T16:05:34.1452990Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-04-03T16:05:34.2123514Z Found 1 objects with prefix pytorch/pytorch/8536950919/linux-jammy-py3.8-gcc11/ 2024-04-03T16:05:34.2124529Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-04-03T16:05:37.8247910Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-04-03T16:05:37.8254281Z Artifact download has finished successfully 2024-04-03T16:05:37.8441126Z ##[group]Run unzip -o artifacts.zip 2024-04-03T16:05:37.8441528Z unzip -o artifacts.zip 2024-04-03T16:05:37.8449223Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:05:37.8449720Z env: 2024-04-03T16:05:37.8449972Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:37.8450278Z ##[endgroup] 2024-04-03T16:05:37.8899210Z Archive: artifacts.zip 2024-04-03T16:05:37.8900085Z creating: dist/ 2024-04-03T16:05:38.7882576Z inflating: dist/torch-2.4.0a0+gitd253bd7-cp38-cp38-linux_x86_64.whl 2024-04-03T16:05:38.7883620Z creating: build/custom_test_artifacts/ 2024-04-03T16:05:38.7884743Z creating: build/custom_test_artifacts/custom-op-build/ 2024-04-03T16:05:38.7885767Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-04-03T16:05:38.7887156Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-04-03T16:05:38.7888175Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-04-03T16:05:38.7889526Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-04-03T16:05:38.7891079Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-04-03T16:05:38.7892089Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-04-03T16:05:38.7893532Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-04-03T16:05:38.7895291Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-04-03T16:05:38.7896911Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-04-03T16:05:38.7898684Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-04-03T16:05:38.7900672Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-04-03T16:05:38.7903131Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-04-03T16:05:38.7904867Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-04-03T16:05:38.7906201Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-04-03T16:05:38.7907374Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-04-03T16:05:38.7908563Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-04-03T16:05:38.7909768Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-04-03T16:05:38.7910756Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-04-03T16:05:38.7911690Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-04-03T16:05:38.7912627Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-04-03T16:05:38.7913652Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-04-03T16:05:38.7914833Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-04-03T16:05:38.7916251Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-04-03T16:05:38.7917317Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-04-03T16:05:38.7918402Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-04-03T16:05:38.7919507Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-04-03T16:05:38.7920613Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-04-03T16:05:38.7921705Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-04-03T16:05:38.7923303Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-04-03T16:05:38.7924404Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-04-03T16:05:38.8045477Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-04-03T16:05:38.8047462Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-04-03T16:05:38.8049438Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-04-03T16:05:38.8051175Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-04-03T16:05:38.8052378Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-04-03T16:05:38.8053501Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-04-03T16:05:38.8054662Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-04-03T16:05:38.8055842Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-04-03T16:05:38.8057003Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-04-03T16:05:38.8058182Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-04-03T16:05:38.8059330Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-04-03T16:05:38.8069544Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-04-03T16:05:38.8146016Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-04-03T16:05:38.8148177Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-03T16:05:38.8149768Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-04-03T16:05:38.8151668Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-04-03T16:05:38.8152588Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-04-03T16:05:38.8153511Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-04-03T16:05:38.8154395Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-04-03T16:05:38.8155167Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-04-03T16:05:38.8156220Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-04-03T16:05:38.8258107Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-04-03T16:05:38.8317930Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-04-03T16:05:38.8318970Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-04-03T16:05:38.8319657Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-04-03T16:05:38.8320483Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-04-03T16:05:38.8321925Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-04-03T16:05:38.8323486Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-04-03T16:05:38.8324434Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-04-03T16:05:38.8325877Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-04-03T16:05:38.8327496Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-04-03T16:05:38.8328624Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-04-03T16:05:38.8330192Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-04-03T16:05:38.8331630Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-04-03T16:05:38.8333313Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-04-03T16:05:38.8335239Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-04-03T16:05:38.8337483Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-04-03T16:05:38.8339599Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-04-03T16:05:38.8341523Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-04-03T16:05:38.8342711Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-04-03T16:05:38.8343903Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-04-03T16:05:38.8344887Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-04-03T16:05:38.8345810Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-04-03T16:05:38.8346759Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-04-03T16:05:38.8347821Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-04-03T16:05:38.8349043Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-04-03T16:05:38.8350207Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-04-03T16:05:38.8351504Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-04-03T16:05:38.8352629Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-04-03T16:05:38.8353877Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-04-03T16:05:38.8355015Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-04-03T16:05:38.8356395Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-04-03T16:05:38.8357525Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-04-03T16:05:38.8360829Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-04-03T16:05:38.8420740Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-04-03T16:05:38.8422890Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-03T16:05:38.8424498Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-04-03T16:05:38.8426409Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-04-03T16:05:38.8427333Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-04-03T16:05:38.8428217Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-04-03T16:05:38.8429076Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-04-03T16:05:38.8429843Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-04-03T16:05:38.8430623Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-04-03T16:05:38.8475397Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-04-03T16:05:38.8476441Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-04-03T16:05:38.8477205Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-04-03T16:05:38.8478081Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-04-03T16:05:38.8479231Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-04-03T16:05:38.8480425Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-04-03T16:05:38.8481434Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-04-03T16:05:38.8482509Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-04-03T16:05:38.8483630Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-04-03T16:05:38.8484840Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-04-03T16:05:38.8486026Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-04-03T16:05:38.8487114Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-04-03T16:05:38.8488194Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-04-03T16:05:38.8489445Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-04-03T16:05:38.8490841Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-04-03T16:05:38.8492077Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-04-03T16:05:38.8493333Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-04-03T16:05:38.8494817Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-04-03T16:05:38.8496083Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-04-03T16:05:38.8497155Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-04-03T16:05:38.8498258Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-04-03T16:05:38.8499272Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-04-03T16:05:38.8500439Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-04-03T16:05:38.8501750Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-04-03T16:05:38.8503010Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-04-03T16:05:38.8504167Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-04-03T16:05:38.8505387Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-04-03T16:05:38.8506621Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-04-03T16:05:38.8507835Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-04-03T16:05:38.8509055Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-04-03T16:05:38.8510260Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-04-03T16:05:38.8511551Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-04-03T16:05:38.8617426Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-04-03T16:05:38.8618636Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-04-03T16:05:38.8619866Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-04-03T16:05:38.8621240Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-04-03T16:05:38.8622543Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-04-03T16:05:38.8623803Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-04-03T16:05:38.8625084Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-04-03T16:05:38.8626379Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-04-03T16:05:38.8627670Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-04-03T16:05:38.8628963Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-04-03T16:05:38.8630239Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-04-03T16:05:38.8640123Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-04-03T16:05:38.8690959Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-04-03T16:05:38.8692305Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-03T16:05:38.8693585Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-04-03T16:05:38.8694825Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-04-03T16:05:38.8695822Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-04-03T16:05:38.8696805Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-04-03T16:05:38.8697851Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-04-03T16:05:38.8698681Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-04-03T16:05:38.8699556Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-04-03T16:05:38.8795026Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-04-03T16:05:38.8836430Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-04-03T16:05:38.8837071Z creating: build/lib/ 2024-04-03T16:05:38.9105001Z inflating: build/lib/libclog.a 2024-04-03T16:05:38.9115021Z inflating: build/lib/libpthreadpool.a 2024-04-03T16:05:38.9123113Z inflating: build/lib/libittnotify.a 2024-04-03T16:05:38.9188297Z inflating: build/lib/libgtest.a 2024-04-03T16:05:38.9196643Z inflating: build/lib/libcpuinfo.a 2024-04-03T16:05:38.9204681Z inflating: build/lib/libcpuinfo_internals.a 2024-04-03T16:05:38.9205583Z inflating: build/lib/libfoxi_loader.a 2024-04-03T16:05:38.9226639Z inflating: build/lib/libfmt.a 2024-04-03T16:05:38.9307955Z inflating: build/lib/libprotobuf-lite.a 2024-04-03T16:05:38.9308984Z inflating: build/lib/libtorch_global_deps.so 2024-04-03T16:05:38.9380898Z inflating: build/lib/libbenchmark.a 2024-04-03T16:05:38.9408273Z inflating: build/lib/libtensorpipe_uv.a 2024-04-03T16:05:38.9410778Z inflating: build/lib/libnnpack_reference_layers.a 2024-04-03T16:05:38.9521994Z inflating: build/lib/libgloo.a 2024-04-03T16:05:38.9522868Z inflating: build/lib/libgtest_main.a 2024-04-03T16:05:38.9540468Z inflating: build/lib/libgmock.a 2024-04-03T16:05:38.9541296Z inflating: build/lib/libbenchmark_main.a 2024-04-03T16:05:38.9604118Z inflating: build/lib/libasmjit.a 2024-04-03T16:05:38.9617446Z inflating: build/lib/libqnnpack.a 2024-04-03T16:05:38.9636774Z inflating: build/lib/libpytorch_qnnpack.a 2024-04-03T16:05:38.9915003Z inflating: build/lib/libkineto.a 2024-04-03T16:05:39.0347854Z inflating: build/lib/libprotobuf.a 2024-04-03T16:05:39.0348757Z inflating: build/lib/libgmock_main.a 2024-04-03T16:05:39.1359673Z inflating: build/lib/libfbgemm.a 2024-04-03T16:05:39.1895470Z inflating: build/lib/libtensorpipe.a 2024-04-03T16:05:39.1984141Z inflating: build/lib/libc10.so 2024-04-03T16:05:39.2002187Z inflating: build/lib/libnnpack.a 2024-04-03T16:05:39.2487779Z inflating: build/lib/libprotoc.a 2024-04-03T16:05:39.2667466Z inflating: build/lib/libXNNPACK.a 2024-04-03T16:05:40.1408920Z inflating: build/lib/libdnnl.a 2024-04-03T16:05:40.1445509Z inflating: build/lib/libcaffe2_protos.a 2024-04-03T16:05:40.1484837Z inflating: build/lib/libonnx_proto.a 2024-04-03T16:05:40.2138912Z inflating: build/lib/libonnx.a 2024-04-03T16:05:42.4130013Z inflating: build/lib/libtorch_cpu.so 2024-04-03T16:05:42.4130917Z inflating: build/lib/libtorch.so 2024-04-03T16:05:42.4135023Z inflating: build/lib/libshm.so 2024-04-03T16:05:42.4154704Z inflating: build/lib/libjitbackend_test.so 2024-04-03T16:05:42.4180027Z inflating: build/lib/libbackend_with_compiler.so 2024-04-03T16:05:42.4184905Z inflating: build/lib/libunbox_lib.a 2024-04-03T16:05:42.4241736Z inflating: build/lib/libtorchbind_test.so 2024-04-03T16:05:42.5993471Z inflating: build/lib/libtorch_python.so 2024-04-03T16:05:42.6027533Z inflating: build/lib/libnnapi_backend.so 2024-04-03T16:05:42.6028385Z creating: build/bin/ 2024-04-03T16:05:42.6028999Z creating: build/bin/CMakeFiles/ 2024-04-03T16:05:42.6030308Z inflating: build/bin/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-03T16:05:42.6031557Z extracting: build/bin/CMakeFiles/progress.marks 2024-04-03T16:05:42.6098925Z inflating: build/bin/pow_test 2024-04-03T16:05:42.6099734Z inflating: build/bin/Makefile 2024-04-03T16:05:42.6202943Z inflating: build/bin/kernel_lambda_test 2024-04-03T16:05:42.6203863Z inflating: build/bin/cmake_install.cmake 2024-04-03T16:05:42.6258209Z inflating: build/bin/stride_properties_test 2024-04-03T16:05:42.6259169Z inflating: build/bin/CTestTestfile.cmake 2024-04-03T16:05:42.6311337Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-04-03T16:05:42.6362276Z inflating: build/bin/c10_StreamGuard_test 2024-04-03T16:05:42.6414645Z inflating: build/bin/c10_Device_test 2024-04-03T16:05:42.6468174Z inflating: build/bin/c10_DeviceGuard_test 2024-04-03T16:05:42.6521338Z inflating: build/bin/c10_SymInt_test 2024-04-03T16:05:42.6572441Z inflating: build/bin/c10_ConstexprCrc_test 2024-04-03T16:05:42.6628776Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-04-03T16:05:42.6686626Z inflating: build/bin/c10_SizesAndStrides_test 2024-04-03T16:05:42.6747890Z inflating: build/bin/c10_DispatchKeySet_test 2024-04-03T16:05:42.6803593Z inflating: build/bin/c10_Scalar_test 2024-04-03T16:05:42.6858712Z inflating: build/bin/c10_Bitset_test 2024-04-03T16:05:42.6916391Z inflating: build/bin/c10_InlineStreamGuard_test 2024-04-03T16:05:42.6968266Z inflating: build/bin/c10_DeadlockDetection_test 2024-04-03T16:05:42.7040724Z inflating: build/bin/c10_cow_test 2024-04-03T16:05:42.7093585Z inflating: build/bin/c10_Half_test 2024-04-03T16:05:42.7151404Z inflating: build/bin/c10_LeftRight_test 2024-04-03T16:05:42.7203669Z inflating: build/bin/c10_Synchronized_test 2024-04-03T16:05:42.7257408Z inflating: build/bin/c10_TypeIndex_test 2024-04-03T16:05:42.7314948Z inflating: build/bin/c10_Metaprogramming_test 2024-04-03T16:05:42.7372497Z inflating: build/bin/c10_ThreadLocal_test 2024-04-03T16:05:42.7423508Z inflating: build/bin/c10_TypeTraits_test 2024-04-03T16:05:42.7476860Z inflating: build/bin/c10_TypeList_test 2024-04-03T16:05:42.7530701Z inflating: build/bin/c10_accumulate_test 2024-04-03T16:05:42.7583076Z inflating: build/bin/c10_flags_test 2024-04-03T16:05:42.7636141Z inflating: build/bin/c10_irange_test 2024-04-03T16:05:42.7690689Z inflating: build/bin/c10_exception_test 2024-04-03T16:05:42.7742789Z inflating: build/bin/c10_generic_math_test 2024-04-03T16:05:42.7801299Z inflating: build/bin/c10_complex_math_test 2024-04-03T16:05:42.7858655Z inflating: build/bin/c10_bfloat16_test 2024-04-03T16:05:42.7917351Z inflating: build/bin/c10_logging_test 2024-04-03T16:05:42.7969863Z inflating: build/bin/c10_bit_cast_test 2024-04-03T16:05:42.8027056Z inflating: build/bin/c10_complex_test 2024-04-03T16:05:42.8082556Z inflating: build/bin/c10_registry_test 2024-04-03T16:05:42.8146666Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-04-03T16:05:42.8223792Z inflating: build/bin/c10_optional_test 2024-04-03T16:05:42.8276478Z inflating: build/bin/c10_tempfile_test 2024-04-03T16:05:42.8330613Z inflating: build/bin/c10_string_util_test 2024-04-03T16:05:42.8379454Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-04-03T16:05:42.8433450Z inflating: build/bin/c10_ssize_test 2024-04-03T16:05:42.8585087Z inflating: build/bin/c10_small_vector_test 2024-04-03T16:05:42.8748576Z inflating: build/bin/c10_intrusive_ptr_test 2024-04-03T16:05:42.8809161Z inflating: build/bin/c10_string_view_test 2024-04-03T16:05:42.8867542Z inflating: build/bin/c10_typeid_test 2024-04-03T16:05:42.9195405Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-04-03T16:05:42.9555242Z inflating: build/bin/vec_test_all_types_AVX2 2024-04-03T16:05:42.9901297Z inflating: build/bin/vec_test_all_types_AVX512 2024-04-03T16:05:43.0353263Z inflating: build/bin/protoc-3.13.0.0 2024-04-03T16:05:43.0805498Z inflating: build/bin/protoc 2024-04-03T16:05:43.0859924Z inflating: build/bin/Dimname_test 2024-04-03T16:05:43.0914507Z inflating: build/bin/HashStoreTest 2024-04-03T16:05:43.0972197Z inflating: build/bin/TCPStoreTest 2024-04-03T16:05:43.1026803Z inflating: build/bin/FileStoreTest 2024-04-03T16:05:43.1079494Z inflating: build/bin/operator_name_test 2024-04-03T16:05:43.1081916Z inflating: build/bin/example_allreduce 2024-04-03T16:05:43.1138423Z inflating: build/bin/type_ptr_test 2024-04-03T16:05:43.1205278Z inflating: build/bin/ProcessGroupGlooTest 2024-04-03T16:05:43.1265078Z inflating: build/bin/atest 2024-04-03T16:05:43.1340031Z inflating: build/bin/Dict_test 2024-04-03T16:05:43.1406293Z inflating: build/bin/MaybeOwned_test 2024-04-03T16:05:43.1466486Z inflating: build/bin/apply_utils_test 2024-04-03T16:05:43.1522751Z inflating: build/bin/broadcast_test 2024-04-03T16:05:43.1580998Z inflating: build/bin/NamedTensor_test 2024-04-03T16:05:43.1645924Z inflating: build/bin/basic 2024-04-03T16:05:43.1698142Z inflating: build/bin/cpu_allocator_test 2024-04-03T16:05:43.1749706Z inflating: build/bin/dispatch_key_set_test 2024-04-03T16:05:43.1803898Z inflating: build/bin/cpu_profiling_allocator_test 2024-04-03T16:05:43.1862385Z inflating: build/bin/static_runtime_bench 2024-04-03T16:05:43.1913477Z inflating: build/bin/lazy_tensor_test 2024-04-03T16:05:43.1973351Z inflating: build/bin/cpu_generator_test 2024-04-03T16:05:43.2024900Z inflating: build/bin/dlconvertor_test 2024-04-03T16:05:43.2081917Z inflating: build/bin/half_test 2024-04-03T16:05:43.2137062Z inflating: build/bin/memory_format_test 2024-04-03T16:05:43.2243014Z inflating: build/bin/List_test 2024-04-03T16:05:43.2302112Z inflating: build/bin/extension_backend_test 2024-04-03T16:05:43.2356670Z inflating: build/bin/math_kernel_test 2024-04-03T16:05:43.2411348Z inflating: build/bin/memory_overlapping_test 2024-04-03T16:05:43.2464170Z inflating: build/bin/StorageUtils_test 2024-04-03T16:05:43.2585636Z inflating: build/bin/kernel_function_legacy_test 2024-04-03T16:05:43.2643901Z inflating: build/bin/scalar_tensor_test 2024-04-03T16:05:43.2696651Z inflating: build/bin/reportMemoryUsage_test 2024-04-03T16:05:43.2749976Z inflating: build/bin/packedtensoraccessor_test 2024-04-03T16:05:43.2802364Z inflating: build/bin/operators_test 2024-04-03T16:05:43.2859739Z inflating: build/bin/native_test 2024-04-03T16:05:43.2918699Z inflating: build/bin/quantized_test 2024-04-03T16:05:43.2921135Z inflating: build/bin/thread_init_test 2024-04-03T16:05:43.3018094Z inflating: build/bin/kernel_function_test 2024-04-03T16:05:43.3077513Z inflating: build/bin/scalar_test 2024-04-03T16:05:43.3328056Z inflating: build/bin/static_runtime_test 2024-04-03T16:05:43.3379281Z inflating: build/bin/reduce_ops_test 2024-04-03T16:05:43.3506228Z inflating: build/bin/kernel_lambda_legacy_test 2024-04-03T16:05:43.3507336Z inflating: build/bin/verify_api_visibility 2024-04-03T16:05:43.3561155Z inflating: build/bin/undefined_tensor_test 2024-04-03T16:05:43.3611721Z inflating: build/bin/op_allowlist_test 2024-04-03T16:05:43.3708096Z inflating: build/bin/ivalue_test 2024-04-03T16:05:43.3762960Z inflating: build/bin/mobile_memory_cleanup 2024-04-03T16:05:43.3815459Z inflating: build/bin/CppSignature_test 2024-04-03T16:05:43.3877267Z inflating: build/bin/kernel_stackbased_test 2024-04-03T16:05:43.3929988Z inflating: build/bin/weakref_test 2024-04-03T16:05:43.3995435Z inflating: build/bin/inline_container_test 2024-04-03T16:05:43.4048255Z inflating: build/bin/wrapdim_test 2024-04-03T16:05:43.4108838Z inflating: build/bin/IListRef_test 2024-04-03T16:05:43.4179521Z inflating: build/bin/legacy_vmap_test 2024-04-03T16:05:43.4232967Z inflating: build/bin/xla_tensor_test 2024-04-03T16:05:43.4312738Z inflating: build/bin/tensor_iterator_test 2024-04-03T16:05:43.4373684Z inflating: build/bin/type_test 2024-04-03T16:05:43.4467357Z inflating: build/bin/cpu_rng_test 2024-04-03T16:05:43.4534654Z inflating: build/bin/KernelFunction_test 2024-04-03T16:05:43.4589987Z inflating: build/bin/test_parallel 2024-04-03T16:05:43.4686288Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-04-03T16:05:43.4994622Z inflating: build/bin/op_registration_test 2024-04-03T16:05:43.4998645Z inflating: build/bin/torch_shm_manager 2024-04-03T16:05:43.5055447Z inflating: build/bin/backend_fallback_test 2024-04-03T16:05:43.5068359Z inflating: build/bin/tutorial_tensorexpr 2024-04-03T16:05:43.5124563Z inflating: build/bin/test_dist_autograd 2024-04-03T16:05:43.5126672Z inflating: build/bin/parallel_benchmark 2024-04-03T16:05:43.5196778Z inflating: build/bin/test_cpp_rpc 2024-04-03T16:05:43.5205438Z inflating: build/bin/aot_model_compiler_test 2024-04-03T16:05:43.5273177Z inflating: build/bin/test_mobile_nnc 2024-04-03T16:05:43.5327866Z inflating: build/bin/test_edge_op_registration 2024-04-03T16:05:43.6114600Z inflating: build/bin/test_tensorexpr 2024-04-03T16:05:43.6452841Z inflating: build/bin/test_lazy 2024-04-03T16:05:43.7574881Z inflating: build/bin/test_api 2024-04-03T16:05:43.8136351Z inflating: build/bin/test_jit 2024-04-03T16:05:43.8136790Z creating: .additional_ci_files/ 2024-04-03T16:05:43.8171701Z inflating: .additional_ci_files/test-times.json 2024-04-03T16:05:43.8314501Z inflating: .additional_ci_files/test-class-times.json 2024-04-03T16:05:43.8342619Z ##[group]Run rm artifacts.zip 2024-04-03T16:05:43.8342995Z rm artifacts.zip 2024-04-03T16:05:43.8351849Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:05:43.8352338Z env: 2024-04-03T16:05:43.8352582Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:43.8352910Z ##[endgroup] 2024-04-03T16:05:43.8782993Z ##[group]Run df -H 2024-04-03T16:05:43.8783291Z df -H 2024-04-03T16:05:43.8790881Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:05:43.8791371Z env: 2024-04-03T16:05:43.8791617Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:43.8791943Z ##[endgroup] 2024-04-03T16:05:43.9116464Z Filesystem Size Used Avail Use% Mounted on 2024-04-03T16:05:43.9116997Z devtmpfs 8.2G 0 8.2G 0% /dev 2024-04-03T16:05:43.9117680Z tmpfs 8.2G 1.7M 8.2G 1% /dev/shm 2024-04-03T16:05:43.9118346Z tmpfs 8.2G 410k 8.2G 1% /run 2024-04-03T16:05:43.9119163Z tmpfs 8.2G 0 8.2G 0% /sys/fs/cgroup 2024-04-03T16:05:43.9119625Z /dev/nvme0n1p1 162G 23G 139G 15% / 2024-04-03T16:05:43.9220992Z Prepare all required actions 2024-04-03T16:05:43.9221438Z Getting action download info 2024-04-03T16:05:44.0474411Z ##[group]Run ./.github/actions/download-td-artifacts 2024-04-03T16:05:44.0474863Z with: 2024-04-03T16:05:44.0475096Z env: 2024-04-03T16:05:44.0475347Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:44.0475807Z ##[endgroup] 2024-04-03T16:05:44.0561438Z ##[group]Run seemethere/download-artifact-s3@v4 2024-04-03T16:05:44.0561870Z with: 2024-04-03T16:05:44.0562115Z name: td_results 2024-04-03T16:05:44.0562396Z s3-bucket: gha-artifacts 2024-04-03T16:05:44.0562813Z region: us-east-1 2024-04-03T16:05:44.0563087Z env: 2024-04-03T16:05:44.0563335Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:44.0563652Z ##[endgroup] 2024-04-03T16:05:44.5277583Z (node:14416) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-04-03T16:05:44.5278306Z 2024-04-03T16:05:44.5278643Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-04-03T16:05:44.5279602Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-04-03T16:05:44.5280950Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-04-03T16:05:44.5969408Z Found 1 objects with prefix pytorch/pytorch/8536950919/td_results/ 2024-04-03T16:05:44.5971147Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-04-03T16:05:44.6561206Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-04-03T16:05:44.6567307Z Artifact download has finished successfully 2024-04-03T16:05:44.6717626Z ##[group]Run mkdir -p .additional_ci_files 2024-04-03T16:05:44.6718211Z mkdir -p .additional_ci_files 2024-04-03T16:05:44.6718760Z mv td_results.json .additional_ci_files/td_results.json 2024-04-03T16:05:44.6726520Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:05:44.6727009Z env: 2024-04-03T16:05:44.6727267Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:44.6727594Z ##[endgroup] 2024-04-03T16:05:44.6847166Z ##[group]Run .github/scripts/parse_ref.py 2024-04-03T16:05:44.6847617Z .github/scripts/parse_ref.py 2024-04-03T16:05:44.6854945Z shell: /usr/bin/bash -e {0} 2024-04-03T16:05:44.6855276Z env: 2024-04-03T16:05:44.6855517Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:44.6855843Z ##[endgroup] 2024-04-03T16:05:44.7240263Z Prepare all required actions 2024-04-03T16:05:44.7294617Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-04-03T16:05:44.7295065Z with: 2024-04-03T16:05:44.7295607Z github-token: *** 2024-04-03T16:05:44.7295886Z env: 2024-04-03T16:05:44.7296139Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:44.7296461Z ##[endgroup] 2024-04-03T16:05:44.7447610Z ##[group]Run set -eux 2024-04-03T16:05:44.7447936Z set -eux 2024-04-03T16:05:44.7448511Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-04-03T16:05:44.7456258Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:05:44.7456729Z env: 2024-04-03T16:05:44.7456987Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:05:44.7457505Z GITHUB_TOKEN: *** 2024-04-03T16:05:44.7457794Z ##[endgroup] 2024-04-03T16:05:44.7479760Z + python3 .github/scripts/get_workflow_job_id.py 8536950919 i-07486a5f9ad15937b 2024-04-03T16:06:27.7826455Z RuntimeError(('Is github alright?', "Recieved status code '502' when attempting to retrieve https://api.github.com/repos/pytorch/pytorch/actions/runs/8536950919/jobs?per_page=100:\n", 'Bad Gateway\n\nheaders=Server: GitHub.com\nDate: Wed, 03 Apr 2024 16:06:27 GMT\nContent-Type: application/json\nContent-Length: 32\nETag: "660d79bc-20"\nVary: Accept-Encoding, Accept, X-Requested-With\nX-GitHub-Request-Id: 93BA:1E4431:191AEC1:290E684:660D7E79\nconnection: close\n\n')) 2024-04-03T16:06:27.7829281Z workflow-8536950919 2024-04-03T16:06:27.8059933Z Prepare all required actions 2024-04-03T16:06:27.8060370Z Getting action download info 2024-04-03T16:06:27.9230079Z ##[group]Run ./.github/actions/filter-test-configs 2024-04-03T16:06:27.9230531Z with: 2024-04-03T16:06:27.9230993Z github-token: *** 2024-04-03T16:06:27.9233941Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "default", "shard": 2, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "default", "shard": 3, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "docs_test", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge", "unstable": "unstable"}, {"config": "jit_legacy", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "backwards_compat", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge"}]} 2024-04-03T16:06:27.9237263Z env: 2024-04-03T16:06:27.9237510Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:06:27.9237835Z ##[endgroup] 2024-04-03T16:06:27.9280951Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-04-03T16:06:27.9281504Z with: 2024-04-03T16:06:27.9281749Z shell: bash 2024-04-03T16:06:27.9282011Z timeout_minutes: 10 2024-04-03T16:06:27.9282310Z max_attempts: 5 2024-04-03T16:06:27.9282599Z retry_wait_seconds: 30 2024-04-03T16:06:27.9283445Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-04-03T16:06:27.9284286Z polling_interval_seconds: 1 2024-04-03T16:06:27.9284636Z warning_on_retry: true 2024-04-03T16:06:27.9284953Z continue_on_error: false 2024-04-03T16:06:27.9285270Z env: 2024-04-03T16:06:27.9285650Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:06:27.9286143Z GITHUB_TOKEN: *** 2024-04-03T16:06:27.9286417Z ##[endgroup] 2024-04-03T16:06:27.9835304Z + python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-04-03T16:06:28.1936334Z Defaulting to user installation because normal site-packages is not writeable 2024-04-03T16:06:28.2109035Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0) 2024-04-03T16:06:28.2236887Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-04-03T16:06:28.2245075Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.18) 2024-04-03T16:06:28.2417183Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2024.2.2) 2024-04-03T16:06:28.2426263Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.6) 2024-04-03T16:06:28.2437532Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12) 2024-04-03T16:06:28.9830768Z Command completed after 1 attempt(s). 2024-04-03T16:06:28.9872202Z ##[group]Run set -x 2024-04-03T16:06:28.9872517Z set -x 2024-04-03T16:06:28.9872789Z  2024-04-03T16:06:28.9873323Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-04-03T16:06:28.9873977Z # in runner workspace 2024-04-03T16:06:28.9874477Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-04-03T16:06:28.9882570Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:06:28.9883145Z env: 2024-04-03T16:06:28.9883393Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:06:28.9883716Z ##[endgroup] 2024-04-03T16:06:28.9905955Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-04-03T16:06:29.0105887Z ##[group]Run set -x 2024-04-03T16:06:29.0106372Z set -x 2024-04-03T16:06:29.0106648Z  2024-04-03T16:06:29.0107179Z # TODO: This is a very hacky way to get the job name. GitHub runner has the info 2024-04-03T16:06:29.0108105Z # but doesn't expose it in anyway. The job name is part of the job message the 2024-04-03T16:06:29.0109121Z # runner receives, so it's there and printed out to the diag log. Below is the 2024-04-03T16:06:29.0110028Z # code responsible for printing it. Need to check with GitHub to see if they can 2024-04-03T16:06:29.0110781Z # expose this variable as part of GitHub context. 2024-04-03T16:06:29.0111609Z # https://github.com/actions/runner/blob/main/src/Runner.Worker/JobExtension.cs#L345 2024-04-03T16:06:29.0112499Z pushd "/home/ec2-user/actions-runner/_work/pytorch/../../_diag" 2024-04-03T16:06:29.0127856Z pwd 2024-04-03T16:06:29.0128212Z  2024-04-03T16:06:29.0128768Z LOG_FILE=$(grep -l -r "4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0" *.log | tail -n 1) 2024-04-03T16:06:29.0129465Z if [ -n "${LOG_FILE}" ]; then 2024-04-03T16:06:29.0130149Z  # For some reasons, awk {print $2} on Linux and Windows (bash) work correctly while it 2024-04-03T16:06:29.0130874Z  # needs to be awk {print $3} on MacOS 2024-04-03T16:06:29.0131326Z  case ${RUNNER_OS} in 2024-04-03T16:06:29.0131683Z  macOS) 2024-04-03T16:06:29.0132326Z  JOB_NAME=$(grep -r "\"jobDisplayName\"" "${LOG_FILE}" | awk -F '[:]' '{print $3}' | sed 's/"//g' | xargs) 2024-04-03T16:06:29.0133051Z  ;; 2024-04-03T16:06:29.0133329Z  *) 2024-04-03T16:06:29.0133947Z  JOB_NAME=$(grep -r "\"jobDisplayName\"" "${LOG_FILE}" | awk -F '[:]' '{print $2}' | sed 's/"//g' | xargs) 2024-04-03T16:06:29.0134840Z  ;; 2024-04-03T16:06:29.0135116Z  esac 2024-04-03T16:06:29.0135517Z  echo "job-name=${JOB_NAME}" >> "${GITHUB_OUTPUT}" 2024-04-03T16:06:29.0135981Z fi 2024-04-03T16:06:29.0136232Z  2024-04-03T16:06:29.0136479Z popd 2024-04-03T16:06:29.0144064Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:06:29.0144530Z env: 2024-04-03T16:06:29.0144786Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:06:29.0145098Z ##[endgroup] 2024-04-03T16:06:29.0166592Z + pushd /home/ec2-user/actions-runner/_work/pytorch/../../_diag 2024-04-03T16:06:29.0167418Z ~/actions-runner/_diag ~/actions-runner/_work/pytorch/pytorch 2024-04-03T16:06:29.0169548Z + pwd 2024-04-03T16:06:29.0169951Z /home/ec2-user/actions-runner/_diag 2024-04-03T16:06:29.0173914Z ++ grep -l -r 4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 Runner_20240403-102308-utc.log Runner_20240403-102311-utc.log Worker_20240403-102316-utc.log Worker_20240403-103559-utc.log Worker_20240403-120847-utc.log Worker_20240403-124001-utc.log Worker_20240403-124502-utc.log Worker_20240403-132709-utc.log Worker_20240403-155331-utc.log Worker_20240403-160009-utc.log 2024-04-03T16:06:29.0175962Z ++ tail -n 1 2024-04-03T16:06:29.0590759Z + LOG_FILE=Worker_20240403-160009-utc.log 2024-04-03T16:06:29.0591292Z + '[' -n Worker_20240403-160009-utc.log ']' 2024-04-03T16:06:29.0591829Z + case ${RUNNER_OS} in 2024-04-03T16:06:29.0596845Z ++ grep -r '"jobDisplayName"' Worker_20240403-160009-utc.log 2024-04-03T16:06:29.0597560Z ++ awk -F '[:]' '{print $2}' 2024-04-03T16:06:29.0597981Z ++ sed 's/"//g' 2024-04-03T16:06:29.0598690Z ++ xargs 2024-04-03T16:06:29.0645742Z + JOB_NAME='linux-jammy-py3.8-gcc11 / test (distributed, 1, 2, linux.2xlarge),' 2024-04-03T16:06:29.0646904Z + echo 'job-name=linux-jammy-py3.8-gcc11 / test (distributed, 1, 2, linux.2xlarge),' 2024-04-03T16:06:29.0647568Z + popd 2024-04-03T16:06:29.0647895Z ~/actions-runner/_work/pytorch/pytorch 2024-04-03T16:06:29.0673873Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-04-03T16:06:29.0674391Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-04-03T16:06:29.0674837Z echo "Job name: ${JOB_NAME}" 2024-04-03T16:06:29.0675332Z  2024-04-03T16:06:29.0676039Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-04-03T16:06:29.0676710Z # in runner workspace 2024-04-03T16:06:29.0677269Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-04-03T16:06:29.0677890Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-04-03T16:06:29.0678336Z  --job-name "${JOB_NAME}" \ 2024-04-03T16:06:29.0681435Z  --test-matrix "{"include": [{"config": "default", "shard": 1, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "default", "shard": 2, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "default", "shard": 3, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "docs_test", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge", "unstable": "unstable"}, {"config": "jit_legacy", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "backwards_compat", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge"}]}" \ 2024-04-03T16:06:29.0684640Z  --pr-number "${PR_NUMBER}" \ 2024-04-03T16:06:29.0685048Z  --tag "${TAG}" \ 2024-04-03T16:06:29.0685406Z  --event-name "${EVENT_NAME}" \ 2024-04-03T16:06:29.0685832Z  --schedule "${SCHEDULE}" \ 2024-04-03T16:06:29.0686235Z  --branch "${HEAD_BRANCH}" 2024-04-03T16:06:29.0693644Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:06:29.0694118Z env: 2024-04-03T16:06:29.0694374Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:06:29.0694897Z GITHUB_TOKEN: *** 2024-04-03T16:06:29.0695391Z JOB_NAME: linux-jammy-py3.8-gcc11 / test (distributed, 1, 2, linux.2xlarge), 2024-04-03T16:06:29.0696121Z PR_NUMBER: 123154 2024-04-03T16:06:29.0696405Z TAG: 2024-04-03T16:06:29.0696660Z EVENT_NAME: pull_request 2024-04-03T16:06:29.0696975Z SCHEDULE: 2024-04-03T16:06:29.0697238Z HEAD_BRANCH: 2024-04-03T16:06:29.0697495Z ##[endgroup] 2024-04-03T16:06:29.0717050Z Workflow: pull 2024-04-03T16:06:29.0717975Z Job name: linux-jammy-py3.8-gcc11 / test (distributed, 1, 2, linux.2xlarge), 2024-04-03T16:06:29.3105941Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-04-03T16:06:29.4586185Z fatal: unknown commit main 2024-04-03T16:06:29.4608575Z /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/filter_test_configs.py:458: UserWarning: failed to get commit messages: Command '['git', 'cherry', '-v', 'main']' returned non-zero exit status 128. 2024-04-03T16:06:29.4610214Z warnings.warn(f"failed to get commit messages: {e}") 2024-04-03T16:06:29.4732508Z ##[group]Run echo "Filtered matrix:" 2024-04-03T16:06:29.4732952Z echo "Filtered matrix:" 2024-04-03T16:06:29.4735989Z echo "{"include": [{"config": "default", "shard": 1, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "default", "shard": 2, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "default", "shard": 3, "num_shards": 3, "runner": "linux.2xlarge"}, {"config": "docs_test", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge", "unstable": "unstable"}, {"config": "jit_legacy", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "backwards_compat", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge"}, {"config": "distributed", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge"}]}" 2024-04-03T16:06:29.4739006Z  2024-04-03T16:06:29.4739243Z echo 2024-04-03T16:06:29.4739595Z echo "Is the current job unstable? False" 2024-04-03T16:06:29.4740043Z  2024-04-03T16:06:29.4740272Z echo 2024-04-03T16:06:29.4740599Z echo "Is keep-going label set? False" 2024-04-03T16:06:29.4741147Z  2024-04-03T16:06:29.4741380Z echo 2024-04-03T16:06:29.4741670Z echo "Renabled issues? " 2024-04-03T16:06:29.4749122Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:06:29.4749603Z env: 2024-04-03T16:06:29.4749847Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:06:29.4750181Z ##[endgroup] 2024-04-03T16:06:29.4770804Z Filtered matrix: 2024-04-03T16:06:29.4776754Z {include: [{config: default, shard: 1, num_shards: 3, runner: linux.2xlarge}, {config: default, shard: 2, num_shards: 3, runner: linux.2xlarge}, {config: default, shard: 3, num_shards: 3, runner: linux.2xlarge}, {config: docs_test, shard: 1, num_shards: 1, runner: linux.2xlarge, unstable: unstable}, {config: jit_legacy, shard: 1, num_shards: 1, runner: linux.2xlarge}, {config: backwards_compat, shard: 1, num_shards: 1, runner: linux.2xlarge}, {config: distributed, shard: 1, num_shards: 2, runner: linux.2xlarge}, {config: distributed, shard: 2, num_shards: 2, runner: linux.2xlarge}]} 2024-04-03T16:06:29.4782265Z 2024-04-03T16:06:29.4782523Z Is the current job unstable? False 2024-04-03T16:06:29.4783056Z 2024-04-03T16:06:29.4783412Z Is keep-going label set? False 2024-04-03T16:06:29.4783862Z 2024-04-03T16:06:29.4784078Z Renabled issues? 2024-04-03T16:06:29.4842410Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-04-03T16:06:29.4843204Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-04-03T16:06:29.4850510Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T16:06:29.4850993Z env: 2024-04-03T16:06:29.4851237Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:06:29.4851563Z JOB_TIMEOUT: 240 2024-04-03T16:06:29.4851845Z ##[endgroup] 2024-04-03T16:06:29.4923341Z ##[group]Run set -x 2024-04-03T16:06:29.4923727Z set -x 2024-04-03T16:06:29.4924001Z  2024-04-03T16:06:29.4924470Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-04-03T16:06:29.4924990Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-04-03T16:06:29.4925606Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-04-03T16:06:29.4926128Z  TEST_COMMAND=.ci/onnx/test.sh 2024-04-03T16:06:29.4926509Z else 2024-04-03T16:06:29.4926822Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-04-03T16:06:29.4927221Z fi 2024-04-03T16:06:29.4927457Z  2024-04-03T16:06:29.4927900Z # detached container should get cleaned up by teardown_ec2_linux 2024-04-03T16:06:29.4928651Z # TODO: Stop building test binaries as part of the build phase 2024-04-03T16:06:29.4929303Z # Used for GPU_FLAG since that doesn't play nice 2024-04-03T16:06:29.4929882Z # shellcheck disable=SC2086,SC2090 2024-04-03T16:06:29.4930315Z container_name=$(docker run \ 2024-04-03T16:06:29.4930709Z  ${GPU_FLAG:-} \ 2024-04-03T16:06:29.4931060Z  -e BUILD_ENVIRONMENT \ 2024-04-03T16:06:29.4931427Z  -e PR_NUMBER \ 2024-04-03T16:06:29.4931766Z  -e GITHUB_ACTIONS \ 2024-04-03T16:06:29.4932134Z  -e GITHUB_REPOSITORY \ 2024-04-03T16:06:29.4932505Z  -e GITHUB_WORKFLOW \ 2024-04-03T16:06:29.4932864Z  -e GITHUB_JOB \ 2024-04-03T16:06:29.4933198Z  -e GITHUB_RUN_ID \ 2024-04-03T16:06:29.4933557Z  -e GITHUB_RUN_NUMBER \ 2024-04-03T16:06:29.4933922Z  -e GITHUB_RUN_ATTEMPT \ 2024-04-03T16:06:29.4934291Z  -e JOB_ID \ 2024-04-03T16:06:29.4934602Z  -e JOB_NAME \ 2024-04-03T16:06:29.4934909Z  -e BASE_SHA \ 2024-04-03T16:06:29.4935226Z  -e BRANCH \ 2024-04-03T16:06:29.4935535Z  -e SHA1 \ 2024-04-03T16:06:29.4935838Z  -e AWS_DEFAULT_REGION \ 2024-04-03T16:06:29.4936215Z  -e IN_WHEEL_TEST \ 2024-04-03T16:06:29.4936567Z  -e SHARD_NUMBER \ 2024-04-03T16:06:29.4936903Z  -e TEST_CONFIG \ 2024-04-03T16:06:29.4937244Z  -e NUM_TEST_SHARDS \ 2024-04-03T16:06:29.4937614Z  -e REENABLED_ISSUES \ 2024-04-03T16:06:29.4937987Z  -e CONTINUE_THROUGH_ERROR \ 2024-04-03T16:06:29.4938386Z  -e VERBOSE_TEST_LOGS \ 2024-04-03T16:06:29.4938762Z  -e NO_TEST_TIMEOUT \ 2024-04-03T16:06:29.4939097Z  -e NO_TD \ 2024-04-03T16:06:29.4939405Z  -e PR_LABELS \ 2024-04-03T16:06:29.4939786Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-04-03T16:06:29.4940210Z  -e SCCACHE_BUCKET \ 2024-04-03T16:06:29.4940578Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-04-03T16:06:29.4940966Z  -e XLA_CUDA \ 2024-04-03T16:06:29.4941324Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-04-03T16:06:29.4941814Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-04-03T16:06:29.4942315Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-04-03T16:06:29.4942805Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-04-03T16:06:29.4943250Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-04-03T16:06:29.4943646Z  -e DASHBOARD_TAG \ 2024-04-03T16:06:29.4944084Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-04-03T16:06:29.4944602Z  --ulimit stack=10485760:83886080 \ 2024-04-03T16:06:29.4945064Z  --security-opt seccomp=unconfined \ 2024-04-03T16:06:29.4945507Z  --cap-add=SYS_PTRACE \ 2024-04-03T16:06:29.4945862Z  --ipc=host \ 2024-04-03T16:06:29.4946194Z  --shm-size="${SHM_SIZE}" \ 2024-04-03T16:06:29.4946568Z  --tty \ 2024-04-03T16:06:29.4946843Z  --detach \ 2024-04-03T16:06:29.4947174Z  --name="${container_name}" \ 2024-04-03T16:06:29.4947582Z  --user jenkins \ 2024-04-03T16:06:29.4948147Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-04-03T16:06:29.4948705Z  -w /var/lib/jenkins/workspace \ 2024-04-03T16:06:29.4949204Z  "${DOCKER_IMAGE}" 2024-04-03T16:06:29.4949519Z ) 2024-04-03T16:06:29.4949909Z # Propagate download.pytorch.org IP to container 2024-04-03T16:06:29.4950835Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-04-03T16:06:29.4951802Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-04-03T16:06:29.4952722Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-04-03T16:06:29.4960329Z shell: /usr/bin/bash -e {0} 2024-04-03T16:06:29.4960669Z env: 2024-04-03T16:06:29.4960926Z GIT_DEFAULT_BRANCH: main 2024-04-03T16:06:29.4961307Z BUILD_ENVIRONMENT: linux-jammy-py3.8-gcc11 2024-04-03T16:06:29.4961737Z PR_NUMBER: 123154 2024-04-03T16:06:29.4962067Z GITHUB_REPOSITORY: pytorch/pytorch 2024-04-03T16:06:29.4962445Z GITHUB_WORKFLOW: pull 2024-04-03T16:06:29.4962755Z GITHUB_JOB: test 2024-04-03T16:06:29.4963150Z GITHUB_RUN_ID: 8536950919 2024-04-03T16:06:29.4963472Z GITHUB_RUN_NUMBER: 192590 2024-04-03T16:06:29.4963801Z GITHUB_RUN_ATTEMPT: 2 2024-04-03T16:06:29.4964101Z JOB_ID: 2024-04-03T16:06:29.4964334Z JOB_NAME: 2024-04-03T16:06:29.4964599Z BRANCH: pull/123154 2024-04-03T16:06:29.4964966Z SHA1: d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:06:29.4965455Z BASE_SHA: 1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:06:29.4965901Z TEST_CONFIG: distributed 2024-04-03T16:06:29.4966227Z SHARD_NUMBER: 1 2024-04-03T16:06:29.4966496Z NUM_TEST_SHARDS: 2 2024-04-03T16:06:29.4966790Z REENABLED_ISSUES: 2024-04-03T16:06:29.4967103Z CONTINUE_THROUGH_ERROR: False 2024-04-03T16:06:29.4967451Z VERBOSE_TEST_LOGS: False 2024-04-03T16:06:29.4967778Z NO_TEST_TIMEOUT: False 2024-04-03T16:06:29.4968088Z NO_TD: False 2024-04-03T16:06:29.4968434Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-04-03T16:06:29.4968910Z SCCACHE_S3_KEY_PREFIX: pull 2024-04-03T16:06:29.4969249Z SHM_SIZE: 1g 2024-04-03T16:06:29.4970095Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:06:29.4971041Z XLA_CUDA: 2024-04-03T16:06:29.4971505Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-04-03T16:06:29.4972107Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-04-03T16:06:29.4972503Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-04-03T16:06:29.4972886Z DASHBOARD_TAG: 2024-04-03T16:06:29.4973177Z HUGGING_FACE_HUB_TOKEN: 2024-04-03T16:06:29.4973483Z ##[endgroup] 2024-04-03T16:06:29.4993571Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-04-03T16:06:29.4994453Z + [[ linux-jammy-py3.8-gcc11 == *onnx* ]] 2024-04-03T16:06:29.4994886Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-04-03T16:06:29.5002283Z +++ nproc --ignore=2 2024-04-03T16:06:29.5026576Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e PR_LABELS -e MAX_JOBS=6 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_8536950919 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T16:06:30.0357019Z + container_name=b0f53da97a9da663582739611fc2277e27fd12f5b71cdfe54025ecd3466cc388 2024-04-03T16:06:30.0358445Z + grep download.pytorch.org /etc/hosts 2024-04-03T16:06:30.0361296Z + docker exec -i b0f53da97a9da663582739611fc2277e27fd12f5b71cdfe54025ecd3466cc388 sudo bash -c '/bin/cat >> /etc/hosts' 2024-04-03T16:06:30.1654048Z + echo DOCKER_CONTAINER_ID=b0f53da97a9da663582739611fc2277e27fd12f5b71cdfe54025ecd3466cc388 2024-04-03T16:06:30.1658407Z ++ echo dist/torch-2.4.0a0+gitd253bd7-cp38-cp38-linux_x86_64.whl 2024-04-03T16:06:30.1660862Z + docker exec -t b0f53da97a9da663582739611fc2277e27fd12f5b71cdfe54025ecd3466cc388 sh -c 'pip install dist/torch-2.4.0a0+gitd253bd7-cp38-cp38-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-04-03T16:06:30.7611182Z Processing ./dist/torch-2.4.0a0+gitd253bd7-cp38-cp38-linux_x86_64.whl 2024-04-03T16:06:31.1833201Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.4.0a0+gitd253bd7) (3.13.1) 2024-04-03T16:06:31.1841190Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.4.0a0+gitd253bd7) (4.10.0) 2024-04-03T16:06:31.1845449Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.4.0a0+gitd253bd7) (1.12) 2024-04-03T16:06:31.1848688Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.4.0a0+gitd253bd7) (2.8.8) 2024-04-03T16:06:31.1852381Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.4.0a0+gitd253bd7) (3.1.3) 2024-04-03T16:06:31.1856255Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.4.0a0+gitd253bd7) (2024.2.0) 2024-04-03T16:06:31.1866831Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.4.0a0+gitd253bd7) (3.3.0) 2024-04-03T16:06:31.1931719Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from opt-einsum>=3.3->torch==2.4.0a0+gitd253bd7) (1.21.2) 2024-04-03T16:06:31.2351560Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from jinja2->torch==2.4.0a0+gitd253bd7) (2.1.5) 2024-04-03T16:06:31.2514074Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from sympy->torch==2.4.0a0+gitd253bd7) (1.3.0) 2024-04-03T16:06:32.4571036Z Installing collected packages: torch 2024-04-03T16:06:39.6547634Z Successfully installed torch-2.4.0a0+gitd253bd7 2024-04-03T16:06:39.7327781Z + echo 'Environment variables:' 2024-04-03T16:06:39.7328236Z Environment variables: 2024-04-03T16:06:39.7328539Z + env 2024-04-03T16:06:39.7344488Z INSTALLED_DB=yes 2024-04-03T16:06:39.7345636Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:06:39.7346512Z CONTINUE_THROUGH_ERROR=False 2024-04-03T16:06:39.7347188Z BUILD_ENVIRONMENT=linux-jammy-py3.8-gcc11 2024-04-03T16:06:39.7347954Z HOSTNAME=b0f53da97a9d 2024-04-03T16:06:39.7349162Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_60cc9c06-c660-4d6b-b546-2afb827e6971 2024-04-03T16:06:39.7350373Z GITHUB_ACTION=__self 2024-04-03T16:06:39.7350917Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-04-03T16:06:39.7351555Z GITHUB_RUN_NUMBER=192590 2024-04-03T16:06:39.7352178Z TEST_CONFIG=distributed 2024-04-03T16:06:39.7352616Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-04-03T16:06:39.7353401Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-04-03T16:06:39.7353875Z GITHUB_TRIGGERING_ACTOR=soulitzer 2024-04-03T16:06:39.7354333Z GITHUB_REF_TYPE=branch 2024-04-03T16:06:39.7354650Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-04-03T16:06:39.7355104Z BASE_SHA=1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:06:39.7356074Z XLA_CUDA= 2024-04-03T16:06:39.7356492Z HUGGING_FACE_HUB_TOKEN= 2024-04-03T16:06:39.7360863Z *** 2024-04-03T16:06:39.7361148Z GITHUB_REPOSITORY_ID=65600975 2024-04-03T16:06:39.7361689Z GITHUB_ACTIONS=true 2024-04-03T16:06:39.7362039Z SHA1=d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:06:39.7362544Z GITHUB_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:06:39.7363333Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/123154/merge 2024-04-03T16:06:39.7363968Z UCC_HOME=/usr 2024-04-03T16:06:39.7364249Z VERBOSE_TEST_LOGS=False 2024-04-03T16:06:39.7364567Z GITHUB_REF=refs/pull/123154/merge 2024-04-03T16:06:39.7364929Z SHARD_NUMBER=1 2024-04-03T16:06:39.7365220Z GITHUB_REF_PROTECTED=false 2024-04-03T16:06:39.7365543Z HOME=/var/lib/jenkins 2024-04-03T16:06:39.7365892Z GITHUB_API_URL=https://api.github.com 2024-04-03T16:06:39.7366316Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-04-03T16:06:39.7366683Z UCX_COMMIT= 2024-04-03T16:06:39.7366960Z SCCACHE_S3_KEY_PREFIX=pull 2024-04-03T16:06:39.7367299Z NUM_TEST_SHARDS=2 2024-04-03T16:06:39.7367566Z UCX_HOME=/usr 2024-04-03T16:06:39.7368431Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_60cc9c06-c660-4d6b-b546-2afb827e6971 2024-04-03T16:06:39.7369265Z JOB_NAME= 2024-04-03T16:06:39.7370041Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_60cc9c06-c660-4d6b-b546-2afb827e6971 2024-04-03T16:06:39.7371179Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-04-03T16:06:39.7371843Z GITHUB_EVENT_NAME=pull_request 2024-04-03T16:06:39.7372172Z DASHBOARD_TAG= 2024-04-03T16:06:39.7372445Z GITHUB_RUN_ID=8536950919 2024-04-03T16:06:39.7373364Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_60cc9c06-c660-4d6b-b546-2afb827e6971 2024-04-03T16:06:39.7374225Z GITHUB_ACTOR=caic99 2024-04-03T16:06:39.7374513Z PR_NUMBER=123154 2024-04-03T16:06:39.7374785Z DESIRED_CUDA= 2024-04-03T16:06:39.7375047Z GITHUB_RUN_ATTEMPT=2 2024-04-03T16:06:39.7375355Z ANACONDA_PYTHON_VERSION=3.8 2024-04-03T16:06:39.7375770Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-04-03T16:06:39.7376205Z TERM=xterm 2024-04-03T16:06:39.7376462Z INSTALLED_VISION=yes 2024-04-03T16:06:39.7376759Z BRANCH=pull/123154 2024-04-03T16:06:39.7377048Z OPENSSL_ROOT_DIR=/opt/openssl 2024-04-03T16:06:39.7377396Z CUDA_PATH=/usr/local/cuda 2024-04-03T16:06:39.7378139Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-04-03T16:06:39.7378904Z GITHUB_SERVER_URL=https://github.com 2024-04-03T16:06:39.7379278Z UCC_COMMIT= 2024-04-03T16:06:39.7379540Z REENABLED_ISSUES= 2024-04-03T16:06:39.7379799Z DOCS=yes 2024-04-03T16:06:39.7380054Z INSTALLED_ANDROID= 2024-04-03T16:06:39.7380335Z SHLVL=1 2024-04-03T16:06:39.7380558Z MAX_JOBS=6 2024-04-03T16:06:39.7380823Z GITHUB_ACTOR_ID=78061359 2024-04-03T16:06:39.7381265Z GITHUB_WORKFLOW_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:06:39.7381768Z GITHUB_REF_NAME=123154/merge 2024-04-03T16:06:39.7382354Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-04-03T16:06:39.7382921Z GITHUB_JOB=test 2024-04-03T16:06:39.7383183Z NO_TEST_TIMEOUT=False 2024-04-03T16:06:39.7383507Z GITHUB_REPOSITORY=pytorch/pytorch 2024-04-03T16:06:39.7383882Z GITHUB_RETENTION_DAYS=90 2024-04-03T16:06:39.7384193Z OPENSSL_DIR=/opt/openssl 2024-04-03T16:06:39.7384521Z GITHUB_ACTION_REPOSITORY= 2024-04-03T16:06:39.7385574Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:06:39.7386643Z GITHUB_BASE_REF=main 2024-04-03T16:06:39.7386937Z INSTALLED_ACL= 2024-04-03T16:06:39.7387199Z CI=true 2024-04-03T16:06:39.7390615Z GITHUB_REPOSITORY_OWNER=pytorch 2024-04-03T16:06:39.7391278Z JOB_ID= 2024-04-03T16:06:39.7391727Z INSTALLED_PROTOBUF=yes 2024-04-03T16:06:39.7392374Z GITHUB_HEAD_REF=patch-1 2024-04-03T16:06:39.7393096Z GITHUB_ACTION_REF= 2024-04-03T16:06:39.7393864Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-04-03T16:06:39.7394671Z GITHUB_WORKFLOW=pull 2024-04-03T16:06:39.7395357Z DEBIAN_FRONTEND=noninteractive 2024-04-03T16:06:39.7397329Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_60cc9c06-c660-4d6b-b546-2afb827e6971 2024-04-03T16:06:39.7398844Z NO_TD=False 2024-04-03T16:06:39.7399334Z SKIP_SCCACHE_INITIALIZATION=1 2024-04-03T16:06:39.7399947Z _=/usr/bin/env 2024-04-03T16:06:39.7400765Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-04-03T16:06:39.7530610Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch 2024-04-03T16:06:39.7531912Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-04-03T16:06:39.7532877Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib 2024-04-03T16:06:39.7533926Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/test 2024-04-03T16:06:39.7534608Z + BUILD_DIR=build 2024-04-03T16:06:39.7534973Z + BUILD_RENAMED_DIR=build_renamed 2024-04-03T16:06:39.7535425Z + BUILD_BIN_DIR=build/bin 2024-04-03T16:06:39.7535752Z + SHARD_NUMBER=1 2024-04-03T16:06:39.7536020Z + NUM_TEST_SHARDS=2 2024-04-03T16:06:39.7536315Z + export VALGRIND=ON 2024-04-03T16:06:39.7536608Z + VALGRIND=ON 2024-04-03T16:06:39.7536960Z + [[ linux-jammy-py3.8-gcc11 == *clang9* ]] 2024-04-03T16:06:39.7537364Z + [[ 0 == \1 ]] 2024-04-03T16:06:39.7537633Z + [[ False == \1 ]] 2024-04-03T16:06:39.7537991Z + [[ linux-jammy-py3.8-gcc11 != *bazel* ]] 2024-04-03T16:06:39.7538433Z ++ realpath build/custom_test_artifacts 2024-04-03T16:06:39.7552958Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-04-03T16:06:39.7553673Z + [[ -n '' ]] 2024-04-03T16:06:39.7555680Z ++ dirname .ci/pytorch/test.sh 2024-04-03T16:06:39.7588690Z + source .ci/pytorch/common.sh 2024-04-03T16:06:39.7594907Z +++ dirname .ci/pytorch/common.sh 2024-04-03T16:06:39.7601607Z ++ source .ci/pytorch/common_utils.sh 2024-04-03T16:06:39.7607574Z +++ declare -f -t trap_add 2024-04-03T16:06:39.7611116Z ++ set -ex 2024-04-03T16:06:39.7611512Z ++ [[ linux-jammy-py3.8-gcc11 == *rocm* ]] 2024-04-03T16:06:39.7611958Z ++ BUILD_TEST_LIBTORCH=0 2024-04-03T16:06:39.7612338Z + echo 'Environment variables' 2024-04-03T16:06:39.7612682Z Environment variables 2024-04-03T16:06:39.7612979Z + env 2024-04-03T16:06:39.7620239Z INSTALLED_DB=yes 2024-04-03T16:06:39.7621415Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T16:06:39.7622360Z CONTINUE_THROUGH_ERROR=False 2024-04-03T16:06:39.7623086Z BUILD_ENVIRONMENT=linux-jammy-py3.8-gcc11 2024-04-03T16:06:39.7623570Z HOSTNAME=b0f53da97a9d 2024-04-03T16:06:39.7624766Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_60cc9c06-c660-4d6b-b546-2afb827e6971 2024-04-03T16:06:39.7625651Z GITHUB_ACTION=__self 2024-04-03T16:06:39.7625967Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-04-03T16:06:39.7626361Z GITHUB_RUN_NUMBER=192590 2024-04-03T16:06:39.7626694Z TEST_CONFIG=distributed 2024-04-03T16:06:39.7627025Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-04-03T16:06:39.7627497Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-04-03T16:06:39.7628048Z GITHUB_TRIGGERING_ACTOR=soulitzer 2024-04-03T16:06:39.7628584Z GITHUB_REF_TYPE=branch 2024-04-03T16:06:39.7628908Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-04-03T16:06:39.7629315Z BASE_SHA=1d52c2d98507759a2143d52b922c541dbe36b6bf 2024-04-03T16:06:39.7629728Z XLA_CUDA= 2024-04-03T16:06:39.7629988Z HUGGING_FACE_HUB_TOKEN= 2024-04-03T16:06:39.7630406Z *** 2024-04-03T16:06:39.7630659Z GITHUB_REPOSITORY_ID=65600975 2024-04-03T16:06:39.7630990Z GITHUB_ACTIONS=true 2024-04-03T16:06:39.7631339Z SHA1=d253bd7befc64137de913dbb8727b511f1052411 2024-04-03T16:06:39.7631843Z GITHUB_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:06:39.7632709Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/123154/merge 2024-04-03T16:06:39.7633341Z UCC_HOME=/usr 2024-04-03T16:06:39.7633862Z VERBOSE_TEST_LOGS=False 2024-04-03T16:06:39.7634184Z GITHUB_REF=refs/pull/123154/merge 2024-04-03T16:06:39.7634538Z SHARD_NUMBER=1 2024-04-03T16:06:39.7634960Z GITHUB_REF_PROTECTED=false 2024-04-03T16:06:39.7635290Z HOME=/var/lib/jenkins 2024-04-03T16:06:39.7636101Z GITHUB_API_URL=https://api.github.com 2024-04-03T16:06:39.7636528Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-04-03T16:06:39.7636890Z UCX_COMMIT= 2024-04-03T16:06:39.7637161Z SCCACHE_S3_KEY_PREFIX=pull 2024-04-03T16:06:39.7637491Z NUM_TEST_SHARDS=2 2024-04-03T16:06:39.7637758Z UCX_HOME=/usr 2024-04-03T16:06:39.7638614Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_60cc9c06-c660-4d6b-b546-2afb827e6971 2024-04-03T16:06:39.7639605Z JOB_NAME= 2024-04-03T16:06:39.7640382Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_60cc9c06-c660-4d6b-b546-2afb827e6971 2024-04-03T16:06:39.7641593Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-04-03T16:06:39.7642323Z GITHUB_EVENT_NAME=pull_request 2024-04-03T16:06:39.7642655Z DASHBOARD_TAG= 2024-04-03T16:06:39.7643057Z GITHUB_RUN_ID=8536950919 2024-04-03T16:06:39.7643992Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_60cc9c06-c660-4d6b-b546-2afb827e6971 2024-04-03T16:06:39.7644860Z GITHUB_ACTOR=caic99 2024-04-03T16:06:39.7645154Z PR_NUMBER=123154 2024-04-03T16:06:39.7645483Z DESIRED_CUDA= 2024-04-03T16:06:39.7645916Z GITHUB_RUN_ATTEMPT=2 2024-04-03T16:06:39.7646225Z VALGRIND=ON 2024-04-03T16:06:39.7646494Z ANACONDA_PYTHON_VERSION=3.8 2024-04-03T16:06:39.7646900Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-04-03T16:06:39.7647343Z TERM=xterm 2024-04-03T16:06:39.7647607Z INSTALLED_VISION=yes 2024-04-03T16:06:39.7647890Z BRANCH=pull/123154 2024-04-03T16:06:39.7648215Z OPENSSL_ROOT_DIR=/opt/openssl 2024-04-03T16:06:39.7648565Z CUDA_PATH=/usr/local/cuda 2024-04-03T16:06:39.7649316Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-04-03T16:06:39.7650069Z GITHUB_SERVER_URL=https://github.com 2024-04-03T16:06:39.7650622Z UCC_COMMIT= 2024-04-03T16:06:39.7650870Z REENABLED_ISSUES= 2024-04-03T16:06:39.7651142Z DOCS=yes 2024-04-03T16:06:39.7651391Z INSTALLED_ANDROID= 2024-04-03T16:06:39.7651651Z SHLVL=1 2024-04-03T16:06:39.7651888Z MAX_JOBS=6 2024-04-03T16:06:39.7652147Z GITHUB_ACTOR_ID=78061359 2024-04-03T16:06:39.7652579Z GITHUB_WORKFLOW_SHA=4302cb7ecb2b1ae758f931ad18aeedf6ae927cb0 2024-04-03T16:06:39.7653089Z GITHUB_REF_NAME=123154/merge 2024-04-03T16:06:39.7653836Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-04-03T16:06:39.7654393Z GITHUB_JOB=test 2024-04-03T16:06:39.7654673Z NO_TEST_TIMEOUT=False 2024-04-03T16:06:39.7654999Z GITHUB_REPOSITORY=pytorch/pytorch 2024-04-03T16:06:39.7655361Z GITHUB_RETENTION_DAYS=90 2024-04-03T16:06:39.7655680Z OPENSSL_DIR=/opt/openssl 2024-04-03T16:06:39.7656002Z GITHUB_ACTION_REPOSITORY= 2024-04-03T16:06:39.7657049Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:06:39.7658310Z GITHUB_BASE_REF=main 2024-04-03T16:06:39.7658611Z INSTALLED_ACL= 2024-04-03T16:06:39.7658858Z CI=true 2024-04-03T16:06:39.7659120Z GITHUB_REPOSITORY_OWNER=pytorch 2024-04-03T16:06:39.7659459Z JOB_ID= 2024-04-03T16:06:39.7659696Z INSTALLED_PROTOBUF=yes 2024-04-03T16:06:39.7660033Z GITHUB_HEAD_REF=patch-1 2024-04-03T16:06:39.7660342Z GITHUB_ACTION_REF= 2024-04-03T16:06:39.7660873Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-04-03T16:06:39.7661354Z GITHUB_WORKFLOW=pull 2024-04-03T16:06:39.7661672Z DEBIAN_FRONTEND=noninteractive 2024-04-03T16:06:39.7662577Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_60cc9c06-c660-4d6b-b546-2afb827e6971 2024-04-03T16:06:39.7663403Z NO_TD=False 2024-04-03T16:06:39.7663809Z SKIP_SCCACHE_INITIALIZATION=1 2024-04-03T16:06:39.7664332Z _=/usr/bin/env 2024-04-03T16:06:39.7665111Z + echo 'Testing pytorch' 2024-04-03T16:06:39.7665703Z Testing pytorch 2024-04-03T16:06:39.7666212Z + export LANG=C.UTF-8 2024-04-03T16:06:39.7666932Z + LANG=C.UTF-8 2024-04-03T16:06:39.7667430Z + PR_NUMBER=123154 2024-04-03T16:06:39.7667981Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-04-03T16:06:39.7668731Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-04-03T16:06:39.7669535Z + [[ linux-jammy-py3.8-gcc11 == *rocm* ]] 2024-04-03T16:06:39.7670186Z + [[ distributed == \s\l\o\w ]] 2024-04-03T16:06:39.7670679Z + [[ linux-jammy-py3.8-gcc11 == *slow-gradcheck* ]] 2024-04-03T16:06:39.7671259Z + [[ linux-jammy-py3.8-gcc11 == *cuda* ]] 2024-04-03T16:06:39.7671766Z + [[ linux-jammy-py3.8-gcc11 == *rocm* ]] 2024-04-03T16:06:39.7672249Z + [[ linux-jammy-py3.8-gcc11 == *xpu* ]] 2024-04-03T16:06:39.7672686Z + [[ distributed == *crossref* ]] 2024-04-03T16:06:39.7673138Z + [[ linux-jammy-py3.8-gcc11 == *rocm* ]] 2024-04-03T16:06:39.7673601Z + [[ linux-jammy-py3.8-gcc11 == *xpu* ]] 2024-04-03T16:06:39.7674104Z + [[ linux-jammy-py3.8-gcc11 != *-bazel-* ]] 2024-04-03T16:06:39.7674584Z + pip_install --user ninja==1.10.2 2024-04-03T16:06:39.7696825Z + pip install --progress-bar off --user ninja==1.10.2 2024-04-03T16:06:40.1804640Z Collecting ninja==1.10.2 2024-04-03T16:06:40.1983663Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-04-03T16:06:40.2165672Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-04-03T16:06:41.3279624Z Installing collected packages: ninja 2024-04-03T16:06:41.3366843Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-04-03T16:06:41.3368253Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-04-03T16:06:41.3419259Z Successfully installed ninja-1.10.2 2024-04-03T16:06:41.4247471Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:06:41.4249694Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:06:41.4250952Z + install_tlparse 2024-04-03T16:06:41.4251376Z + pip_install --user tlparse==0.3.7 2024-04-03T16:06:41.4251904Z + pip install --progress-bar off --user tlparse==0.3.7 2024-04-03T16:06:41.8217410Z Collecting tlparse==0.3.7 2024-04-03T16:06:41.8389465Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-04-03T16:06:41.8581815Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-04-03T16:06:42.9883462Z Installing collected packages: tlparse 2024-04-03T16:06:43.0294526Z Successfully installed tlparse-0.3.7 2024-04-03T16:06:43.1066196Z ++ python -m site --user-base 2024-04-03T16:06:43.1281201Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-03T16:06:43.1283201Z + [[ linux-jammy-py3.8-gcc11 == *asan* ]] 2024-04-03T16:06:43.1283807Z + [[ linux-jammy-py3.8-gcc11 == *-debug* ]] 2024-04-03T16:06:43.1284383Z + [[ linux-jammy-py3.8-gcc11 != *-bazel-* ]] 2024-04-03T16:06:43.1285219Z + echo 'We are not in debug mode: linux-jammy-py3.8-gcc11. Expect the assertion to pass' 2024-04-03T16:06:43.1286233Z We are not in debug mode: linux-jammy-py3.8-gcc11. Expect the assertion to pass 2024-04-03T16:06:43.1286906Z + cd test 2024-04-03T16:06:43.1287470Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-04-03T16:06:44.3650041Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-04-03T16:06:44.3651256Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-04-03T16:06:44.3652070Z + DYNAMO_BENCHMARK_FLAGS=() 2024-04-03T16:06:44.3652726Z + [[ distributed == *dynamo_eager* ]] 2024-04-03T16:06:44.3653662Z + [[ distributed == *aot_eager* ]] 2024-04-03T16:06:44.3654417Z + [[ distributed == *aot_inductor* ]] 2024-04-03T16:06:44.3655050Z + [[ distributed == *inductor* ]] 2024-04-03T16:06:44.3655438Z + [[ distributed == *dynamic* ]] 2024-04-03T16:06:44.3655829Z + [[ distributed == *cpu_inductor* ]] 2024-04-03T16:06:44.3656399Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-04-03T16:06:44.3661230Z + [[ linux-jammy-py3.8-gcc11 == *tbb* ]] 2024-04-03T16:06:44.3669958Z + [[ linux-jammy-py3.8-gcc11 == *libtorch* ]] 2024-04-03T16:06:44.3670668Z + [[ linux-jammy-py3.8-gcc11 == *-bazel-* ]] 2024-04-03T16:06:44.3673027Z + cd test 2024-04-03T16:06:44.3673588Z + python -c 'import torch; print(torch.__config__.show())' 2024-04-03T16:06:45.3969540Z PyTorch built with: 2024-04-03T16:06:45.3970340Z - GCC 11.4 2024-04-03T16:06:45.3970904Z - C++ Version: 201703 2024-04-03T16:06:45.3972166Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-04-03T16:06:45.3973271Z - Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361) 2024-04-03T16:06:45.3973908Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-04-03T16:06:45.3974398Z - LAPACK is enabled (usually provided by MKL) 2024-04-03T16:06:45.3974864Z - NNPACK is enabled 2024-04-03T16:06:45.3975216Z - CPU capability usage: AVX512 2024-04-03T16:06:45.3983107Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-04-03T16:06:45.3990300Z 2024-04-03T16:06:45.6011801Z + cd test 2024-04-03T16:06:45.6012820Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-04-03T16:06:46.5995366Z ATen/Parallel: 2024-04-03T16:06:46.5996293Z at::get_num_threads() : 4 2024-04-03T16:06:46.5996996Z at::get_num_interop_threads() : 8 2024-04-03T16:06:46.5997816Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-04-03T16:06:46.5998533Z omp_get_max_threads() : 4 2024-04-03T16:06:46.6000320Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-04-03T16:06:46.6001978Z mkl_get_max_threads() : 4 2024-04-03T16:06:46.6003224Z Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361) 2024-04-03T16:06:46.6004438Z std::thread::hardware_concurrency() : 8 2024-04-03T16:06:46.6005232Z Environment variables: 2024-04-03T16:06:46.6005846Z OMP_NUM_THREADS : [not set] 2024-04-03T16:06:46.6006533Z MKL_NUM_THREADS : [not set] 2024-04-03T16:06:46.6007234Z ATen parallel backend: OpenMP 2024-04-03T16:06:46.6007706Z 2024-04-03T16:06:46.7798126Z + [[ distributed == *backward* ]] 2024-04-03T16:06:46.7798890Z + [[ distributed == *xla* ]] 2024-04-03T16:06:46.7799880Z + [[ distributed == *executorch* ]] 2024-04-03T16:06:46.7800558Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-04-03T16:06:46.7801530Z + [[ linux-jammy-py3.8-gcc11 == *libtorch* ]] 2024-04-03T16:06:46.7802584Z + [[ distributed == distributed ]] 2024-04-03T16:06:46.7803285Z + test_distributed 2024-04-03T16:06:46.7804000Z + echo 'Testing distributed python tests' 2024-04-03T16:06:46.7804792Z Testing distributed python tests 2024-04-03T16:06:46.7805918Z + python test/run_test.py --distributed-tests --shard 1 2 --verbose 2024-04-03T16:06:46.8635041Z test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-04-03T16:06:46.8637180Z import pkg_resources 2024-04-03T16:06:49.1723158Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-04-03T16:06:49.2559452Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-04-03T16:06:49.3285098Z Ignoring disabled issues: [''] 2024-04-03T16:06:49.3353007Z Found test times from artifacts 2024-04-03T16:06:49.3611736Z Found test times from artifacts 2024-04-03T16:06:49.3621195Z Running all tests 2024-04-03T16:06:49.3747641Z Name: tests to run (est. time: 55.07min) 2024-04-03T16:06:49.3748303Z Serial tests (98): 2024-04-03T16:06:49.3748992Z distributed/_tensor/debug/test_op_coverage 1/1 2024-04-03T16:06:49.3749804Z distributed/_spmd/test_graph_utils 1/1 2024-04-03T16:06:49.3750594Z distributed/_tensor/test_dtensor 1/1 2024-04-03T16:06:49.3751433Z distributed/_spmd/test_data_parallel 1/1 2024-04-03T16:06:49.3752312Z distributed/test_distributed_spawn 2/3 2024-04-03T16:06:49.3753171Z distributed/checkpoint/test_tp_checkpoint 1/1 2024-04-03T16:06:49.3754058Z distributed/_tensor/test_tensor_ops 1/1 2024-04-03T16:06:49.3754924Z distributed/_composable/fsdp/test_fully_shard_init 1/1 2024-04-03T16:06:49.3756040Z distributed/_tensor/test_xla_integration 1/1 2024-04-03T16:06:49.3756937Z distributed/checkpoint/test_format_utils 1/1 2024-04-03T16:06:49.3757428Z distributed/test_dynamo_distributed 1/1 2024-04-03T16:06:49.3757928Z distributed/checkpoint/test_fsdp_optim_state 1/1 2024-04-03T16:06:49.3758410Z distributed/_tensor/test_api 1/1 2024-04-03T16:06:49.3758822Z distributed/_tensor/test_random_ops 1/1 2024-04-03T16:06:49.3759255Z distributed/test_device_mesh 1/1 2024-04-03T16:06:49.3759729Z distributed/checkpoint/test_dtensor_resharding 1/1 2024-04-03T16:06:49.3760290Z distributed/_tensor/experimental/test_tp_transform 1/1 2024-04-03T16:06:49.3760894Z distributed/_composable/fsdp/test_fully_shard_autograd 1/1 2024-04-03T16:06:49.3761453Z distributed/test_compute_comm_reordering 1/1 2024-04-03T16:06:49.3762028Z distributed/tensor/parallel/test_ddp_2d_parallel 1/1 2024-04-03T16:06:49.3762527Z distributed/checkpoint/test_fsspec 1/1 2024-04-03T16:06:49.3763095Z distributed/_tensor/test_experimental_ops 1/1 2024-04-03T16:06:49.3763571Z distributed/fsdp/test_fsdp_memory 1/1 2024-04-03T16:06:49.3764022Z distributed/_tensor/test_convolution_ops 1/1 2024-04-03T16:06:49.3764510Z distributed/_tools/test_memory_tracker 1/1 2024-04-03T16:06:49.3764986Z distributed/_tensor/test_pointwise_ops 1/1 2024-04-03T16:06:49.3765460Z distributed/tensor/parallel/test_tp_style 1/1 2024-04-03T16:06:49.3765988Z distributed/checkpoint/test_dtensor_checkpoint 1/1 2024-04-03T16:06:49.3766510Z distributed/optim/test_named_optimizer 1/1 2024-04-03T16:06:49.3766968Z distributed/_tensor/test_op_strategy 1/1 2024-04-03T16:06:49.3767508Z distributed/_composable/fsdp/test_fully_shard_memory 1/1 2024-04-03T16:06:49.3768059Z distributed/checkpoint/e2e/test_pipeline 1/1 2024-04-03T16:06:49.3768538Z distributed/_tensor/test_embedding_ops 1/1 2024-04-03T16:06:49.3769055Z distributed/checkpoint/e2e/test_e2e_save_and_load 1/1 2024-04-03T16:06:49.3769917Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 2024-04-03T16:06:49.3770572Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 2024-04-03T16:06:49.3771408Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 1/1 2024-04-03T16:06:49.3772019Z distributed/fsdp/test_shard_utils 1/1 2024-04-03T16:06:49.3772528Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-04-03T16:06:49.3773078Z distributed/_composable/test_compose 1/1 2024-04-03T16:06:49.3773520Z distributed/test_fake_pg 1/1 2024-04-03T16:06:49.3773983Z distributed/tensor/parallel/test_parallelize_api 1/1 2024-04-03T16:06:49.3774579Z distributed/_composable/fully_shard/test_fully_shard_util 1/1 2024-04-03T16:06:49.3775253Z distributed/test_inductor_collectives 1/1 2024-04-03T16:06:49.3775974Z distributed/fsdp/test_fsdp_backward_prefetch 1/1 2024-04-03T16:06:49.3776480Z distributed/_composable/test_replicate 1/1 2024-04-03T16:06:49.3777023Z distributed/_shard/sharding_plan/test_sharding_plan 1/1 2024-04-03T16:06:49.3777668Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 2024-04-03T16:06:49.3778343Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 2024-04-03T16:06:49.3778909Z distributed/_tensor/test_optimizers 1/1 2024-04-03T16:06:49.3779372Z distributed/fsdp/test_fsdp_pure_fp16 1/1 2024-04-03T16:06:49.3779835Z distributed/fsdp/test_fsdp_optim_state 1/1 2024-04-03T16:06:49.3780286Z distributed/_shard/test_sharder 1/1 2024-04-03T16:06:49.3780716Z distributed/fsdp/test_fsdp_overlap 1/1 2024-04-03T16:06:49.3781138Z distributed/test_c10d_logger 1/1 2024-04-03T16:06:49.3781631Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 2024-04-03T16:06:49.3782201Z distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 2024-04-03T16:06:49.3782705Z distributed/checkpoint/test_checkpoint 1/1 2024-04-03T16:06:49.3783173Z distributed/test_multi_threaded_pg 1/1 2024-04-03T16:06:49.3783637Z distributed/nn/jit/test_instantiator 1/1 2024-04-03T16:06:49.3784151Z distributed/checkpoint/test_file_system_checkpoint 1/1 2024-04-03T16:06:49.3784670Z distributed/rpc/test_share_memory 1/1 2024-04-03T16:06:49.3785142Z distributed/fsdp/test_fsdp_freezing_weights 1/1 2024-04-03T16:06:49.3785628Z distributed/pipeline/sync/test_pipe 1/1 2024-04-03T16:06:49.3786093Z distributed/fsdp/test_fsdp_fine_tune 1/1 2024-04-03T16:06:49.3786540Z distributed/fsdp/test_fsdp_core 1/1 2024-04-03T16:06:49.3786949Z distributed/fsdp/test_fsdp_meta 1/1 2024-04-03T16:06:49.3787448Z distributed/optim/test_apply_optimizer_in_backward 1/1 2024-04-03T16:06:49.3787988Z distributed/fsdp/test_fsdp_checkpoint 1/1 2024-04-03T16:06:49.3788456Z distributed/pipeline/sync/test_worker 1/1 2024-04-03T16:06:49.3788953Z distributed/pipeline/sync/test_transparency 1/1 2024-04-03T16:06:49.3789609Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 1/1 2024-04-03T16:06:49.3790259Z distributed/fsdp/test_fsdp_unshard_params 1/1 2024-04-03T16:06:49.3790715Z distributed/fsdp/test_fsdp_misc 1/1 2024-04-03T16:06:49.3791181Z distributed/fsdp/test_fsdp_hybrid_shard 1/1 2024-04-03T16:06:49.3791675Z distributed/pipeline/sync/test_microbatch 1/1 2024-04-03T16:06:49.3792205Z distributed/pipeline/sync/test_checkpoint 1/1 2024-04-03T16:06:49.3793001Z distributed/rpc/test_faulty_agent 1/1 2024-04-03T16:06:49.3793742Z distributed/pipeline/sync/skip/test_portal 1/1 2024-04-03T16:06:49.3794533Z distributed/_shard/sharded_tensor/ops/test_init 1/1 2024-04-03T16:06:49.3795268Z distributed/rpc/test_tensorpipe_agent 1/3 2024-04-03T16:06:49.3796109Z distributed/rpc/test_tensorpipe_agent 3/3 2024-04-03T16:06:49.3796717Z distributed/test_nccl 1/1 2024-04-03T16:06:49.3797389Z distributed/pipeline/sync/skip/test_verify_skippables 1/1 2024-04-03T16:06:49.3798162Z distributed/test_data_parallel 1/1 2024-04-03T16:06:49.3798726Z distributed/test_store 1/1 2024-04-03T16:06:49.3799642Z distributed/pipeline/sync/skip/test_inspect_skip_layout 1/1 2024-04-03T16:06:49.3800565Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 2024-04-03T16:06:49.3801501Z distributed/test_launcher 1/1 2024-04-03T16:06:49.3802253Z distributed/optim/test_zero_redundancy_optimizer 1/1 2024-04-03T16:06:49.3803240Z distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 2024-04-03T16:06:49.3804065Z distributed/elastic/events/lib_test 1/1 2024-04-03T16:06:49.3804836Z distributed/elastic/metrics/api_test 1/1 2024-04-03T16:06:49.3805660Z distributed/elastic/multiprocessing/api_test 1/1 2024-04-03T16:06:49.3806449Z distributed/elastic/timer/api_test 1/1 2024-04-03T16:06:49.3807299Z distributed/elastic/timer/local_timer_example 1/1 2024-04-03T16:06:49.3808417Z distributed/elastic/timer/local_timer_test 1/1 2024-04-03T16:06:49.3809325Z distributed/elastic/utils/distributed_test 1/1 2024-04-03T16:06:49.3810132Z Parallel tests (0): 2024-04-03T16:06:49.3810708Z Name: excluded (est. time: 0.0min) 2024-04-03T16:06:49.3811353Z Serial tests (0): 2024-04-03T16:06:49.3811882Z Parallel tests (0): 2024-04-03T16:06:49.3812913Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-04-03T16:06:49.3883307Z Running distributed/_tensor/debug/test_op_coverage 1/1 ... [2024-04-03 16:06:49.387851] 2024-04-03T16:06:49.3889380Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/debug/test_op_coverage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:06:49.388543] 2024-04-03T16:06:53.0130733Z 2024-04-03T16:06:53.0132796Z distributed/_tensor/debug/test_op_coverage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_fa314eb63de0fd16_.log 2024-04-03T16:06:53.0134637Z Running 1 items in this shard: test/distributed/_tensor/debug/test_op_coverage.py::TestOpCoverage::test_trace_with_inductor_decomp 2024-04-03T16:06:53.0135404Z 2024-04-03T16:06:53.0139446Z Running distributed/_spmd/test_graph_utils 1/1 ... [2024-04-03 16:06:53.013715] 2024-04-03T16:06:53.0144168Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_spmd/test_graph_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:06:53.014111] 2024-04-03T16:06:55.1776218Z 2024-04-03T16:06:55.1778078Z distributed/_spmd/test_graph_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._spmd.test_graph_utils_1.1_5d5d3bef216be4c0_.log 2024-04-03T16:06:55.1779135Z 2024-04-03T16:06:55.1782920Z Running distributed/_tensor/test_dtensor 1/1 ... [2024-04-03 16:06:55.178066] 2024-04-03T16:06:55.1787705Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_dtensor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:06:55.178490] 2024-04-03T16:09:53.1767118Z 2024-04-03T16:09:53.1768798Z distributed/_tensor/test_dtensor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_1.1_33e020d2158f5785_.log 2024-04-03T16:09:53.1804550Z Running 57 items in this shard: test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_async_output_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_async_output_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_constructor_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_constructor_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_new_empty_strided_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_new_empty_strided_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_properties_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_properties_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_save_load_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_save_load_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_hash_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_hash_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_read_only_after_set_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_read_only_after_set_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_stride_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_stride_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_negative_dim_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_negative_dim_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_then_to_local_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_then_to_local_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding_raise_error_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding_raise_error_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_grad_hint_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_grad_hint_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_sync_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_sync_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_meta_dtensor_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_meta_dtensor_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_modules_w_meta_dtensor_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_modules_w_meta_dtensor_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local_grad_hint_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local_grad_hint_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_default_value_sub_mesh_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_default_value_sub_mesh_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_device_mesh_nd_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_device_mesh_nd_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_2d_mesh_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_2d_mesh_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_api_device_mesh_context_manager_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_api_device_mesh_context_manager_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_device_mesh_device_conversion_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_device_mesh_device_conversion_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_spec_local_shard_offset_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_spec_local_shard_offset_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_from_local_sub_mesh_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_from_local_sub_mesh_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_implicit_replication_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_implicit_replication_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_redistribute_sub_mesh_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_redistribute_sub_mesh_use_native_funcol_True, test/distributed/_tensor/test_dtensor.py::TestDTensorPlacementTypes::test_split_tensor_1D_use_native_funcol_False, test/distributed/_tensor/test_dtensor.py::TestDTensorPlacementTypes::test_split_tensor_1D_use_native_funcol_True 2024-04-03T16:09:53.1860312Z 2024-04-03T16:09:53.1861188Z Running distributed/_spmd/test_data_parallel 1/1 ... [2024-04-03 16:09:53.177539] 2024-04-03T16:09:53.1864732Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_spmd/test_data_parallel.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:09:53.177964] 2024-04-03T16:09:55.4567917Z 2024-04-03T16:09:55.4569554Z distributed/_spmd/test_data_parallel 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._spmd.test_data_parallel_1.1_789eae00e80cee81_.log 2024-04-03T16:09:55.4570664Z 2024-04-03T16:09:55.4574187Z Running distributed/test_distributed_spawn 2/3 ... [2024-04-03 16:09:55.457213] 2024-04-03T16:09:55.4622034Z MPI not available -- MPI backend tests will be skipped 2024-04-03T16:09:55.4623700Z Running distributed tests for the test backend with env init_method 2024-04-03T16:09:55.4645528Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:09:55.464214] 2024-04-03T16:09:58.0534613Z 2024-04-03T16:09:58.0536272Z distributed/test_distributed_spawn 2/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.3_4e662b051c6bb258_.log 2024-04-03T16:09:58.0537472Z Running 0 items in this shard: 2024-04-03T16:09:58.0537742Z 2024-04-03T16:09:58.0543115Z Running distributed tests for the test backend with file init_method 2024-04-03T16:09:58.0548920Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:09:58.054601] 2024-04-03T16:10:00.5828855Z 2024-04-03T16:10:00.5831166Z distributed/test_distributed_spawn 2/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.3_1f5cef3fdf6ecb33_.log 2024-04-03T16:10:00.5832628Z Running 0 items in this shard: 2024-04-03T16:10:00.5832877Z 2024-04-03T16:10:00.5839053Z Running distributed tests for the gloo backend with env init_method 2024-04-03T16:10:00.5844885Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:10:00.584116] 2024-04-03T16:17:25.3083507Z 2024-04-03T16:17:25.3085037Z distributed/test_distributed_spawn 2/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.3_08865f72cafa1d62_.log 2024-04-03T16:17:25.3145522Z Running 90 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2024-04-03T16:17:25.3204449Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2024-04-03T16:17:25.3206045Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2024-04-03T16:17:25.3207753Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2024-04-03T16:17:25.3209446Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2024-04-03T16:17:25.3210963Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2024-04-03T16:17:25.3212403Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2024-04-03T16:17:25.3213962Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2024-04-03T16:17:25.3215530Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2024-04-03T16:17:25.3216976Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2024-04-03T16:17:25.3218416Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2024-04-03T16:17:25.3219815Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2024-04-03T16:17:25.3221160Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2024-04-03T16:17:25.3222508Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2024-04-03T16:17:25.3224071Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2024-04-03T16:17:25.3225550Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2024-04-03T16:17:25.3227074Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2024-04-03T16:17:25.3228506Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2024-04-03T16:17:25.3229870Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2024-04-03T16:17:25.3231222Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2024-04-03T16:17:25.3232510Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2024-04-03T16:17:25.3233796Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2024-04-03T16:17:25.3235112Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2024-04-03T16:17:25.3237438Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2024-04-03T16:17:25.3239395Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2024-04-03T16:17:25.3240883Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2024-04-03T16:17:25.3242376Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2024-04-03T16:17:25.3244020Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2024-04-03T16:17:25.3245621Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2024-04-03T16:17:25.3247192Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2024-04-03T16:17:25.3248588Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group 2024-04-03T16:17:25.3249867Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2024-04-03T16:17:25.3251175Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2024-04-03T16:17:25.3252551Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2024-04-03T16:17:25.3253936Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2024-04-03T16:17:25.3255258Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2024-04-03T16:17:25.3256555Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2024-04-03T16:17:25.3257903Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2024-04-03T16:17:25.3259399Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2024-04-03T16:17:25.3261050Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2024-04-03T16:17:25.3262507Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2024-04-03T16:17:25.3264042Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2024-04-03T16:17:25.3265450Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2024-04-03T16:17:25.3266883Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2024-04-03T16:17:25.3268246Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2024-04-03T16:17:25.3269594Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2024-04-03T16:17:25.3270998Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2024-04-03T16:17:25.3272339Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2024-04-03T16:17:25.3273687Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2024-04-03T16:17:25.3275210Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2024-04-03T16:17:25.3277102Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2024-04-03T16:17:25.3278901Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2024-04-03T16:17:25.3280544Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2024-04-03T16:17:25.3281991Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2024-04-03T16:17:25.3283481Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2024-04-03T16:17:25.3284906Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2024-04-03T16:17:25.3286412Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2024-04-03T16:17:25.3287888Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2024-04-03T16:17:25.3289251Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2024-04-03T16:17:25.3290558Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2024-04-03T16:17:25.3291923Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2024-04-03T16:17:25.3293340Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2024-04-03T16:17:25.3294673Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2024-04-03T16:17:25.3295987Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2024-04-03T16:17:25.3297412Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2024-04-03T16:17:25.3298732Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2024-04-03T16:17:25.3300107Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2024-04-03T16:17:25.3301375Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2024-04-03T16:17:25.3302854Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2024-04-03T16:17:25.3304419Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2024-04-03T16:17:25.3305898Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2024-04-03T16:17:25.3307330Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2024-04-03T16:17:25.3308918Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2024-04-03T16:17:25.3310533Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2024-04-03T16:17:25.3312014Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2024-04-03T16:17:25.3313449Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2024-04-03T16:17:25.3314965Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2024-04-03T16:17:25.3316712Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2024-04-03T16:17:25.3318155Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2024-04-03T16:17:25.3319514Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2024-04-03T16:17:25.3320822Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2024-04-03T16:17:25.3322091Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2024-04-03T16:17:25.3323454Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2024-04-03T16:17:25.3324787Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2024-04-03T16:17:25.3326064Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2024-04-03T16:17:25.3327373Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2024-04-03T16:17:25.3328800Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2024-04-03T16:17:25.3330289Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2024-04-03T16:17:25.3331666Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2024-04-03T16:17:25.3333168Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2024-04-03T16:17:25.3334507Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2024-04-03T16:17:25.3335299Z 2024-04-03T16:17:25.3335594Z Running distributed tests for the gloo backend with file init_method 2024-04-03T16:17:25.3337663Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:17:25.312210] 2024-04-03T16:24:54.3581360Z 2024-04-03T16:24:54.3584453Z distributed/test_distributed_spawn 2/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.3_fd23c646647f4173_.log 2024-04-03T16:24:54.3645318Z Running 90 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2024-04-03T16:24:54.3711052Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2024-04-03T16:24:54.3712630Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2024-04-03T16:24:54.3714317Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2024-04-03T16:24:54.3716165Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2024-04-03T16:24:54.3717685Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2024-04-03T16:24:54.3719134Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2024-04-03T16:24:54.3720708Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2024-04-03T16:24:54.3722275Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2024-04-03T16:24:54.3723784Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2024-04-03T16:24:54.3725229Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2024-04-03T16:24:54.3726632Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2024-04-03T16:24:54.3728183Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2024-04-03T16:24:54.3729535Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2024-04-03T16:24:54.3731011Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2024-04-03T16:24:54.3732477Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2024-04-03T16:24:54.3733907Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2024-04-03T16:24:54.3735282Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2024-04-03T16:24:54.3736653Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2024-04-03T16:24:54.3738005Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2024-04-03T16:24:54.3739288Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2024-04-03T16:24:54.3740571Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2024-04-03T16:24:54.3741876Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2024-04-03T16:24:54.3743279Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2024-04-03T16:24:54.3744767Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2024-04-03T16:24:54.3746240Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2024-04-03T16:24:54.3747747Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2024-04-03T16:24:54.3749300Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2024-04-03T16:24:54.3750886Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2024-04-03T16:24:54.3752450Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2024-04-03T16:24:54.3753846Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_group 2024-04-03T16:24:54.3755111Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2024-04-03T16:24:54.3756537Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2024-04-03T16:24:54.3757902Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2024-04-03T16:24:54.3759280Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2024-04-03T16:24:54.3760592Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2024-04-03T16:24:54.3761887Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2024-04-03T16:24:54.3763409Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2024-04-03T16:24:54.3764916Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2024-04-03T16:24:54.3766437Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2024-04-03T16:24:54.3767885Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2024-04-03T16:24:54.3769328Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2024-04-03T16:24:54.3770740Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2024-04-03T16:24:54.3772152Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2024-04-03T16:24:54.3773524Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2024-04-03T16:24:54.3774885Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2024-04-03T16:24:54.3776287Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2024-04-03T16:24:54.3777597Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2024-04-03T16:24:54.3778934Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2024-04-03T16:24:54.3780469Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2024-04-03T16:24:54.3782187Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2024-04-03T16:24:54.3783967Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2024-04-03T16:24:54.3785614Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2024-04-03T16:24:54.3787068Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2024-04-03T16:24:54.3788502Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2024-04-03T16:24:54.3789904Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2024-04-03T16:24:54.3791404Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2024-04-03T16:24:54.3792897Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2024-04-03T16:24:54.3794263Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2024-04-03T16:24:54.3795660Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2024-04-03T16:24:54.3797031Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2024-04-03T16:24:54.3798465Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2024-04-03T16:24:54.3799892Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2024-04-03T16:24:54.3801213Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2024-04-03T16:24:54.3802684Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2024-04-03T16:24:54.3804011Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2024-04-03T16:24:54.3817653Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2024-04-03T16:24:54.3818951Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2024-04-03T16:24:54.3820443Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2024-04-03T16:24:54.3822038Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2024-04-03T16:24:54.3823532Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2024-04-03T16:24:54.3824945Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2024-04-03T16:24:54.3826536Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2024-04-03T16:24:54.3828165Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2024-04-03T16:24:54.3829648Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2024-04-03T16:24:54.3831073Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2024-04-03T16:24:54.3832607Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2024-04-03T16:24:54.3834204Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2024-04-03T16:24:54.3835764Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2024-04-03T16:24:54.3837107Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2024-04-03T16:24:54.3838453Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2024-04-03T16:24:54.3839735Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2024-04-03T16:24:54.3841036Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2024-04-03T16:24:54.3842375Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2024-04-03T16:24:54.3843751Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2024-04-03T16:24:54.3845072Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2024-04-03T16:24:54.3846508Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2024-04-03T16:24:54.3848167Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2024-04-03T16:24:54.3849565Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2024-04-03T16:24:54.3851026Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2024-04-03T16:24:54.3852354Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2024-04-03T16:24:54.3853073Z 2024-04-03T16:24:54.3853631Z Running distributed/checkpoint/test_tp_checkpoint 1/1 ... [2024-04-03 16:24:54.363809] 2024-04-03T16:24:54.3855660Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_tp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:24:54.364151] 2024-04-03T16:25:02.2975433Z 2024-04-03T16:25:02.2977463Z distributed/checkpoint/test_tp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_tp_checkpoint_1.1_c032f8f6c8cb2683_.log 2024-04-03T16:25:02.2979844Z Running 2 items in this shard: test/distributed/checkpoint/test_tp_checkpoint.py::TestTpCheckpoint::test_tp_checkpoint, test/distributed/checkpoint/test_tp_checkpoint.py::TestTpCheckpoint::test_tp_checkpoint_load_on_meta_device 2024-04-03T16:25:02.2981540Z 2024-04-03T16:25:02.2981967Z Running distributed/_tensor/test_tensor_ops 1/1 ... [2024-04-03 16:25:02.297958] 2024-04-03T16:25:02.2986841Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_tensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:25:02.298376] 2024-04-03T16:26:08.1068600Z 2024-04-03T16:26:08.1071905Z distributed/_tensor/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_tensor_ops_1.1_1cac935f4887e855_.log 2024-04-03T16:26:08.1084000Z Running 22 items in this shard: test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_aten_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_clone, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_detach, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_dtensor_dtype_conversion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_empty_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_equal, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_full_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_gather, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_index, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_inplace_op, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_full, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_op_out_variant, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_stack, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_where_type_promotion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zero_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like_partial_sum 2024-04-03T16:26:08.1093914Z 2024-04-03T16:26:08.1094669Z Running distributed/_composable/fsdp/test_fully_shard_init 1/1 ... [2024-04-03 16:26:08.107359] 2024-04-03T16:26:08.1096913Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:26:08.107807] 2024-04-03T16:26:10.8801969Z 2024-04-03T16:26:10.8803723Z distributed/_composable/fsdp/test_fully_shard_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_init_1.1_bf64ce7a055b8394_.log 2024-04-03T16:26:10.8827500Z Running 20 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardDeviceTensor::test_move_states_to_device_tensor, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardDeviceDTensor::test_move_states_to_device_dtensor_invalid, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardDeviceDTensor::test_move_states_to_device_dtensor_valid, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMeshArg::test_invalid_mesh_ndim, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_duplicate, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_nested, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_nested_fully_shard_and_replicate, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_single, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_states_nested_fully_shard, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_states_shared_params_and_buffers, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardParamModuleInfos::test_get_param_module_infos_duplicates, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardParamModuleInfos::test_get_param_module_infos_shared_params, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardedParameterTensor::test_shard_tensor_parameters, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardedParameterDTensor::test_shard_dtensor_parameters, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_fully_shard_double_lazy_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_fully_shard_is_root, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_fully_shard_module_and_param_fqns, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMetaDeviceInit::test_invalid_meta_device_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMetaDeviceInit::test_meta_device_1d_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMetaDeviceInit::test_meta_device_2d_init 2024-04-03T16:26:10.8847348Z 2024-04-03T16:26:10.8847889Z Running distributed/_tensor/test_xla_integration 1/1 ... [2024-04-03 16:26:10.880744] 2024-04-03T16:26:10.8849872Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_xla_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:26:10.881280] 2024-04-03T16:26:13.5539863Z 2024-04-03T16:26:13.5542293Z distributed/_tensor/test_xla_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_xla_integration_1.1_36479aca781c3e15_.log 2024-04-03T16:26:13.5545705Z Running 3 items in this shard: test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_replicate, test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_shard, test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_2d 2024-04-03T16:26:13.5548502Z 2024-04-03T16:26:13.5548988Z Running distributed/checkpoint/test_format_utils 1/1 ... [2024-04-03 16:26:13.554531] 2024-04-03T16:26:13.5553057Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_format_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:26:13.554951] 2024-04-03T16:26:23.9940285Z 2024-04-03T16:26:23.9942324Z distributed/checkpoint/test_format_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_format_utils_1.1_a6010019ccb53782_.log 2024-04-03T16:26:23.9945232Z Running 3 items in this shard: test/distributed/checkpoint/test_format_utils.py::TestFormatUtils::test_dcp_to_torch_save, test/distributed/checkpoint/test_format_utils.py::TestFormatUtils::test_online_torch_save_to_dcp, test/distributed/checkpoint/test_format_utils.py::TestFormatUtils::test_torch_save_to_dcp 2024-04-03T16:26:23.9947201Z 2024-04-03T16:26:23.9947634Z Running distributed/test_dynamo_distributed 1/1 ... [2024-04-03 16:26:23.994423] 2024-04-03T16:26:23.9951528Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:26:23.994854] 2024-04-03T16:26:40.2901651Z 2024-04-03T16:26:40.2903453Z distributed/test_dynamo_distributed 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_dynamo_distributed_1.1_d88d9e2c9d7eb77b_.log 2024-04-03T16:26:40.2924447Z Running 37 items in this shard: test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_call_method_forward, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_issue90375, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_symbol_splitting, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_baseline_aot_eager_multiprocess, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_aot_autograd, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_async_subclass_no_specialize, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_custom_layer, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_aot_eager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_empty_graph_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_diff_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_same_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_orig_params_assert, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_guards, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_register_attr_or_module, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_staticmethod, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_ctx_manager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_inference, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_training, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_transpose, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_higher_order_op, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ignored_parameters, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_no_split 2024-04-03T16:26:40.2943311Z 2024-04-03T16:26:40.2944110Z Running distributed/checkpoint/test_fsdp_optim_state 1/1 ... [2024-04-03 16:26:40.290902] 2024-04-03T16:26:40.2947097Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_optim_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:26:40.291428] 2024-04-03T16:26:48.1755992Z 2024-04-03T16:26:48.1757933Z distributed/checkpoint/test_fsdp_optim_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_optim_state_1.1_227ae436e16e9c78_.log 2024-04-03T16:26:48.1761163Z Running 2 items in this shard: test/distributed/checkpoint/test_fsdp_optim_state.py::FsdpOptimStateCheckpoint::test_load_sharded_optimizer_state_dict_pass_planner_False, test/distributed/checkpoint/test_fsdp_optim_state.py::FsdpOptimStateCheckpoint::test_load_sharded_optimizer_state_dict_pass_planner_True 2024-04-03T16:26:48.1763536Z 2024-04-03T16:26:48.1763937Z Running distributed/_tensor/test_api 1/1 ... [2024-04-03 16:26:48.176009] 2024-04-03T16:26:48.1767740Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:26:48.176438] 2024-04-03T16:27:11.3990706Z 2024-04-03T16:27:11.3993628Z distributed/_tensor/test_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_api_1.1_f7565b272946c558_.log 2024-04-03T16:27:11.3998961Z Running 8 items in this shard: test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_casting, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn_warning, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_meta, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor_errors, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor_uneven_sharding 2024-04-03T16:27:11.4003048Z 2024-04-03T16:27:11.4003700Z Running distributed/_tensor/test_random_ops 1/1 ... [2024-04-03 16:27:11.399565] 2024-04-03T16:27:11.4005655Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_random_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:27:11.400002] 2024-04-03T16:27:31.9095917Z 2024-04-03T16:27:31.9097562Z distributed/_tensor/test_random_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_random_ops_1.1_27332ff31679919e_.log 2024-04-03T16:27:31.9102726Z Running 7 items in this shard: test/distributed/_tensor/test_random_ops.py::DistTensorRandomInitTest::test_init_ops, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_dropout_1d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_rand_1d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_uniform_2d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_manual_seed, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_meta_tensor_init, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_rng_tracker_init 2024-04-03T16:27:31.9106857Z 2024-04-03T16:27:31.9107242Z Running distributed/test_device_mesh 1/1 ... [2024-04-03 16:27:31.910117] 2024-04-03T16:27:31.9109106Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_device_mesh.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:27:31.910524] 2024-04-03T16:30:21.8829375Z 2024-04-03T16:30:21.8830877Z distributed/test_device_mesh 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_device_mesh_1.1_041398b27a05e6ca_.log 2024-04-03T16:30:21.8863892Z Running 56 items in this shard: test/distributed/test_device_mesh.py::DeviceMeshTest::test_assert_invalid_mesh_tensor, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_2d_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_2d_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshTest::test_fake_pg_device_mesh_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshTest::test_fake_pg_device_mesh_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_group_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_group_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank_raises_exception_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank_raises_exception_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshTest::test_init_process_group_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshTest::test_init_process_group_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_nd, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_init_device_mesh, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_duplicate_mesh_dim_names, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_mesh_shape_mesh_dim_names_mismatch, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_cache_and_reuse_submesh_slice_result_use_native_funcol_False, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_cache_and_reuse_submesh_slice_result_use_native_funcol_True, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_1d_use_native_funcol_False, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_1d_use_native_funcol_True, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_use_native_funcol_False, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_use_native_funcol_True, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_invalid_mesh_dim_name_use_native_funcol_False, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_invalid_mesh_dim_name_use_native_funcol_True, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_no_mesh_dim_found_use_native_funcol_False, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_no_mesh_dim_found_use_native_funcol_True, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_mesh_dim_by_name_use_native_funcol_False, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_mesh_dim_by_name_use_native_funcol_True, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_parent_mesh_dim_exist_use_native_funcol_False, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_parent_mesh_dim_exist_use_native_funcol_True, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_parent_mesh_dim_not_exist_use_native_funcol_False, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_parent_mesh_dim_not_exist_use_native_funcol_True, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_parent_mesh_use_native_funcol_False, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_parent_mesh_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_gather_uneven_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_gather_uneven_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_to_all_1d_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_to_all_1d_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_to_all_nd_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_to_all_nd_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_1d_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_1d_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_nd_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_nd_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_contiguous_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_contiguous_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_uneven_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_uneven_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_1d_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_1d_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_nd_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_nd_use_native_funcol_True, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_uneven_use_native_funcol_False, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_uneven_use_native_funcol_True 2024-04-03T16:30:21.8894716Z 2024-04-03T16:30:21.8895272Z Running distributed/checkpoint/test_dtensor_resharding 1/1 ... [2024-04-03 16:30:21.883670] 2024-04-03T16:30:21.8897436Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_dtensor_resharding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:30:21.884082] 2024-04-03T16:30:37.7347653Z 2024-04-03T16:30:37.7350467Z distributed/checkpoint/test_dtensor_resharding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dtensor_resharding_1.1_9d8ede7695361473_.log 2024-04-03T16:30:37.7359304Z Running 5 items in this shard: test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_1d_to_1d_reshard_placement_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_2d_to_2d_reshard_placement_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_1d_to_2d_reshard_mesh_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_2d_to_1d_reshard_mesh_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_dtensor_checkpoint_resharding_with_empty_shard 2024-04-03T16:30:37.7365550Z 2024-04-03T16:30:37.7366339Z Running distributed/_tensor/experimental/test_tp_transform 1/1 ... [2024-04-03 16:30:37.735442] 2024-04-03T16:30:37.7369377Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/experimental/test_tp_transform.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:30:37.736005] 2024-04-03T16:30:57.1437406Z 2024-04-03T16:30:57.1440285Z distributed/_tensor/experimental/test_tp_transform 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.experimental.test_tp_transform_1.1_f38eb6c192607b12_.log 2024-04-03T16:30:57.1448193Z Running 6 items in this shard: test/distributed/_tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_e2e_use_native_funcol_False, test/distributed/_tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_e2e_use_native_funcol_True, test/distributed/_tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_no_bias_use_native_funcol_False, test/distributed/_tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_no_bias_use_native_funcol_True, test/distributed/_tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_with_uncovered_op_use_native_funcol_False, test/distributed/_tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_with_uncovered_op_use_native_funcol_True 2024-04-03T16:30:57.1452636Z 2024-04-03T16:30:57.1453208Z Running distributed/_composable/fsdp/test_fully_shard_autograd 1/1 ... [2024-04-03 16:30:57.144158] 2024-04-03T16:30:57.1455349Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_autograd.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:30:57.144619] 2024-04-03T16:30:59.9176516Z 2024-04-03T16:30:59.9178458Z distributed/_composable/fsdp/test_fully_shard_autograd 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_autograd_1.1_f8665121d4d73bf0_.log 2024-04-03T16:30:59.9182193Z Running 3 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardAutograd::test_nontensor_activations, test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardAutograd::test_unused_forward_module, test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardAutograd::test_unused_forward_output 2024-04-03T16:30:59.9184518Z 2024-04-03T16:30:59.9185284Z Running distributed/test_compute_comm_reordering 1/1 ... [2024-04-03 16:30:59.918103] 2024-04-03T16:30:59.9188075Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_compute_comm_reordering.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:30:59.918524] 2024-04-03T16:31:02.3902712Z 2024-04-03T16:31:02.3904554Z distributed/test_compute_comm_reordering 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_compute_comm_reordering_1.1_ca74324cfcf30973_.log 2024-04-03T16:31:02.3909756Z Running 6 items in this shard: test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_nccl_heuristics, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_raise_comms, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap_custom_runtime_estimation, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits_raise_comms 2024-04-03T16:31:02.3914050Z 2024-04-03T16:31:02.3914564Z Running distributed/tensor/parallel/test_ddp_2d_parallel 1/1 ... [2024-04-03 16:31:02.390709] 2024-04-03T16:31:02.3916844Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/tensor/parallel/test_ddp_2d_parallel.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:31:02.391151] 2024-04-03T16:31:07.7187820Z 2024-04-03T16:31:07.7189775Z distributed/tensor/parallel/test_ddp_2d_parallel 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_ddp_2d_parallel_1.1_cd2542b98e34b99f_.log 2024-04-03T16:31:07.7191839Z Running 1 items in this shard: test/distributed/tensor/parallel/test_ddp_2d_parallel.py::Test2dParallelIntegration::test_2d_ddp_integration_functionality 2024-04-03T16:31:07.7192953Z 2024-04-03T16:31:07.7193642Z Running distributed/checkpoint/test_fsspec 1/1 ... [2024-04-03 16:31:07.719160] 2024-04-03T16:31:07.7199248Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_fsspec.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:31:07.719611] 2024-04-03T16:31:10.5427220Z 2024-04-03T16:31:10.5428856Z distributed/checkpoint/test_fsspec 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsspec_1.1_8646094d777e2c5b_.log 2024-04-03T16:31:10.5431000Z Running 2 items in this shard: test/distributed/checkpoint/test_fsspec.py::TestFSSpecNoDist::test_fsspec_no_dist, test/distributed/checkpoint/test_fsspec.py::TestFSSpecWithDist::test_fsspec_with_dist 2024-04-03T16:31:10.5432230Z 2024-04-03T16:31:10.5433750Z Running distributed/_tensor/test_experimental_ops 1/1 ... [2024-04-03 16:31:10.543151] 2024-04-03T16:31:10.5438855Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_experimental_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:31:10.543592] 2024-04-03T16:31:20.5278198Z 2024-04-03T16:31:20.5280171Z distributed/_tensor/test_experimental_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_experimental_ops_1.1_d083603fe615f616_.log 2024-04-03T16:31:20.5283018Z Running 3 items in this shard: test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_bernoulli, test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_nll, test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_slice 2024-04-03T16:31:20.5284827Z 2024-04-03T16:31:20.5285243Z Running distributed/fsdp/test_fsdp_memory 1/1 ... [2024-04-03 16:31:20.528297] 2024-04-03T16:31:20.5290030Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:31:20.528708] 2024-04-03T16:31:28.1601093Z 2024-04-03T16:31:28.1603469Z distributed/fsdp/test_fsdp_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_memory_1.1_ac72d4849852c254_.log 2024-04-03T16:31:28.1607138Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_memory.py::TestFSDPMemory::test_fsdp_memory_ckpt_ckpt, test/distributed/fsdp/test_fsdp_memory.py::TestFSDPMemory::test_fsdp_memory_ckpt_no_ckpt 2024-04-03T16:31:28.1609308Z 2024-04-03T16:31:28.1610088Z Running distributed/_tensor/test_convolution_ops 1/1 ... [2024-04-03 16:31:28.160584] 2024-04-03T16:31:28.1614132Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_convolution_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:31:28.161044] 2024-04-03T16:32:33.7134947Z 2024-04-03T16:32:33.7138157Z distributed/_tensor/test_convolution_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_convolution_ops_1.1_3f2f0898e35e09cf_.log 2024-04-03T16:32:33.7140542Z Running 2 items in this shard: test/distributed/_tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_depthwise_convolution, test/distributed/_tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_downsampling_convolution 2024-04-03T16:32:33.7141881Z 2024-04-03T16:32:33.7142298Z Running distributed/_tools/test_memory_tracker 1/1 ... [2024-04-03 16:32:33.712579] 2024-04-03T16:32:33.7159127Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tools/test_memory_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:32:33.715026] 2024-04-03T16:32:36.4379474Z 2024-04-03T16:32:36.4381211Z distributed/_tools/test_memory_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_memory_tracker_1.1_8b750102f351db5a_.log 2024-04-03T16:32:36.4382851Z Running 1 items in this shard: test/distributed/_tools/test_memory_tracker.py::TestMemoryTracker::test_local_model 2024-04-03T16:32:36.4383526Z 2024-04-03T16:32:36.4386387Z Running distributed/_tensor/test_pointwise_ops 1/1 ... [2024-04-03 16:32:36.438407] 2024-04-03T16:32:36.4391152Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_pointwise_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:32:36.438823] 2024-04-03T16:32:39.2611673Z 2024-04-03T16:32:39.2613617Z distributed/_tensor/test_pointwise_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_pointwise_ops_1.1_a6806cb85994eafd_.log 2024-04-03T16:32:39.2619012Z Running 7 items in this shard: test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_activations, test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_dropout, test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_dropout_backward, test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_dropout_errors, test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_mul_out, test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_partial_add, test/distributed/_tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_partial_mul 2024-04-03T16:32:39.2623195Z 2024-04-03T16:32:39.2623692Z Running distributed/tensor/parallel/test_tp_style 1/1 ... [2024-04-03 16:32:39.261607] 2024-04-03T16:32:39.2625733Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_style.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:32:39.261998] 2024-04-03T16:33:02.8282950Z 2024-04-03T16:33:02.8286075Z distributed/tensor/parallel/test_tp_style 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_style_1.1_11c8558d029f4ef3_.log 2024-04-03T16:33:02.8292270Z Running 8 items in this shard: test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input_multiple_inputs, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_output, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_sequence_parallel_style 2024-04-03T16:33:02.8297352Z 2024-04-03T16:33:02.8297843Z Running distributed/checkpoint/test_dtensor_checkpoint 1/1 ... [2024-04-03 16:33:02.828761] 2024-04-03T16:33:02.8299930Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_dtensor_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:33:02.829164] 2024-04-03T16:33:08.3572417Z 2024-04-03T16:33:08.3574303Z distributed/checkpoint/test_dtensor_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dtensor_checkpoint_1.1_e7c87f9e13dc86ef_.log 2024-04-03T16:33:08.3576203Z Running 1 items in this shard: test/distributed/checkpoint/test_dtensor_checkpoint.py::DTensorPlanner::test_distributed_tensor_planner 2024-04-03T16:33:08.3577033Z 2024-04-03T16:33:08.3579626Z Running distributed/optim/test_named_optimizer 1/1 ... [2024-04-03 16:33:08.357719] 2024-04-03T16:33:08.3584542Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/optim/test_named_optimizer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:33:08.358158] 2024-04-03T16:33:10.7201957Z 2024-04-03T16:33:10.7203621Z distributed/optim/test_named_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_named_optimizer_1.1_d9e65fdc400309c1_.log 2024-04-03T16:33:10.7204746Z 2024-04-03T16:33:10.7208773Z Running distributed/_tensor/test_op_strategy 1/1 ... [2024-04-03 16:33:10.720656] 2024-04-03T16:33:10.7213858Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_op_strategy.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:33:10.721099] 2024-04-03T16:33:13.5433864Z 2024-04-03T16:33:13.5436309Z distributed/_tensor/test_op_strategy 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_op_strategy_1.1_65c9f6b3ba596f9e_.log 2024-04-03T16:33:13.5445155Z Running 15 items in this shard: test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_batch_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_bmm_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_free_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_mm_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_bmm_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_bmm_2d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_linearity_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_mm_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_mm_2d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_pointwise_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_bmm_strategies, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_mm_strategies, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_redistribute_cost_latency, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_redistribute_cost_mesh_1d, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_redistribute_cost_mesh_2d 2024-04-03T16:33:13.5452199Z 2024-04-03T16:33:13.5452730Z Running distributed/_composable/fsdp/test_fully_shard_memory 1/1 ... [2024-04-03 16:33:13.543843] 2024-04-03T16:33:13.5454863Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:33:13.544243] 2024-04-03T16:33:16.3165302Z 2024-04-03T16:33:16.3167458Z distributed/_composable/fsdp/test_fully_shard_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_memory_1.1_0dc73789d534c1e5_.log 2024-04-03T16:33:16.3169462Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_memory.py::TestFullyShardMemory::test_fully_shard_training_memory 2024-04-03T16:33:16.3170404Z 2024-04-03T16:33:16.3171913Z Running distributed/checkpoint/e2e/test_pipeline 1/1 ... [2024-04-03 16:33:16.316974] 2024-04-03T16:33:16.3177068Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/e2e/test_pipeline.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:33:16.317415] 2024-04-03T16:33:19.1402456Z 2024-04-03T16:33:19.1404961Z distributed/checkpoint/e2e/test_pipeline 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_pipeline_1.1_a266e90144a3ed76_.log 2024-04-03T16:33:19.1406738Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_pipeline.py::TestPipeline::test_pipeline 2024-04-03T16:33:19.1407577Z 2024-04-03T16:33:19.1409490Z Running distributed/_tensor/test_embedding_ops 1/1 ... [2024-04-03 16:33:19.140673] 2024-04-03T16:33:19.1414282Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_embedding_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:33:19.141095] 2024-04-03T16:33:30.0353037Z 2024-04-03T16:33:30.0354979Z distributed/_tensor/test_embedding_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_embedding_ops_1.1_904e8eee66d6f76b_.log 2024-04-03T16:33:30.0358275Z Running 3 items in this shard: test/distributed/_tensor/test_embedding_ops.py::TestEmbeddingOp::test_sharded_embedding_colwise, test/distributed/_tensor/test_embedding_ops.py::TestEmbeddingOp::test_sharded_embedding_colwise_max_norm_errors, test/distributed/_tensor/test_embedding_ops.py::TestEmbeddingOp::test_sharded_embedding_rowwise 2024-04-03T16:33:30.0360490Z 2024-04-03T16:33:30.0361110Z Running distributed/checkpoint/e2e/test_e2e_save_and_load 1/1 ... [2024-04-03 16:33:30.035799] 2024-04-03T16:33:30.0366010Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/e2e/test_e2e_save_and_load.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:33:30.036238] 2024-04-03T16:33:58.3126823Z 2024-04-03T16:33:58.3129943Z distributed/checkpoint/e2e/test_e2e_save_and_load 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_e2e_save_and_load_1.1_fe2c4d2d84e6026e_.log 2024-04-03T16:33:58.3137538Z Running 10 items in this shard: test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_different_ordered_state_dict_keys, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_async, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_False_model_type0, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_False_model_type1, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_False_model_type2, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_True_model_type0, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_True_model_type1, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_True_model_type2, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_no_dist, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestNoCPU::test_no_cpu 2024-04-03T16:33:58.3143727Z 2024-04-03T16:33:58.3144299Z Running distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 ... [2024-04-03 16:33:58.313019] 2024-04-03T16:33:58.3146469Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:33:58.313421] 2024-04-03T16:34:01.0856081Z 2024-04-03T16:34:01.0858174Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_dict_1.1_f7fa839b5fd5e915_.log 2024-04-03T16:34:01.0861072Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDict::test_1d_state_dict_save_load, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDict::test_2d_state_dict_save_load 2024-04-03T16:34:01.0862697Z 2024-04-03T16:34:01.0863288Z Running distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 ... [2024-04-03 16:34:01.086023] 2024-04-03T16:34:01.0867884Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:34:01.086441] 2024-04-03T16:34:03.9086782Z 2024-04-03T16:34:03.9088723Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_f008010bcd5b5ae1_.log 2024-04-03T16:34:03.9094717Z Running 6 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_compute_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_reduce_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_float16_on_one_submodule, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_norm_modules_bf16, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_norm_modules_fp16, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_submodules_with_external_inputs 2024-04-03T16:34:03.9099258Z 2024-04-03T16:34:03.9120944Z Running distributed/_composable/fully_shard/test_fully_shard_mixed_precision 1/1 ... [2024-04-03 16:34:03.909244] 2024-04-03T16:34:03.9123543Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:34:03.909639] 2024-04-03T16:34:09.0859373Z 2024-04-03T16:34:09.0862132Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_mixed_precision_1.1_cbbc96b825c99539_.log 2024-04-03T16:34:09.0864391Z Running 1 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_mixed_precision.py::TestMixedPrecision::test_float16_cast_forward 2024-04-03T16:34:09.0865385Z 2024-04-03T16:34:09.0865820Z Running distributed/fsdp/test_shard_utils 1/1 ... [2024-04-03 16:34:09.086333] 2024-04-03T16:34:09.0870380Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_shard_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:34:09.086739] 2024-04-03T16:34:16.7160672Z 2024-04-03T16:34:16.7163263Z distributed/fsdp/test_shard_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_shard_utils_1.1_7d0dad92f6e0e480_.log 2024-04-03T16:34:16.7167485Z Running 2 items in this shard: test/distributed/fsdp/test_shard_utils.py::TestShardUtilsDistributed::test_create_chunk_sharded_tensor, test/distributed/fsdp/test_shard_utils.py::TestShardUtilsDistributedDTensor::test_create_chunk_dtensor 2024-04-03T16:34:16.7170086Z 2024-04-03T16:34:16.7170994Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-04-03 16:34:16.716733] 2024-04-03T16:34:16.7177202Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:34:16.717282] 2024-04-03T16:34:19.5399085Z 2024-04-03T16:34:19.5401615Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_3601819a77c13780_.log 2024-04-03T16:34:19.5415876Z Running 17 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::test_compiled_fsdp, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardForwardInputs::test_root_move_forward_input_to_device, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardRegisteredParams::test_param_registration_after_backward, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardRegisteredParams::test_param_registration_after_forward, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardCastAfterInit::test_to_float64_after_init, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_multi_forward_module, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_non_root_forward_backward, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group_compile, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group_eager, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_single_group, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCompose::test_train_parity_with_activation_checkpointing, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardSharedParams::test_train_parity_with_shared_params_ac, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardSharedParams::test_train_parity_with_shared_params_no_ac, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardGradientAccumulation::test_set_requires_gradient_sync, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard2DTraining::test_train_parity_2d_mlp, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard2DTraining::test_train_parity_2d_transformer_checkpoint_resume, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardHSDPTraining::test_train_parity_hsdp 2024-04-03T16:34:19.5427867Z 2024-04-03T16:34:19.5428316Z Running distributed/_composable/test_compose 1/1 ... [2024-04-03 16:34:19.540375] 2024-04-03T16:34:19.5430261Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/test_compose.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:34:19.540800] 2024-04-03T16:34:44.0990040Z 2024-04-03T16:34:44.0991999Z distributed/_composable/test_compose 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_compose_1.1_0f88bf4630f41b64_.log 2024-04-03T16:34:44.0999452Z Running 9 items in this shard: test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_checkpoint_fsdp_submodules_non_reentrant, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_checkpoint_fsdp_submodules_with_param, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_checkpoint_fsdp_submodules_with_param_no_shard, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_checkpoint_fully_shard_cast_forward_inputs, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_composable_fsdp_replicate, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_fully_shard_replicate_composability, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_fully_shard_replicate_correct_replicate_params, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_state_dict_fsdp_submodules, test/distributed/_composable/test_compose.py::TestFSDPCheckpoint::test_wrap_same_submodule 2024-04-03T16:34:44.1004943Z 2024-04-03T16:34:44.1005338Z Running distributed/test_fake_pg 1/1 ... [2024-04-03 16:34:44.099600] 2024-04-03T16:34:44.1007152Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_fake_pg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:34:44.100049] 2024-04-03T16:34:46.8721453Z 2024-04-03T16:34:46.8723805Z distributed/test_fake_pg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_fake_pg_1.1_d32589e6657fb4db_.log 2024-04-03T16:34:46.8730154Z Running 13 items in this shard: test/distributed/test_fake_pg.py::TestFakePG::test_all_reduce, test/distributed/test_fake_pg.py::TestFakePG::test_allgather, test/distributed/test_fake_pg.py::TestFakePG::test_alltoall, test/distributed/test_fake_pg.py::TestFakePG::test_alltoall_base, test/distributed/test_fake_pg.py::TestFakePG::test_broadcast, test/distributed/test_fake_pg.py::TestFakePG::test_construct_fsdp, test/distributed/test_fake_pg.py::TestFakePG::test_fake_pg_tracing, test/distributed/test_fake_pg.py::TestFakePG::test_fsdp_fake_e2e, test/distributed/test_fake_pg.py::TestFakePG::test_fsdp_tp_fake_e2e, test/distributed/test_fake_pg.py::TestFakePG::test_recv, test/distributed/test_fake_pg.py::TestFakePG::test_reduce_scatter, test/distributed/test_fake_pg.py::TestFakePG::test_scatter, test/distributed/test_fake_pg.py::TestFakePG::test_send 2024-04-03T16:34:46.8735477Z 2024-04-03T16:34:46.8735996Z Running distributed/tensor/parallel/test_parallelize_api 1/1 ... [2024-04-03 16:34:46.872626] 2024-04-03T16:34:46.8738120Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/tensor/parallel/test_parallelize_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:34:46.873039] 2024-04-03T16:35:05.2783177Z 2024-04-03T16:35:05.2785297Z distributed/tensor/parallel/test_parallelize_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_parallelize_api_1.1_b154395c4836d606_.log 2024-04-03T16:35:05.2790814Z Running 6 items in this shard: test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_linear_col_wise_parallel, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_linear_row_wise_parallel, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_mlp_with_module_api, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_mlp_with_module_api_nested, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_prepare_module_input, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_prepare_module_output 2024-04-03T16:35:05.2795244Z 2024-04-03T16:35:05.2795982Z Running distributed/_composable/fully_shard/test_fully_shard_util 1/1 ... [2024-04-03 16:35:05.278798] 2024-04-03T16:35:05.2798158Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_util.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:35:05.279252] 2024-04-03T16:35:10.5055577Z 2024-04-03T16:35:10.5058551Z distributed/_composable/fully_shard/test_fully_shard_util 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_util_1.1_fdc8c5f5885e449f_.log 2024-04-03T16:35:10.5062433Z Running 1 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_util.py::TestUtils::test_get_sharded_module_tree_with_module_name_to_fqns 2024-04-03T16:35:10.5063997Z 2024-04-03T16:35:10.5065204Z Running distributed/test_inductor_collectives 1/1 ... [2024-04-03 16:35:10.506280] 2024-04-03T16:35:10.5073121Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_inductor_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:35:10.506860] 2024-04-03T16:35:13.5797603Z 2024-04-03T16:35:13.5799171Z distributed/test_inductor_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_inductor_collectives_1.1_41b9cd5a6ae916a9_.log 2024-04-03T16:35:13.5871197Z Running 75 items in this shard: test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_input_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_output_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_contiguous_input, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_into_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_output_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_input_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_broadcast_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_c10d_functional_tagged_pt2_compliant, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_inductor_allreduce_eager_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_permute_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_reduce_scatter_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_backwards_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_backwards_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_use_native_funcol_False_source_GroupMember_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_use_native_funcol_False_source__get_default_group, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_use_native_funcol_False_source_group_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_use_native_funcol_True_source_GroupMember_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_use_native_funcol_True_source__get_default_group, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_use_native_funcol_True_source_group_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_graphbreaks_unsupported_async_op_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_graphbreaks_unsupported_async_op_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_pg_var_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_pg_var_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_args_match_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_args_match_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_list_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_list_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_to_all_single_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_to_all_single_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_False_reduce_op0, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_False_reduce_op1, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_False_reduce_op2, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_False_reduce_op3, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_False_reduce_op4, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_True_reduce_op0, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_True_reduce_op1, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_True_reduce_op2, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_True_reduce_op3, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_use_native_funcol_True_reduce_op4, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_False_pg_mode_kwargs, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_False_pg_mode_kwargs_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_False_pg_mode_positional, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_False_pg_mode_positional_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_False_pg_mode_unspecified, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_True_pg_mode_kwargs, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_True_pg_mode_kwargs_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_True_pg_mode_positional, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_True_pg_mode_positional_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_use_native_funcol_True_pg_mode_unspecified, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_reduce_scatter_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_reduce_scatter_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_support_collective_op_with_async_op_False_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_support_collective_op_with_async_op_False_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_pg_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_pg_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allgather_coalesced_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allgather_coalesced_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allreduce_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allreduce_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_reduce_scatter_tensor_use_native_funcol_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_reduce_scatter_tensor_use_native_funcol_True, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_all_gather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_doesnt_mutate_shared, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_reduce_scatter_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_single_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_steal_buffer, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_meta 2024-04-03T16:35:13.5967988Z 2024-04-03T16:35:13.5969015Z Running distributed/fsdp/test_fsdp_backward_prefetch 1/1 ... [2024-04-03 16:35:13.580240] 2024-04-03T16:35:13.5972856Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_backward_prefetch.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:35:13.580652] 2024-04-03T16:35:18.8075025Z 2024-04-03T16:35:18.8077019Z distributed/fsdp/test_fsdp_backward_prefetch 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_backward_prefetch_1.1_c243a53aa6046137_.log 2024-04-03T16:35:18.8079013Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_backward_prefetch.py::TestBackwardPrefetch::test_backward_prefetch 2024-04-03T16:35:18.8079875Z 2024-04-03T16:35:18.8081406Z Running distributed/_composable/test_replicate 1/1 ... [2024-04-03 16:35:18.807917] 2024-04-03T16:35:18.8086527Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/test_replicate.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:35:18.808346] 2024-04-03T16:35:43.6685059Z 2024-04-03T16:35:43.6686706Z distributed/_composable/test_replicate 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_replicate_1.1_e986aeb3d978d0d6_.log 2024-04-03T16:35:43.6695209Z Running 9 items in this shard: test/distributed/_composable/test_replicate.py::ReplicateStateDictTest::test_replicate_non_root_multiple_save_load, test/distributed/_composable/test_replicate.py::ReplicateStateDictTest::test_replicate_single_module_save_load, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_device_id, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_ignore_module, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_move_args_kwargs_to_device, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_multi_module, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_single_module, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_with_kwargs, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_wrong_device_id_type 2024-04-03T16:35:43.6700894Z 2024-04-03T16:35:43.6701447Z Running distributed/_shard/sharding_plan/test_sharding_plan 1/1 ... [2024-04-03 16:35:43.669098] 2024-04-03T16:35:43.6703563Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:35:43.669537] 2024-04-03T16:35:46.0912343Z 2024-04-03T16:35:46.0914427Z distributed/_shard/sharding_plan/test_sharding_plan 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharding_plan.test_sharding_plan_1.1_6227847af6ad7149_.log 2024-04-03T16:35:46.0919442Z Running 3 items in this shard: test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_custom_sharding_planner, test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_shard_module_sub_process_group, test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_sharding_plan_errors 2024-04-03T16:35:46.0921767Z 2024-04-03T16:35:46.0922417Z Running distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 ... [2024-04-03 16:35:46.091680] 2024-04-03T16:35:46.0924789Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:35:46.092106] 2024-04-03T16:35:48.8645908Z 2024-04-03T16:35:48.8647917Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_clip_grad_norm__1.1_7d59e50b4c6f60eb_.log 2024-04-03T16:35:48.8650799Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py::TestClipGradNormWorldSize2::test_clip_grad_norm_1d, test/distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py::TestClipGradNormWorldSize4::test_clip_grad_norm_2d 2024-04-03T16:35:48.8652470Z 2024-04-03T16:35:48.8653025Z Running distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 ... [2024-04-03 16:35:48.865035] 2024-04-03T16:35:48.8658167Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_tp_checkpoint_conversion.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:35:48.865487] 2024-04-03T16:35:54.1929231Z 2024-04-03T16:35:54.1931410Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion_1.1_bfa7bf9286935c1e_.log 2024-04-03T16:35:54.1933534Z Running 1 items in this shard: test/distributed/checkpoint/test_fsdp_tp_checkpoint_conversion.py::TestFsdpTpCheckpointConversion::test_fsdp_to_tp 2024-04-03T16:35:54.1934486Z 2024-04-03T16:35:54.1936255Z Running distributed/_tensor/test_optimizers 1/1 ... [2024-04-03 16:35:54.193368] 2024-04-03T16:35:54.1941428Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_tensor/test_optimizers.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:35:54.193799] 2024-04-03T16:36:25.5797256Z 2024-04-03T16:36:25.5799765Z distributed/_tensor/test_optimizers 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_optimizers_1.1_b7a9f8258a370dc6_.log 2024-04-03T16:36:25.5806833Z Running 10 items in this shard: test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_RMSprop_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adadelta_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adagrad_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adam_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adamax_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adamw_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_asgd_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_nadam_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_radam_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_sgd_1d_sharding 2024-04-03T16:36:25.5812460Z 2024-04-03T16:36:25.5812933Z Running distributed/fsdp/test_fsdp_pure_fp16 1/1 ... [2024-04-03 16:36:25.580220] 2024-04-03T16:36:25.5814885Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_pure_fp16.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:36:25.580641] 2024-04-03T16:36:33.6650877Z 2024-04-03T16:36:33.6652441Z distributed/fsdp/test_fsdp_pure_fp16 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_pure_fp16_1.1_63671cd20dfb8e33_.log 2024-04-03T16:36:33.6655165Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_pure_fp16.py::TestPureFP16::test_fp16_dtypes, test/distributed/fsdp/test_fsdp_pure_fp16.py::TestPureFP16::test_pure_fp16_training 2024-04-03T16:36:33.6656255Z 2024-04-03T16:36:33.6658075Z Running distributed/fsdp/test_fsdp_optim_state 1/1 ... [2024-04-03 16:36:33.665583] 2024-04-03T16:36:33.6662702Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_optim_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:36:33.665991] 2024-04-03T16:39:25.8927679Z 2024-04-03T16:39:25.8929439Z distributed/fsdp/test_fsdp_optim_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_optim_state_1.1_bb2ec51a96d40a77_.log 2024-04-03T16:39:25.8979489Z Running 60 items in this shard: test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_compatible_with_trec, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_flatten_sharded_optim_state_dict_nested, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_flatten_sharded_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_full_optim_state_dict_keys, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_full_optim_state_dict_nested_invalid, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_interface_arguments, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_no_grad, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_input_warning, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_without_param_groups, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type0_use_multiple_param_groups_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type0_use_multiple_param_groups_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type1_use_multiple_param_groups_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type1_use_multiple_param_groups_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_names, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_save_load_without_0th_param_state_state_dict_type0, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_save_load_without_0th_param_state_state_dict_type1, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_halve_world_size, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_halve_world_size, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type0_add_to_fsdp_module_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type0_add_to_fsdp_module_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type1_add_to_fsdp_module_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type1_add_to_fsdp_module_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_state_dict_with_none_tensor_state, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_use_orig_params, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_with_empty_optimizer_state, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_with_no_shard 2024-04-03T16:39:25.9026859Z 2024-04-03T16:39:25.9027313Z Running distributed/_shard/test_sharder 1/1 ... [2024-04-03 16:39:25.893563] 2024-04-03T16:39:25.9029293Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:39:25.893981] 2024-04-03T16:39:28.3154872Z 2024-04-03T16:39:28.3157283Z distributed/_shard/test_sharder 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.test_sharder_1.1_83f6f5faaddf25bd_.log 2024-04-03T16:39:28.3159679Z Running 2 items in this shard: test/distributed/_shard/test_sharder.py::TestCustomSharder::test_custom_sharder, test/distributed/_shard/test_sharder.py::TestCustomSharder::test_custom_sharder_errors 2024-04-03T16:39:28.3161021Z 2024-04-03T16:39:28.3161699Z Running distributed/fsdp/test_fsdp_overlap 1/1 ... [2024-04-03 16:39:28.315963] 2024-04-03T16:39:28.3167290Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:39:28.316404] 2024-04-03T16:39:35.9470593Z 2024-04-03T16:39:35.9472499Z distributed/fsdp/test_fsdp_overlap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_overlap_1.1_820a1f0405dce9cc_.log 2024-04-03T16:39:35.9474775Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_overlap.py::TestForwardOverlapWorldSizeOne::test_forward_overlap, test/distributed/fsdp/test_fsdp_overlap.py::TestForwardOverlapWorldSizeTwo::test_forward_overlap 2024-04-03T16:39:35.9476725Z 2024-04-03T16:39:35.9477182Z Running distributed/test_c10d_logger 1/1 ... [2024-04-03 16:39:35.947500] 2024-04-03T16:39:35.9482476Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:39:35.947965] 2024-04-03T16:39:45.7825947Z 2024-04-03T16:39:45.7827760Z distributed/test_c10d_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_logger_1.1_ee87361701f26332_.log 2024-04-03T16:39:45.7830175Z Running 3 items in this shard: test/distributed/test_c10d_logger.py::C10dErrorLoggerTest::test_exception_logger, test/distributed/test_c10d_logger.py::C10dErrorLoggerTest::test_get_or_create_logger, test/distributed/test_c10d_logger.py::C10dErrorLoggerTest::test_time_logger 2024-04-03T16:39:45.7832308Z 2024-04-03T16:39:45.7832848Z Running distributed/_shard/sharding_spec/test_sharding_spec 1/1 ... [2024-04-03 16:39:45.783002] 2024-04-03T16:39:45.7837792Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:39:45.783416] 2024-04-03T16:39:51.4116320Z 2024-04-03T16:39:51.4118153Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharding_spec.test_sharding_spec_1.1_61c905b36aa2fbc1_.log 2024-04-03T16:39:51.4126639Z Running 11 items in this shard: test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_check_overlapping, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_chunked_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_device_placement, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_enumerable_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_chunk_sharding_params, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_chunked_dim_size, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_split_size, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_infer_sharding_spec_from_shards_metadata, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec_shard_tensor, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec_tensor_ctor 2024-04-03T16:39:51.4133601Z 2024-04-03T16:39:51.4134081Z Running distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 ... [2024-04-03 16:39:51.412088] 2024-04-03T16:39:51.4136134Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_dtensor_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:39:51.412535] 2024-04-03T16:40:33.3682536Z 2024-04-03T16:40:33.3684856Z distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_dtensor_state_dict_1.1_92eea54192d92e0e_.log 2024-04-03T16:40:33.3700816Z Running 15 items in this shard: test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_False_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_False_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_True_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_True_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_False_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_False_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_True_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_True_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_False_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_False_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_True_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_True_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_fsdp_init_with_device_mesh_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_fsdp_init_with_device_mesh_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_raises_warning_or_errors 2024-04-03T16:40:33.3716240Z 2024-04-03T16:40:33.3717114Z Running distributed/checkpoint/test_checkpoint 1/1 ... [2024-04-03 16:40:33.368808] 2024-04-03T16:40:33.3719297Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:40:33.369239] 2024-04-03T16:40:41.3026400Z 2024-04-03T16:40:41.3028278Z distributed/checkpoint/test_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_checkpoint_1.1_bc3bf59fd091d0f3_.log 2024-04-03T16:40:41.3034307Z Running 8 items in this shard: test/distributed/checkpoint/test_checkpoint.py::TestDistributedCheckpointing::test_default_metadata, test/distributed/checkpoint/test_checkpoint.py::TestDistributedCheckpointing::test_tensor_metadata_with_missing_rank_spec, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_dummy_reader_works, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_dummy_writer_works, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_load_error_handling, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_load_error_handling_no_dist, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_save_error_handling, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_save_error_handling_no_dist 2024-04-03T16:40:41.3039398Z 2024-04-03T16:40:41.3039827Z Running distributed/test_multi_threaded_pg 1/1 ... [2024-04-03 16:40:41.303069] 2024-04-03T16:40:41.3041732Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_multi_threaded_pg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:40:41.303493] 2024-04-03T16:40:44.1763666Z 2024-04-03T16:40:44.1765578Z distributed/test_multi_threaded_pg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_multi_threaded_pg_1.1_e722a700d2b2e7ee_.log 2024-04-03T16:40:44.1777792Z Running 19 items in this shard: test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_broadcast_object_list, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_non_zero, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_non_zero_all, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_zero, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_skip, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce_coalesced, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce_ops, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_to_all, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_allgather, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_assert_equal_on_rank, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_broadcast, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_broadcast_object_list, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_bwd_sees_fwd_pg, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_gather, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_reduce_scatter, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_scatter, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_subpg, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_using_pg_from_another_thread 2024-04-03T16:40:44.1788117Z 2024-04-03T16:40:44.1788564Z Running distributed/nn/jit/test_instantiator 1/1 ... [2024-04-03 16:40:44.176907] 2024-04-03T16:40:44.1790652Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:40:44.177332] 2024-04-03T16:40:46.9995956Z 2024-04-03T16:40:46.9997610Z distributed/nn/jit/test_instantiator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.nn.jit.test_instantiator_1.1_5681b5ed2dc74aaa_.log 2024-04-03T16:40:47.0000634Z Running 3 items in this shard: test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_get_arg_return_types_from_interface, test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_instantiate_non_scripted_remote_module_template, test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_instantiate_scripted_remote_module_template 2024-04-03T16:40:47.0003022Z 2024-04-03T16:40:47.0003562Z Running distributed/checkpoint/test_file_system_checkpoint 1/1 ... [2024-04-03 16:40:47.000058] 2024-04-03T16:40:47.0008187Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:40:47.000510] 2024-04-03T16:40:49.8730349Z 2024-04-03T16:40:49.8732340Z distributed/checkpoint/test_file_system_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_1.1_7837f255faea5980_.log 2024-04-03T16:40:49.8738314Z Running 6 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_save_load_bytes, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor 2024-04-03T16:40:49.8742947Z 2024-04-03T16:40:49.8743383Z Running distributed/rpc/test_share_memory 1/1 ... [2024-04-03 16:40:49.873487] 2024-04-03T16:40:49.8745371Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/rpc/test_share_memory.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:40:49.873959] 2024-04-03T16:40:56.2765296Z 2024-04-03T16:40:56.2767215Z distributed/rpc/test_share_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.test_share_memory_1.1_73b66a8d4cdff0d9_.log 2024-04-03T16:40:56.2768769Z Running 1 items in this shard: test/distributed/rpc/test_share_memory.py::TestRPCPickler::test_case 2024-04-03T16:40:56.2769946Z Running 1 items in this shard: test/distributed/rpc/test_share_memory.py::TestRPCPickler::test_case 2024-04-03T16:40:56.2770646Z 2024-04-03T16:40:56.2771170Z Running distributed/fsdp/test_fsdp_freezing_weights 1/1 ... [2024-04-03 16:40:56.276892] 2024-04-03T16:40:56.2777014Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_freezing_weights.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:40:56.277304] 2024-04-03T16:42:22.4752202Z 2024-04-03T16:42:22.4754022Z distributed/fsdp/test_fsdp_freezing_weights 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_freezing_weights_1.1_11fa00d472c76dd6_.log 2024-04-03T16:42:22.4804010Z Running 32 items in this shard: test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True 2024-04-03T16:42:22.4874125Z 2024-04-03T16:42:22.4875064Z Running distributed/pipeline/sync/test_pipe 1/1 ... [2024-04-03 16:42:22.475798] 2024-04-03T16:42:22.4879213Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/test_pipe.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:42:22.476271] 2024-04-03T16:42:26.7521883Z 2024-04-03T16:42:26.7523547Z distributed/pipeline/sync/test_pipe 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_pipe_1.1_9ddb0f0081239f3f_.log 2024-04-03T16:42:26.7547993Z Running 56 items in this shard: test/distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc, test/distributed/pipeline/sync/test_pipe.py::test_parameters, test/distributed/pipeline/sync/test_pipe.py::test_public_attrs, test/distributed/pipeline/sync/test_pipe.py::test_sequential_like, test/distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_small, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input, test/distributed/pipeline/sync/test_pipe.py::test_no_grad, test/distributed/pipeline/sync/test_pipe.py::test_exception, test/distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap, test/distributed/pipeline/sync/test_pipe.py::test_nested_input, test/distributed/pipeline/sync/test_pipe.py::test_input_pair, test/distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input, test/distributed/pipeline/sync/test_pipe.py::test_input_singleton, test/distributed/pipeline/sync/test_pipe.py::test_input_varargs, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence, test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[never], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[always], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always], test/distributed/pipeline/sync/test_pipe.py::test_devices, test/distributed/pipeline/sync/test_pipe.py::test_partitions, test/distributed/pipeline/sync/test_pipe.py::test_merged_partitions, test/distributed/pipeline/sync/test_pipe.py::test_deny_moving, test/distributed/pipeline/sync/test_pipe.py::test_empty_module, test/distributed/pipeline/sync/test_pipe.py::test_named_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_forward_lockstep, test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last], test/distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device, test/distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper 2024-04-03T16:42:26.7585312Z 2024-04-03T16:42:26.7586170Z Running distributed/fsdp/test_fsdp_fine_tune 1/1 ... [2024-04-03 16:42:26.752658] 2024-04-03T16:42:26.7589702Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fine_tune.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:42:26.753080] 2024-04-03T16:42:29.5759640Z 2024-04-03T16:42:29.5762200Z distributed/fsdp/test_fsdp_fine_tune 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_fine_tune_1.1_c31aaddc70e2d398_.log 2024-04-03T16:42:29.5768211Z Running 4 items in this shard: test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTune::test_backward_reshard_hooks, test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTune::test_hooks_multi_traversal, test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTune::test_parity_with_ddp, test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTune::test_parity_with_non_frozen_fsdp 2024-04-03T16:42:29.5771920Z 2024-04-03T16:42:29.5772706Z Running distributed/fsdp/test_fsdp_core 1/1 ... [2024-04-03 16:42:29.576591] 2024-04-03T16:42:29.5775909Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:42:29.577139] 2024-04-03T16:45:08.6929022Z 2024-04-03T16:45:08.6932782Z distributed/fsdp/test_fsdp_core 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_core_1.1_e93302c253b9101e_.log 2024-04-03T16:45:08.6971119Z Running 60 items in this shard: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParamInit::test_param_change_after_init_mixed_precision_False, test/distributed/fsdp/test_fsdp_core.py::TestParamInit::test_param_change_after_init_mixed_precision_True, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_pre_backward_hook_registration_after_state_dict, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_pre_backward_hook_registration_cuda_first_False, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_pre_backward_hook_registration_cuda_first_True, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_register_functions_called_cuda_first_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_register_functions_called_cuda_first_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_register_functions_called_cuda_first_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_register_functions_called_cuda_first_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_core.py::TestNoGrad::test_transformer_no_grad_mixed_precision_False, test/distributed/fsdp/test_fsdp_core.py::TestNoGrad::test_transformer_no_grad_mixed_precision_True, test/distributed/fsdp/test_fsdp_core.py::TestAutograd::test_unshard_params_as_tensors 2024-04-03T16:45:08.7007365Z 2024-04-03T16:45:08.7007838Z Running distributed/fsdp/test_fsdp_meta 1/1 ... [2024-04-03 16:45:08.693589] 2024-04-03T16:45:08.7009753Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:45:08.694023] 2024-04-03T16:45:30.9485066Z 2024-04-03T16:45:30.9487234Z distributed/fsdp/test_fsdp_meta 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_meta_1.1_f974e25cb575ac40_.log 2024-04-03T16:45:30.9504389Z Running 15 items in this shard: test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_bad_arg_meta, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_bad_arg_torchdistx, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_meta_device_with_mixed_precision, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_default_init_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_default_init_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_reset_params_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_reset_params_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_default_init_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_default_init_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_init_fn_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_init_fn_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_meta_device_default_init, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_meta_device_reset_params, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_torchdistX_default_init, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_torchdistX_init_fn 2024-04-03T16:45:30.9521198Z 2024-04-03T16:45:30.9522094Z Running distributed/optim/test_apply_optimizer_in_backward 1/1 ... [2024-04-03 16:45:30.949113] 2024-04-03T16:45:30.9525940Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/optim/test_apply_optimizer_in_backward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:45:30.949581] 2024-04-03T16:45:33.3284880Z 2024-04-03T16:45:33.3287843Z distributed/optim/test_apply_optimizer_in_backward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_apply_optimizer_in_backward_1.1_a37fc83f4cbabb8c_.log 2024-04-03T16:45:33.3289312Z 2024-04-03T16:45:33.3291129Z Running distributed/fsdp/test_fsdp_checkpoint 1/1 ... [2024-04-03 16:45:33.328857] 2024-04-03T16:45:33.3296064Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:45:33.329281] 2024-04-03T16:46:19.9041459Z 2024-04-03T16:46:19.9043086Z distributed/fsdp/test_fsdp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_checkpoint_1.1_d2423d4fbd029049_.log 2024-04-03T16:46:19.9059875Z Running 17 items in this shard: test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpointSubmodule::test_checkpoint_submodule_use_reentrant_False 2024-04-03T16:46:19.9074039Z 2024-04-03T16:46:19.9074505Z Running distributed/pipeline/sync/test_worker 1/1 ... [2024-04-03 16:46:19.904779] 2024-04-03T16:46:19.9076764Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/test_worker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:46:19.905232] 2024-04-03T16:46:21.6262820Z 2024-04-03T16:46:21.6264438Z distributed/pipeline/sync/test_worker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_worker_1.1_452ddf41804cba69_.log 2024-04-03T16:46:21.6268599Z Running 6 items in this shard: test/distributed/pipeline/sync/test_worker.py::test_compute_multithreading, test/distributed/pipeline/sync/test_worker.py::test_compute_success, test/distributed/pipeline/sync/test_worker.py::test_compute_exception, test/distributed/pipeline/sync/test_worker.py::test_grad_mode[True], test/distributed/pipeline/sync/test_worker.py::test_grad_mode[False], test/distributed/pipeline/sync/test_worker.py::test_worker_per_device 2024-04-03T16:46:21.6271682Z 2024-04-03T16:46:21.6272172Z Running distributed/pipeline/sync/test_transparency 1/1 ... [2024-04-03 16:46:21.626626] 2024-04-03T16:46:21.6274252Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/test_transparency.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:46:21.627024] 2024-04-03T16:46:23.5990649Z 2024-04-03T16:46:23.5992537Z distributed/pipeline/sync/test_transparency 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_transparency_1.1_61ea1258edadc26f_.log 2024-04-03T16:46:23.5994456Z Running 1 items in this shard: test/distributed/pipeline/sync/test_transparency.py::test_simple_linears 2024-04-03T16:46:23.5995172Z 2024-04-03T16:46:23.5997242Z Running distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 1/1 ... [2024-04-03 16:46:23.599491] 2024-04-03T16:46:23.6002164Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:46:23.599917] 2024-04-03T16:46:31.2309391Z 2024-04-03T16:46:31.2311734Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_optim_checkpoint_1.1_bd80261e2889144b_.log 2024-04-03T16:46:31.2314831Z Running 2 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint.py::TestOptimStateCheckpointing::test_optim_state_dict_save_load, test/distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint.py::TestOptimStateCheckpointing::test_optim_state_dict_submodule_fully_shard 2024-04-03T16:46:31.2317713Z 2024-04-03T16:46:31.2318195Z Running distributed/fsdp/test_fsdp_unshard_params 1/1 ... [2024-04-03 16:46:31.231436] 2024-04-03T16:46:31.2322305Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_unshard_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:46:31.231893] 2024-04-03T16:47:07.8606373Z 2024-04-03T16:47:07.8607947Z distributed/fsdp/test_fsdp_unshard_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_unshard_params_1.1_846a6e45b89ae422_.log 2024-04-03T16:47:07.8621098Z Running 14 items in this shard: test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_named_parameters_and_buffers, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_param_data, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_recurse, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_respects_reshard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_writeback, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_singleton_param_writeback, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_with_grads_core, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_with_grads_none_grads, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsNoShard::test_unshard_params_param_data_no_shard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsNoShard::test_unshard_params_writeback_no_shard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_offload_to_cpu_no_shard_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_rank0_only_with_writeback_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_unshard_params_from_backward_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_unshard_params_from_forward_raises 2024-04-03T16:47:07.8629752Z 2024-04-03T16:47:07.8630210Z Running distributed/fsdp/test_fsdp_misc 1/1 ... [2024-04-03 16:47:07.861079] 2024-04-03T16:47:07.8632117Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_misc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:47:07.861518] 2024-04-03T16:47:45.2897793Z 2024-04-03T16:47:45.2899746Z distributed/fsdp/test_fsdp_misc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_misc_1.1_eb18615a869e9360_.log 2024-04-03T16:47:45.2919426Z Running 27 items in this shard: test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_cpu_init_with_sync_module_states, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_cpu_init_stays_on_cpu, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_cpu_training, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_device_id_use_index_False, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_device_id_use_index_True, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_not_all_outputs_used_in_loss, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_optim_overlap_no_use_orig_params_error, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_optimizer_overlap, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_zero2_eval_with_prefetch, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_cpu_gpu_module, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_device_id_auto_wrap, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_device_id_cpu_offload, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_device_id_no_move_ignored_params_and_bufs, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_ignored_module_meta, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_namedtuple, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_same_model_across_ranks, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_homogeneous_attributes, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_module_device_mismatches_device_id, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_multigpu_module, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_no_params, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_training_device_mismatch_errors, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_unsafe_setattr, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_world_size_1_sharding_strategy_warning 2024-04-03T16:47:45.2935212Z 2024-04-03T16:47:45.2935734Z Running distributed/fsdp/test_fsdp_hybrid_shard 1/1 ... [2024-04-03 16:47:45.290559] 2024-04-03T16:47:45.2937732Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/fsdp/test_fsdp_hybrid_shard.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:47:45.291102] 2024-04-03T16:48:02.5911472Z 2024-04-03T16:48:02.5913131Z distributed/fsdp/test_fsdp_hybrid_shard 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_hybrid_shard_1.1_542bf96b73b55345_.log 2024-04-03T16:48:02.5918062Z Running 6 items in this shard: test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_fsdp_hybrid_shard_basic_setup, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_fsdp_hybrid_shard_parity, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_hsdp_save_load_state_dict, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_hsdp_sync_module_state, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_invalid_pg_specification_raises, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_raises_manual_wrap_hybrid_shard_when_none_policy 2024-04-03T16:48:02.5921900Z 2024-04-03T16:48:02.5922435Z Running distributed/pipeline/sync/test_microbatch 1/1 ... [2024-04-03 16:48:02.591658] 2024-04-03T16:48:02.5924613Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/test_microbatch.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:48:02.592092] 2024-04-03T16:48:04.3139291Z 2024-04-03T16:48:04.3141323Z distributed/pipeline/sync/test_microbatch 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_microbatch_1.1_22981cfa9dded8ac_.log 2024-04-03T16:48:04.3147379Z Running 10 items in this shard: test/distributed/pipeline/sync/test_microbatch.py::test_batch_atomic, test/distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic, test/distributed/pipeline/sync/test_microbatch.py::test_batch_call, test/distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index, test/distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice, test/distributed/pipeline/sync/test_microbatch.py::test_check, test/distributed/pipeline/sync/test_microbatch.py::test_gather_tensors, test/distributed/pipeline/sync/test_microbatch.py::test_gather_tuples, test/distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor, test/distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors 2024-04-03T16:48:04.3152060Z 2024-04-03T16:48:04.3152604Z Running distributed/pipeline/sync/test_checkpoint 1/1 ... [2024-04-03 16:48:04.314353] 2024-04-03T16:48:04.3154819Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/test_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:48:04.314758] 2024-04-03T16:48:06.2367720Z 2024-04-03T16:48:06.2370038Z distributed/pipeline/sync/test_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_checkpoint_1.1_49627a3d14d60058_.log 2024-04-03T16:48:06.2375163Z Running 7 items in this shard: test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad, test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter, test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing, test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint, test/distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output 2024-04-03T16:48:06.2379409Z 2024-04-03T16:48:06.2379832Z Running distributed/rpc/test_faulty_agent 1/1 ... [2024-04-03 16:48:06.237228] 2024-04-03T16:48:06.2381831Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/rpc/test_faulty_agent.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:48:06.237637] 2024-04-03T16:53:30.7244438Z 2024-04-03T16:53:30.7246767Z distributed/rpc/test_faulty_agent 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.test_faulty_agent_1.1_5f7695819a10f268_.log 2024-04-03T16:53:30.7269830Z Running 28 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_builtin_remote_message_dropped_timeout, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_builtin_remote_message_dropped_timeout_to_self, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_check_failed_messages, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_custom_faulty_messages, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_custom_messages_to_delay, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_no_faulty_messages, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_builtin_delay_timeout, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_builtin_delay_timeout_to_self, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_dropped_pickle, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_dropped_pickle_to_self, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_script_delay_timeout, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_script_delay_timeout_to_self, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_rpc_builtin_timeout, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_rpc_script_timeout, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_rref_to_here_timeout, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_udf_remote_message_delay_timeout, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_udf_remote_message_delay_timeout_to_self, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_udf_remote_message_dropped_timeout, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_udf_remote_message_dropped_timeout_to_self, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_verify_backend_options, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentDistAutogradTest::test_context_cleanup_tensor_with_grad, test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentDistAutogradTest::test_verify_backend_options, test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_remote_timeout_to_here_in_jit, test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_rref_timeout_pickle_in_jit, test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_rref_timeout_pickle_script_func, test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_rref_to_here_timeout_in_jit, test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_timeout_in_python, test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_timeout_in_torchscript_function 2024-04-03T16:53:30.7287294Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_builtin_remote_message_dropped_timeout 2024-04-03T16:53:30.7288896Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_builtin_remote_message_dropped_timeout_to_self 2024-04-03T16:53:30.7290396Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_check_failed_messages 2024-04-03T16:53:30.7291762Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_custom_faulty_messages 2024-04-03T16:53:30.7293145Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_custom_messages_to_delay 2024-04-03T16:53:30.7294545Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_no_faulty_messages 2024-04-03T16:53:30.7295966Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_builtin_delay_timeout 2024-04-03T16:53:30.7297528Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_builtin_delay_timeout_to_self 2024-04-03T16:53:30.7299033Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_dropped_pickle 2024-04-03T16:53:30.7300524Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_dropped_pickle_to_self 2024-04-03T16:53:30.7302045Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_script_delay_timeout 2024-04-03T16:53:30.7303587Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_remote_message_script_delay_timeout_to_self 2024-04-03T16:53:30.7305051Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_rpc_builtin_timeout 2024-04-03T16:53:30.7306401Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_rpc_script_timeout 2024-04-03T16:53:30.7307743Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_rref_to_here_timeout 2024-04-03T16:53:30.7309163Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_udf_remote_message_delay_timeout 2024-04-03T16:53:30.7310688Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_udf_remote_message_delay_timeout_to_self 2024-04-03T16:53:30.7312232Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_udf_remote_message_dropped_timeout 2024-04-03T16:53:30.7313777Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_udf_remote_message_dropped_timeout_to_self 2024-04-03T16:53:30.7315239Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentRpcTest::test_verify_backend_options 2024-04-03T16:53:30.7316871Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentDistAutogradTest::test_context_cleanup_tensor_with_grad 2024-04-03T16:53:30.7318426Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyFaultyAgentDistAutogradTest::test_verify_backend_options 2024-04-03T16:53:30.7320053Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_remote_timeout_to_here_in_jit 2024-04-03T16:53:30.7321518Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_rref_timeout_pickle_in_jit 2024-04-03T16:53:30.7323197Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_rref_timeout_pickle_script_func 2024-04-03T16:53:30.7324690Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_rref_to_here_timeout_in_jit 2024-04-03T16:53:30.7326101Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_timeout_in_python 2024-04-03T16:53:30.7327529Z Running 1 items in this shard: test/distributed/rpc/test_faulty_agent.py::FaultyJitFaultyAgentRpcTest::test_timeout_in_torchscript_function 2024-04-03T16:53:30.7328371Z 2024-04-03T16:53:30.7328907Z Running distributed/pipeline/sync/skip/test_portal 1/1 ... [2024-04-03 16:53:30.726255] 2024-04-03T16:53:30.7330966Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_portal.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:53:30.726663] 2024-04-03T16:53:32.4478817Z 2024-04-03T16:53:32.4480626Z distributed/pipeline/sync/skip/test_portal 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_portal_1.1_b3d81fa8d1acf95a_.log 2024-04-03T16:53:32.4487123Z Running 10 items in this shard: test/distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad, test/distributed/pipeline/sync/skip/test_portal.py::test_use_grad, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 2024-04-03T16:53:32.4492536Z 2024-04-03T16:53:32.4493043Z Running distributed/_shard/sharded_tensor/ops/test_init 1/1 ... [2024-04-03 16:53:32.448295] 2024-04-03T16:53:32.4495125Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:53:32.448730] 2024-04-03T16:53:34.8703795Z 2024-04-03T16:53:34.8705811Z distributed/_shard/sharded_tensor/ops/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_init_1.1_c4edff395dec7829_.log 2024-04-03T16:53:34.8709372Z Running 3 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_kaiming_uniform, test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_normal, test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_uniform 2024-04-03T16:53:34.8711696Z 2024-04-03T16:53:34.8712135Z Running distributed/rpc/test_tensorpipe_agent 1/3 ... [2024-04-03 16:53:34.870789] 2024-04-03T16:53:34.8715244Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/rpc/test_tensorpipe_agent.py', '--shard-id=1', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 16:53:34.871199] 2024-04-03T17:09:51.0775448Z 2024-04-03T17:09:51.0777567Z distributed/rpc/test_tensorpipe_agent 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.test_tensorpipe_agent_1.3_4d2b8ceccdba7fab_.log 2024-04-03T17:09:51.0895321Z Running 153 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_add, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_add_with_id, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_all_gather, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_all_gather_timeout, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_class_method, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_class_method_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_class_rref_proxy, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_chained, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_multi_chained_async, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_multi_chained_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_multi_fanout, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_multi_fanout_async, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_nested, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_nested_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_raise, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_raise_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_simple, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_with_future_ctor, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_wrong_return_type, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_record_function_double_end_callbacks, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_static_method, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_builtin_remote_self, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_chain, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_in_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_wrong_arg_num, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_cannot_infer_backend_from_options, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_custom_exception_throw_during_reconstruction, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_deadlock, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_debug_info, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_expected_src, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_future_nested_callback, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_init_pg_then_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_local_shutdown_with_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_local_value_not_on_owner, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_mark_future_twice, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_multi_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_my_parameter_server, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_nested_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_pass_local_rrefs, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_pg_init_no_rpc_init, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_rpc_key_names, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_rpc_memory, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_autograd_context_single_threaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_remote_builtin_single_threaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_async_rpc_single_threaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_sync_rpc_udf, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_sync_rpc_udf_single_threaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_class_constructor, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_class_method, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_function_exception, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_no_return_result, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_raise_in_user_func, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_raise_in_user_func_escaped_str, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_tensors_in_container, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_remote_same_worker, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_remote_throw, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_remote_with_exception, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_return_local_rrefs, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_barrier_multithreaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_barrier_partial_subset, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_barrier_subset, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_profiling_async_function, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_proxy_non_exist, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_proxy_reuse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_proxy_tensor_self, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_py_pickle_not_supported, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_timeout, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_blocking, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_non_blocking, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_owner_non_blocking, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_with_error_non_blocking, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_self_py_udf_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_self_remote_rref_as_remote_arg, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_stress_heavy_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_stress_heavy_rpc_torchscript, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_stress_light_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_user_rref_backward, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_user_rrefs_confirmed, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_user_rrefs_confirmed_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_exit_early_builtin, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_exit_early_script_function, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_multiple_call, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_raise_in_body, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_timeout, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wrong_types, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_async_dist_autograd, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_invalid_args, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_no_grad_on_tensor, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_rref, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_simple, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_simple_python_udf, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_without_context, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_without_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backwards_nested_python_udf, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_clean_context_during_backward, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_debug_info, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_error_in_context, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_grad_copy_sparse_indices_extra_ref, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_grad_only_on_return_value, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_builtin_remote_call, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_py_nested_call_itself, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_multiple_backward, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_nested_backward_accumulate_grads, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_post_hooks, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistOptimizerTest::test_dist_optim, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistOptimizerTest::test_dist_optim_exception, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistOptimizerTest::test_dist_optim_exception_on_constructor, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_all_kwargs_are_populated_by_defaults, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_args_kwargs_are_neither_passed, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_function_simple, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_function_wrong_return_type, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_script_throw, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_script_udf, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_callback_simple, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_callback_with_exception, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_kwargs_not_passed, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_less_than_needed_args_are_specified, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_more_than_needed_args_are_specified, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_no_kwargs_are_populated_by_defaults, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_remote_script_udf, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_return_local_script_module_rref_in_py_and_use_in_script, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_rpc_torchscript_record_function, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_rref_is_owner, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_rref_jit_pickle_not_supported, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_some_kwargs_are_populated_by_defaults, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_torchscript_function, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_torchscript_function_exception, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_user_rrefs_confirmed, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_bad_module, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_forward_sync, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_remote_module_py_pickle_not_supported, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_unsupported_methods, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDdpUnderDistAutogradTest::test_backward_ddp_outside, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDdpUnderDistAutogradTest::test_backward_ddp_outside_uneven_inputs, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDdpComparisonTest::test_ddp_dist_autograd_sparse_grads, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeReinforcementLearningRpcTest::test_rl_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_dynamic_rpc_new_rank_can_communicated_with_existing_rank, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_mismatched_type_for_options, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_multi_builtin_remote_ret_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_multi_rpc_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_nested_rref_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_py_rref_args_user_share_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_rref_get_type_timeout_non_blocking, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_wait_all_workers_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_world_size_one_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_rref_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_context_cleanup_nested_rpc_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_context_cleanup_tensor_with_grad_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_graph_for_py_nested_remote_call_itself_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_graph_for_python_call_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_no_graph_with_tensors_not_require_grad_remote_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_rpc_complex_args_sparse 2024-04-03T17:09:51.0980257Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_add 2024-04-03T17:09:51.0981459Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_add_with_id 2024-04-03T17:09:51.0982676Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_all_gather 2024-04-03T17:09:51.0984056Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_all_gather_timeout 2024-04-03T17:09:51.0985417Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_class_method 2024-04-03T17:09:51.0986742Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_class_method_remote 2024-04-03T17:09:51.0988095Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_class_rref_proxy 2024-04-03T17:09:51.0989413Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_chained 2024-04-03T17:09:51.0990784Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_multi_chained_async 2024-04-03T17:09:51.0992251Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_multi_chained_remote 2024-04-03T17:09:51.0993673Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_multi_fanout 2024-04-03T17:09:51.0995100Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_multi_fanout_async 2024-04-03T17:09:51.0996611Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_nested 2024-04-03T17:09:51.0997974Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_nested_remote 2024-04-03T17:09:51.0999324Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_raise 2024-04-03T17:09:51.1000657Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_raise_remote 2024-04-03T17:09:51.1002004Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_simple 2024-04-03T17:09:51.1003445Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_with_future_ctor 2024-04-03T17:09:51.1004866Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_wrong_return_type 2024-04-03T17:09:51.1006334Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_record_function_double_end_callbacks 2024-04-03T17:09:51.1007746Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_static_method 2024-04-03T17:09:51.1009039Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_builtin_remote_self 2024-04-03T17:09:51.1010311Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_chain 2024-04-03T17:09:51.1011551Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_in_rpc 2024-04-03T17:09:51.1012855Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_wrong_arg_num 2024-04-03T17:09:51.1014247Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_cannot_infer_backend_from_options 2024-04-03T17:09:51.1015765Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_custom_exception_throw_during_reconstruction 2024-04-03T17:09:51.1017125Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_deadlock 2024-04-03T17:09:51.1018322Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_debug_info 2024-04-03T17:09:51.1019812Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_expected_src 2024-04-03T17:09:51.1021167Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_future_nested_callback 2024-04-03T17:09:51.1022462Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_init_pg_then_rpc 2024-04-03T17:09:51.1023768Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_local_shutdown_with_rpc 2024-04-03T17:09:51.1025119Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_local_value_not_on_owner 2024-04-03T17:09:51.1026421Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_mark_future_twice 2024-04-03T17:09:51.1027660Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_multi_rpc 2024-04-03T17:09:51.1028918Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_my_parameter_server 2024-04-03T17:09:51.1030180Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_nested_rpc 2024-04-03T17:09:51.1031401Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_pass_local_rrefs 2024-04-03T17:09:51.1032681Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_pg_init_no_rpc_init 2024-04-03T17:09:51.1034000Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_rpc_key_names 2024-04-03T17:09:51.1035310Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_rpc_memory 2024-04-03T17:09:51.1036885Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_autograd_context_single_threaded 2024-04-03T17:09:51.1038446Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_remote_builtin_single_threaded 2024-04-03T17:09:51.1040010Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_async_rpc_single_threaded 2024-04-03T17:09:51.1041480Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_sync_rpc_udf 2024-04-03T17:09:51.1042988Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_sync_rpc_udf_single_threaded 2024-04-03T17:09:51.1044403Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_class_constructor 2024-04-03T17:09:51.1045693Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_class_method 2024-04-03T17:09:51.1046969Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_function_exception 2024-04-03T17:09:51.1048285Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_no_return_result 2024-04-03T17:09:51.1049592Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_raise_in_user_func 2024-04-03T17:09:51.1050968Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_raise_in_user_func_escaped_str 2024-04-03T17:09:51.1052341Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_tensors_in_container 2024-04-03T17:09:51.1053649Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_remote_same_worker 2024-04-03T17:09:51.1055048Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_remote_throw 2024-04-03T17:09:51.1056318Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_remote_with_exception 2024-04-03T17:09:51.1057689Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_return_local_rrefs 2024-04-03T17:09:51.1059015Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_barrier_multithreaded 2024-04-03T17:09:51.1060384Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_barrier_partial_subset 2024-04-03T17:09:51.1061707Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_barrier_subset 2024-04-03T17:09:51.1063034Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_profiling_async_function 2024-04-03T17:09:51.1064378Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_proxy_non_exist 2024-04-03T17:09:51.1065662Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_proxy_reuse 2024-04-03T17:09:51.1066945Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_proxy_tensor_self 2024-04-03T17:09:51.1068300Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_py_pickle_not_supported 2024-04-03T17:09:51.1069598Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_timeout 2024-04-03T17:09:51.1070854Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_blocking 2024-04-03T17:09:51.1072138Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_non_blocking 2024-04-03T17:09:51.1073494Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_owner_non_blocking 2024-04-03T17:09:51.1074908Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_with_error_non_blocking 2024-04-03T17:09:51.1076432Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_self_py_udf_remote 2024-04-03T17:09:51.1077771Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_self_remote_rref_as_remote_arg 2024-04-03T17:09:51.1079104Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_stress_heavy_rpc 2024-04-03T17:09:51.1080426Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_stress_heavy_rpc_torchscript 2024-04-03T17:09:51.1081769Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_stress_light_rpc 2024-04-03T17:09:51.1083097Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_user_rref_backward 2024-04-03T17:09:51.1084398Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_user_rrefs_confirmed 2024-04-03T17:09:51.1085740Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_user_rrefs_confirmed_remote 2024-04-03T17:09:51.1087094Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_exit_early_builtin 2024-04-03T17:09:51.1088502Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_exit_early_script_function 2024-04-03T17:09:51.1089913Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_multiple_call 2024-04-03T17:09:51.1091344Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_raise_in_body 2024-04-03T17:09:51.1092699Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_timeout 2024-04-03T17:09:51.1093931Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wrong_types 2024-04-03T17:09:51.1095242Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_async_dist_autograd 2024-04-03T17:09:51.1096663Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_invalid_args 2024-04-03T17:09:51.1098104Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_no_grad_on_tensor 2024-04-03T17:09:51.1099530Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_rref 2024-04-03T17:09:51.1100889Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_simple 2024-04-03T17:09:51.1102317Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_simple_python_udf 2024-04-03T17:09:51.1103786Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_without_context 2024-04-03T17:09:51.1105227Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_without_rpc 2024-04-03T17:09:51.1106699Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backwards_nested_python_udf 2024-04-03T17:09:51.1108207Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_clean_context_during_backward 2024-04-03T17:09:51.1109609Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_debug_info 2024-04-03T17:09:51.1110960Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_error_in_context 2024-04-03T17:09:51.1112429Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_grad_copy_sparse_indices_extra_ref 2024-04-03T17:09:51.1113937Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_grad_only_on_return_value 2024-04-03T17:09:51.1115426Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_builtin_remote_call 2024-04-03T17:09:51.1117060Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_py_nested_call_itself 2024-04-03T17:09:51.1118542Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_multiple_backward 2024-04-03T17:09:51.1120008Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_nested_backward_accumulate_grads 2024-04-03T17:09:51.1121436Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_post_hooks 2024-04-03T17:09:51.1122822Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistOptimizerTest::test_dist_optim 2024-04-03T17:09:51.1124219Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistOptimizerTest::test_dist_optim_exception 2024-04-03T17:09:51.1125729Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistOptimizerTest::test_dist_optim_exception_on_constructor 2024-04-03T17:09:51.1127375Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_all_kwargs_are_populated_by_defaults 2024-04-03T17:09:51.1128927Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_args_kwargs_are_neither_passed 2024-04-03T17:09:51.1130337Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_function_simple 2024-04-03T17:09:51.1131739Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_function_wrong_return_type 2024-04-03T17:09:51.1133137Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_script_throw 2024-04-03T17:09:51.1134458Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_script_udf 2024-04-03T17:09:51.1135765Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_callback_simple 2024-04-03T17:09:51.1137091Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_callback_with_exception 2024-04-03T17:09:51.1138435Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_kwargs_not_passed 2024-04-03T17:09:51.1139839Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_less_than_needed_args_are_specified 2024-04-03T17:09:51.1141329Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_more_than_needed_args_are_specified 2024-04-03T17:09:51.1142812Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_no_kwargs_are_populated_by_defaults 2024-04-03T17:09:51.1144207Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_remote_script_udf 2024-04-03T17:09:51.1145730Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_return_local_script_module_rref_in_py_and_use_in_script 2024-04-03T17:09:51.1147298Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_rpc_torchscript_record_function 2024-04-03T17:09:51.1148671Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_rref_is_owner 2024-04-03T17:09:51.1150028Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_rref_jit_pickle_not_supported 2024-04-03T17:09:51.1151507Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_some_kwargs_are_populated_by_defaults 2024-04-03T17:09:51.1152925Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_torchscript_function 2024-04-03T17:09:51.1154340Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_torchscript_function_exception 2024-04-03T17:09:51.1155848Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_user_rrefs_confirmed 2024-04-03T17:09:51.1157182Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_bad_module 2024-04-03T17:09:51.1158491Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_forward_sync 2024-04-03T17:09:51.1159967Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_remote_module_py_pickle_not_supported 2024-04-03T17:09:51.1161478Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_unsupported_methods 2024-04-03T17:09:51.1163119Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDdpUnderDistAutogradTest::test_backward_ddp_outside 2024-04-03T17:09:51.1164726Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDdpUnderDistAutogradTest::test_backward_ddp_outside_uneven_inputs 2024-04-03T17:09:51.1166410Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDdpComparisonTest::test_ddp_dist_autograd_sparse_grads 2024-04-03T17:09:51.1167901Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeReinforcementLearningRpcTest::test_rl_rpc 2024-04-03T17:09:51.1169579Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda 2024-04-03T17:09:51.1171443Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_dynamic_rpc_new_rank_can_communicated_with_existing_rank 2024-04-03T17:09:51.1173177Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_mismatched_type_for_options 2024-04-03T17:09:51.1174786Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_multi_builtin_remote_ret_sparse 2024-04-03T17:09:51.1176322Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_multi_rpc_sparse 2024-04-03T17:09:51.1177787Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_nested_rref_sparse 2024-04-03T17:09:51.1179330Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_py_rref_args_user_share_sparse 2024-04-03T17:09:51.1180951Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_rref_get_type_timeout_non_blocking 2024-04-03T17:09:51.1182542Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_wait_all_workers_sparse 2024-04-03T17:09:51.1184063Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_world_size_one_sparse 2024-04-03T17:09:51.1185632Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_rref_sparse 2024-04-03T17:09:51.1187334Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_context_cleanup_nested_rpc_sparse 2024-04-03T17:09:51.1189119Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_context_cleanup_tensor_with_grad_sparse 2024-04-03T17:09:51.1190956Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_graph_for_py_nested_remote_call_itself_sparse 2024-04-03T17:09:51.1192755Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_graph_for_python_call_sparse 2024-04-03T17:09:51.1194588Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_no_graph_with_tensors_not_require_grad_remote_sparse 2024-04-03T17:09:51.1196579Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_rpc_complex_args_sparse 2024-04-03T17:09:51.1197494Z 2024-04-03T17:09:51.1198017Z Running distributed/rpc/test_tensorpipe_agent 3/3 ... [2024-04-03 17:09:51.085942] 2024-04-03T17:09:51.1200164Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/rpc/test_tensorpipe_agent.py', '--shard-id=3', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:09:51.086358] 2024-04-03T17:27:48.0877213Z 2024-04-03T17:27:48.0878778Z distributed/rpc/test_tensorpipe_agent 3/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.test_tensorpipe_agent_3.3_3f194c313be972fb_.log 2024-04-03T17:27:48.1064170Z Running 167 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_add_done_callback, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_add, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_class_rref_proxy_async, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_class_rref_proxy_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_multi_chained, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_multi_fanout_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_with_future_ctor_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_wrong_return_type_async, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_record_function_cbs_jit_call, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_record_function_legacy, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_static_method_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_builtin_remote_ret, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_call_method_on_rref, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_with_error, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_with_ret, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_wrong_arg_type, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_default_timeout_used, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_disable_gil_profiling, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_duplicate_name, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_graceful_shutdown_with_uneven_workload, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_handle_send_exceptions, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_ignore_rref_leak, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_init_rpc_then_pg, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_invalid_names, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_local_rref_no_fork, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_local_shutdown, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_multi_layer_nested_async_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_multi_py_udf_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_nested_rref, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_nested_rref_stress, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_non_cont_tensors, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_nonzero, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_owner_equality, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_pickle_future, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_export_trace, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_remote_events_profiled, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_remote_events_profiled_single_threaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_rpc_record_shapes, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_async_rpc_builtin, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_async_rpc_udf, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_async_rpc_udf_single_threaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_autograd_context, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_remote_builtin, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_remote_udf_single_threaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_remote_rpc_single_threaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_sync_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_sync_rpc_single_threaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_built_in, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_class_static_method, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_rref_args, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_rref_args_user_share, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_tensors, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_tensors_multi_async_call, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_udf_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_user_defined, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_register_rpc_backend_and_set_and_start_rpc_backend, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_return_future, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_return_future_async, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_return_future_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_barrier_all, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_profiling_async_function_single_threaded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_profiling_remote_record_function, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_return_rref, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_timeouts, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_leak, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_proxy_class, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_proxy_class_self, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_proxy_tensor, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_str, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_owner_blocking, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_with_error_blocking, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_self_remote_rref_as_self_remote_arg, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_self_remote_rref_as_self_rpc_arg, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_server_process_global_profiler, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_set_and_get_default_rpc_timeout, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_shutdown_errors, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_use_rpc_pickler, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_exit_early_python, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_with_exception, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_workers_timeout, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_worker_id, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeParameterServerTest::test_batch_updating_parameter_server, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_accumulate_grads, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_autograd_engine_error, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_complex_python_udf, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_different_dtypes, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_multiple_output_tensors, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_multiple_roots, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_node_failure_python_udf, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_python_udf_error, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_rref_multi, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_unused_tensors, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_verify_hooks, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_context_cleanup_nested_rpc, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_context_cleanup_no_tensors, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_context_cleanup_tensor_no_grad, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_builtin_call, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_py_nested_remote_call, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_py_nested_remote_call_itself, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_python_call, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_mixed_requires_grad, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_multiple_backward_with_errors, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_no_grad_copy, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_no_grad_copy_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_no_graph_with_tensors_not_require_grad, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_no_graph_with_tensors_not_require_grad_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_remote_complex_args, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_thread_local_context_id, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_trainer_ps, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_trainer_ps_torchscript_functions, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_worker_ids_recorded, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_function_wrong_decorator_order, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_function_wrong_return_type_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_call_python_function_remotely_from_script_not_supported, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_call_rpc_with_profiling, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_call_script_function_that_raises_remotely_from_script, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_create_local_script_class_rref_in_py, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_create_local_script_module_rref_in_py, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_create_script_module_on_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_future_passed_between_python_and_jit, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_future_python_annotation, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_load_script_module_with_pickled_rref, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_my_script_module_with_rrefs, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_record_function_on_caller_rpc_async, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_return_local_script_class_rref_in_py_and_use_in_script, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_rpc_async_jit_profiled, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_rref_list_mutate, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_rref_local_value, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_rref_python_annotation, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_user_rrefs_confirmed_remote, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_forward_async, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_forward_async_script, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_forward_with_kwargs, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_get_module_rref, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_remote_module_py_pickle_not_supported_script, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_remote_parameters, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDdpComparisonTest::test_ddp_comparison, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDdpComparisonTest::test_ddp_comparison_uneven_inputs, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_builtin_remote_self_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_dynamic_and_static_init_rpc_together, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_dynamic_rpc_existing_rank_can_communicate_with_new_rank, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_infer_backend_from_options, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_nested_rpc_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_op_with_invalid_args, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_py_sparse_tensors_in_container, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_self_remote_rref_as_remote_arg_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_self_remote_rref_as_rpc_arg_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_self_remote_rref_as_self_rpc_arg_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_send_to_rank_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_stress_heavy_rpc_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_tensorpipe_options_throw_on_timedelta_timeout, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_tensorpipe_set_default_timeout, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_wait_all_workers_twice_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_different_dtypes_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_multiple_round_trips_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_rref_nested_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_simple_python_udf_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_simple_script_call_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backwards_nested_python_udf_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_embedding_bag_with_no_grad_tensors, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_graph_for_builtin_remote_call_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_graph_for_py_nested_call_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_graph_for_py_nested_remote_call_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_graph_for_python_remote_call_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_mixed_requires_grad_sparse, test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_remote_complex_args_sparse 2024-04-03T17:27:48.1165429Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_add_done_callback 2024-04-03T17:27:48.1166707Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_add 2024-04-03T17:27:48.1168013Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_class_rref_proxy_async 2024-04-03T17:27:48.1169445Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_class_rref_proxy_remote 2024-04-03T17:27:48.1170831Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_multi_chained 2024-04-03T17:27:48.1172282Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_multi_fanout_remote 2024-04-03T17:27:48.1173826Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_with_future_ctor_remote 2024-04-03T17:27:48.1175315Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_function_wrong_return_type_async 2024-04-03T17:27:48.1176800Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_record_function_cbs_jit_call 2024-04-03T17:27:48.1178222Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_record_function_legacy 2024-04-03T17:27:48.1179615Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_async_static_method_remote 2024-04-03T17:27:48.1180947Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_builtin_remote_ret 2024-04-03T17:27:48.1182244Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_call_method_on_rref 2024-04-03T17:27:48.1183536Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_with_error 2024-04-03T17:27:48.1184822Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_with_ret 2024-04-03T17:27:48.1186122Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_callback_wrong_arg_type 2024-04-03T17:27:48.1187711Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_default_timeout_used 2024-04-03T17:27:48.1189025Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_disable_gil_profiling 2024-04-03T17:27:48.1190406Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_duplicate_name 2024-04-03T17:27:48.1191770Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_graceful_shutdown_with_uneven_workload 2024-04-03T17:27:48.1193175Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_handle_send_exceptions 2024-04-03T17:27:48.1194480Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_ignore_rref_leak 2024-04-03T17:27:48.1195893Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_init_rpc_then_pg 2024-04-03T17:27:48.1197149Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_invalid_names 2024-04-03T17:27:48.1198413Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_local_rref_no_fork 2024-04-03T17:27:48.1199688Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_local_shutdown 2024-04-03T17:27:48.1201005Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_multi_layer_nested_async_rpc 2024-04-03T17:27:48.1202346Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_multi_py_udf_remote 2024-04-03T17:27:48.1203602Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_nested_rref 2024-04-03T17:27:48.1204848Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_nested_rref_stress 2024-04-03T17:27:48.1206112Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_non_cont_tensors 2024-04-03T17:27:48.1207327Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_nonzero 2024-04-03T17:27:48.1208539Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_owner_equality 2024-04-03T17:27:48.1209772Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_pickle_future 2024-04-03T17:27:48.1211035Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_export_trace 2024-04-03T17:27:48.1212408Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_remote_events_profiled 2024-04-03T17:27:48.1213974Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_remote_events_profiled_single_threaded 2024-04-03T17:27:48.1215447Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_rpc_record_shapes 2024-04-03T17:27:48.1217055Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_async_rpc_builtin 2024-04-03T17:27:48.1218460Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_async_rpc_udf 2024-04-03T17:27:48.1219911Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_async_rpc_udf_single_threaded 2024-04-03T17:27:48.1221449Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_autograd_context 2024-04-03T17:27:48.1222845Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_remote_builtin 2024-04-03T17:27:48.1224424Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_remote_udf_single_threaded 2024-04-03T17:27:48.1226073Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_remote_rpc_single_threaded 2024-04-03T17:27:48.1227559Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_sync_rpc 2024-04-03T17:27:48.1229045Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_profiler_with_script_sync_rpc_single_threaded 2024-04-03T17:27:48.1230438Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_built_in 2024-04-03T17:27:48.1231717Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_class_static_method 2024-04-03T17:27:48.1232978Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_rref_args 2024-04-03T17:27:48.1234255Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_rref_args_user_share 2024-04-03T17:27:48.1235679Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_tensors 2024-04-03T17:27:48.1237000Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_tensors_multi_async_call 2024-04-03T17:27:48.1238296Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_udf_remote 2024-04-03T17:27:48.1239537Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_py_user_defined 2024-04-03T17:27:48.1240979Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_register_rpc_backend_and_set_and_start_rpc_backend 2024-04-03T17:27:48.1242399Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_return_future 2024-04-03T17:27:48.1243664Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_return_future_async 2024-04-03T17:27:48.1244958Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_return_future_remote 2024-04-03T17:27:48.1246233Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_barrier_all 2024-04-03T17:27:48.1247633Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_profiling_async_function_single_threaded 2024-04-03T17:27:48.1249148Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_profiling_remote_record_function 2024-04-03T17:27:48.1250515Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_return_rref 2024-04-03T17:27:48.1251752Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rpc_timeouts 2024-04-03T17:27:48.1252950Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_leak 2024-04-03T17:27:48.1254235Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_proxy_class 2024-04-03T17:27:48.1255527Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_proxy_class_self 2024-04-03T17:27:48.1256819Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_proxy_tensor 2024-04-03T17:27:48.1258034Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_str 2024-04-03T17:27:48.1259439Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_owner_blocking 2024-04-03T17:27:48.1260814Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_rref_type_with_error_blocking 2024-04-03T17:27:48.1262312Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_self_remote_rref_as_self_remote_arg 2024-04-03T17:27:48.1263760Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_self_remote_rref_as_self_rpc_arg 2024-04-03T17:27:48.1265186Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_server_process_global_profiler 2024-04-03T17:27:48.1266600Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_set_and_get_default_rpc_timeout 2024-04-03T17:27:48.1267927Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_shutdown_errors 2024-04-03T17:27:48.1269189Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_use_rpc_pickler 2024-04-03T17:27:48.1270403Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all 2024-04-03T17:27:48.1271684Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_exit_early_python 2024-04-03T17:27:48.1273025Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_with_exception 2024-04-03T17:27:48.1274360Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_wait_all_workers_timeout 2024-04-03T17:27:48.1275790Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRpcTest::test_worker_id 2024-04-03T17:27:48.1277201Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeParameterServerTest::test_batch_updating_parameter_server 2024-04-03T17:27:48.1278726Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_accumulate_grads 2024-04-03T17:27:48.1280229Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_autograd_engine_error 2024-04-03T17:27:48.1281758Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_complex_python_udf 2024-04-03T17:27:48.1283255Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_different_dtypes 2024-04-03T17:27:48.1284747Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_multiple_output_tensors 2024-04-03T17:27:48.1286255Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_multiple_roots 2024-04-03T17:27:48.1287766Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_node_failure_python_udf 2024-04-03T17:27:48.1289278Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_python_udf_error 2024-04-03T17:27:48.1290725Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_rref_multi 2024-04-03T17:27:48.1292156Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_unused_tensors 2024-04-03T17:27:48.1293672Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_backward_verify_hooks 2024-04-03T17:27:48.1295121Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_context_cleanup_nested_rpc 2024-04-03T17:27:48.1296731Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_context_cleanup_no_tensors 2024-04-03T17:27:48.1298318Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_context_cleanup_tensor_no_grad 2024-04-03T17:27:48.1299810Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_builtin_call 2024-04-03T17:27:48.1301298Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_py_nested_remote_call 2024-04-03T17:27:48.1302873Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_py_nested_remote_call_itself 2024-04-03T17:27:48.1304383Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_graph_for_python_call 2024-04-03T17:27:48.1305820Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_mixed_requires_grad 2024-04-03T17:27:48.1307276Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_multiple_backward_with_errors 2024-04-03T17:27:48.1308698Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_no_grad_copy 2024-04-03T17:27:48.1310064Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_no_grad_copy_sparse 2024-04-03T17:27:48.1311567Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_no_graph_with_tensors_not_require_grad 2024-04-03T17:27:48.1313193Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_no_graph_with_tensors_not_require_grad_remote 2024-04-03T17:27:48.1314744Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_remote_complex_args 2024-04-03T17:27:48.1316368Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_thread_local_context_id 2024-04-03T17:27:48.1317757Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_trainer_ps 2024-04-03T17:27:48.1319182Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_trainer_ps_torchscript_functions 2024-04-03T17:27:48.1320668Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDistAutogradTest::test_worker_ids_recorded 2024-04-03T17:27:48.1322132Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_function_wrong_decorator_order 2024-04-03T17:27:48.1323664Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_async_function_wrong_return_type_remote 2024-04-03T17:27:48.1325271Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_call_python_function_remotely_from_script_not_supported 2024-04-03T17:27:48.1326819Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_call_rpc_with_profiling 2024-04-03T17:27:48.1328339Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_call_script_function_that_raises_remotely_from_script 2024-04-03T17:27:48.1329931Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_create_local_script_class_rref_in_py 2024-04-03T17:27:48.1331430Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_create_local_script_module_rref_in_py 2024-04-03T17:27:48.1333016Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_create_script_module_on_remote 2024-04-03T17:27:48.1334568Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_future_passed_between_python_and_jit 2024-04-03T17:27:48.1336106Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_future_python_annotation 2024-04-03T17:27:48.1337550Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_load_script_module_with_pickled_rref 2024-04-03T17:27:48.1339009Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_my_script_module_with_rrefs 2024-04-03T17:27:48.1340477Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_record_function_on_caller_rpc_async 2024-04-03T17:27:48.1342069Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_return_local_script_class_rref_in_py_and_use_in_script 2024-04-03T17:27:48.1343569Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_rpc_async_jit_profiled 2024-04-03T17:27:48.1344905Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_rref_list_mutate 2024-04-03T17:27:48.1346203Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_rref_local_value 2024-04-03T17:27:48.1347530Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_rref_python_annotation 2024-04-03T17:27:48.1348927Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeJitRpcTest::test_user_rrefs_confirmed_remote 2024-04-03T17:27:48.1350311Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_forward_async 2024-04-03T17:27:48.1351714Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_forward_async_script 2024-04-03T17:27:48.1353119Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_forward_with_kwargs 2024-04-03T17:27:48.1354513Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_get_module_rref 2024-04-03T17:27:48.1356135Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_remote_module_py_pickle_not_supported_script 2024-04-03T17:27:48.1357673Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeRemoteModuleTest::test_remote_parameters 2024-04-03T17:27:48.1359045Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDdpComparisonTest::test_ddp_comparison 2024-04-03T17:27:48.1360500Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeDdpComparisonTest::test_ddp_comparison_uneven_inputs 2024-04-03T17:27:48.1362058Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_builtin_remote_self_sparse 2024-04-03T17:27:48.1363683Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_dynamic_and_static_init_rpc_together 2024-04-03T17:27:48.1365434Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_dynamic_rpc_existing_rank_can_communicate_with_new_rank 2024-04-03T17:27:48.1367141Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_infer_backend_from_options 2024-04-03T17:27:48.1368663Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_nested_rpc_sparse 2024-04-03T17:27:48.1370243Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_op_with_invalid_args 2024-04-03T17:27:48.1371864Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_py_sparse_tensors_in_container 2024-04-03T17:27:48.1373593Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_self_remote_rref_as_remote_arg_sparse 2024-04-03T17:27:48.1375253Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_self_remote_rref_as_rpc_arg_sparse 2024-04-03T17:27:48.1376934Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_self_remote_rref_as_self_rpc_arg_sparse 2024-04-03T17:27:48.1378526Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_send_to_rank_sparse 2024-04-03T17:27:48.1380046Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_stress_heavy_rpc_sparse 2024-04-03T17:27:48.1381706Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_tensorpipe_options_throw_on_timedelta_timeout 2024-04-03T17:27:48.1383395Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_tensorpipe_set_default_timeout 2024-04-03T17:27:48.1384991Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentRpcTest::test_wait_all_workers_twice_sparse 2024-04-03T17:27:48.1386670Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_different_dtypes_sparse 2024-04-03T17:27:48.1388446Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_multiple_round_trips_sparse 2024-04-03T17:27:48.1390196Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_rref_nested_sparse 2024-04-03T17:27:48.1391914Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_simple_python_udf_sparse 2024-04-03T17:27:48.1393684Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backward_simple_script_call_sparse 2024-04-03T17:27:48.1395577Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_backwards_nested_python_udf_sparse 2024-04-03T17:27:48.1397419Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_embedding_bag_with_no_grad_tensors 2024-04-03T17:27:48.1399176Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_graph_for_builtin_remote_call_sparse 2024-04-03T17:27:48.1400946Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_graph_for_py_nested_call_sparse 2024-04-03T17:27:48.1402721Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_graph_for_py_nested_remote_call_sparse 2024-04-03T17:27:48.1404529Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_graph_for_python_remote_call_sparse 2024-04-03T17:27:48.1406253Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_mixed_requires_grad_sparse 2024-04-03T17:27:48.1408056Z Running 1 items in this shard: test/distributed/rpc/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentDistAutogradTest::test_remote_complex_args_sparse 2024-04-03T17:27:48.1409075Z 2024-04-03T17:27:48.1409514Z Running distributed/test_nccl 1/1 ... [2024-04-03 17:27:48.098686] 2024-04-03T17:27:48.1411288Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:27:48.099245] 2024-04-03T17:27:49.9442102Z 2024-04-03T17:27:49.9443614Z distributed/test_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_nccl_1.1_797bae420066f849_.log 2024-04-03T17:27:49.9444717Z Running 0 items in this shard: 2024-04-03T17:27:49.9445039Z 2024-04-03T17:27:49.9448603Z Running distributed/pipeline/sync/skip/test_verify_skippables 1/1 ... [2024-04-03 17:27:49.944645] 2024-04-03T17:27:49.9454118Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_verify_skippables.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:27:49.945085] 2024-04-03T17:27:51.6656263Z 2024-04-03T17:27:51.6658303Z distributed/pipeline/sync/skip/test_verify_skippables 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_verify_skippables_1.1_8de2ff51501c1f1d_.log 2024-04-03T17:27:51.6664419Z Running 9 items in this shard: test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated 2024-04-03T17:27:51.6669332Z 2024-04-03T17:27:51.6669724Z Running distributed/test_data_parallel 1/1 ... [2024-04-03 17:27:51.666070] 2024-04-03T17:27:51.6671608Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_data_parallel.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:27:51.666477] 2024-04-03T17:27:54.4389420Z 2024-04-03T17:27:54.4391324Z distributed/test_data_parallel 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_data_parallel_1.1_c66b2111ea83ccb8_.log 2024-04-03T17:27:54.4418012Z Running 46 items in this shard: test/distributed/test_data_parallel.py::TestDataParallel::test_autocast, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_buffers_requiring_grad, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_complex, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_device_args, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_function_deletion, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_lazy_linear, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_model_device, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_model_no_refcycles, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_module_zero_inputs, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_multiple_input, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_nested_input, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_nested_output, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_no_grad, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_rnn, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_small_back, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_sparse, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_cpu, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_different_len_dicts, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_gpu, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply_autocast, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply_passes_exception, test/distributed/test_data_parallel.py::TestDataParallel::test_parameter_list_dict_replica, test/distributed/test_data_parallel.py::TestDataParallel::test_replicate, test/distributed/test_data_parallel.py::TestDataParallel::test_replicate_buffers, test/distributed/test_data_parallel.py::TestDataParallel::test_save_replica_module, test/distributed/test_data_parallel.py::TestDataParallel::test_scatter_cpu, test/distributed/test_data_parallel.py::TestDataParallel::test_scatter_gpu, test/distributed/test_data_parallel.py::TestDataParallel::test_strided_grad_layout, test/distributed/test_data_parallel.py::TestDataParallel::test_zero_grad, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCPU::test_data_parallel_module_cpu_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCPU::test_data_parallel_module_cpu_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCPU::test_data_parallel_module_cpu_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCPU::test_data_parallel_module_kwargs_only_cpu_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCPU::test_data_parallel_module_kwargs_only_cpu_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCPU::test_data_parallel_module_kwargs_only_cpu_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCPU::test_data_parallel_module_kwargs_only_empty_dict_cpu_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCPU::test_data_parallel_module_kwargs_only_empty_dict_cpu_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCPU::test_data_parallel_module_kwargs_only_empty_dict_cpu_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCPU::test_data_parallel_module_kwargs_only_empty_list_cpu_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCPU::test_data_parallel_module_kwargs_only_empty_list_cpu_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCPU::test_data_parallel_module_kwargs_only_empty_list_cpu_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCPU::test_data_parallel_module_kwargs_only_empty_tuple_cpu_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCPU::test_data_parallel_module_kwargs_only_empty_tuple_cpu_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCPU::test_data_parallel_module_kwargs_only_empty_tuple_cpu_float64 2024-04-03T17:27:54.4442324Z 2024-04-03T17:27:54.4442727Z Running distributed/test_store 1/1 ... [2024-04-03 17:27:54.439333] 2024-04-03T17:27:54.4444695Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_store.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:27:54.439759] 2024-04-03T17:34:00.6565900Z 2024-04-03T17:34:00.6567256Z distributed/test_store 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_store_1.1_a2edbf29f2c8642f_.log 2024-04-03T17:34:00.6608480Z Running 85 items in this shard: test/distributed/test_store.py::FileStoreTest::test_append, test/distributed/test_store.py::FileStoreTest::test_compare_set, test/distributed/test_store.py::FileStoreTest::test_init_pg_and_rpc_with_same_file, test/distributed/test_store.py::FileStoreTest::test_multi_get, test/distributed/test_store.py::FileStoreTest::test_multi_set, test/distributed/test_store.py::FileStoreTest::test_refcount, test/distributed/test_store.py::FileStoreTest::test_set_get_check, test/distributed/test_store.py::FileStoreTest::test_simple_wait, test/distributed/test_store.py::HashStoreTest::test_append, test/distributed/test_store.py::HashStoreTest::test_compare_set, test/distributed/test_store.py::HashStoreTest::test_multi_get, test/distributed/test_store.py::HashStoreTest::test_multi_set, test/distributed/test_store.py::HashStoreTest::test_set_get_check, test/distributed/test_store.py::HashStoreTest::test_simple_wait, test/distributed/test_store.py::PrefixStoreTest::test_get_underlying_store, test/distributed/test_store.py::PrefixFileStoreTest::test_append, test/distributed/test_store.py::PrefixFileStoreTest::test_compare_set, test/distributed/test_store.py::PrefixFileStoreTest::test_multi_get, test/distributed/test_store.py::PrefixFileStoreTest::test_multi_set, test/distributed/test_store.py::PrefixFileStoreTest::test_set_get_check, test/distributed/test_store.py::PrefixFileStoreTest::test_simple_wait, test/distributed/test_store.py::TCPStoreTest::test_address_already_in_use, test/distributed/test_store.py::TCPStoreTest::test_append, test/distributed/test_store.py::TCPStoreTest::test_compare_set, test/distributed/test_store.py::TCPStoreTest::test_init_pg_and_rpc_with_same_socket, test/distributed/test_store.py::TCPStoreTest::test_multi_get, test/distributed/test_store.py::TCPStoreTest::test_multi_set, test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_fixed_world_size, test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_nonfixed_world_size, test/distributed/test_store.py::TCPStoreTest::test_multitenancy, test/distributed/test_store.py::TCPStoreTest::test_numkeys_delkeys, test/distributed/test_store.py::TCPStoreTest::test_set_get_check, test/distributed/test_store.py::TCPStoreTest::test_simple_wait, test/distributed/test_store.py::TCPStoreTest::test_store_timeout_on_missing_clients, test/distributed/test_store.py::TCPStoreTest::test_take_over_listen_socket, test/distributed/test_store.py::LibUvTCPStoreTest::test_address_already_in_use, test/distributed/test_store.py::LibUvTCPStoreTest::test_append, test/distributed/test_store.py::LibUvTCPStoreTest::test_compare_set, test/distributed/test_store.py::LibUvTCPStoreTest::test_init_pg_and_rpc_with_same_socket, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_get, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_set, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_fixed_world_size, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_nonfixed_world_size, test/distributed/test_store.py::LibUvTCPStoreTest::test_multitenancy, test/distributed/test_store.py::LibUvTCPStoreTest::test_numkeys_delkeys, test/distributed/test_store.py::LibUvTCPStoreTest::test_set_get_check, test/distributed/test_store.py::LibUvTCPStoreTest::test_simple_wait, test/distributed/test_store.py::LibUvTCPStoreTest::test_store_timeout_on_missing_clients, test/distributed/test_store.py::LibUvTCPStoreTest::test_take_over_listen_socket, test/distributed/test_store.py::PrefixTCPStoreTest::test_append, test/distributed/test_store.py::PrefixTCPStoreTest::test_compare_set, test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_get, test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_set, test/distributed/test_store.py::PrefixTCPStoreTest::test_set_get_check, test/distributed/test_store.py::PrefixTCPStoreTest::test_simple_wait, test/distributed/test_store.py::PrefixTCPStoreTest::test_underlying_non_prefix_store, test/distributed/test_store.py::PythonStoreTest::test_set_get, test/distributed/test_store.py::RendezvousTest::test_unknown_handler, test/distributed/test_store.py::RendezvousTest::test_url_with_node_params, test/distributed/test_store.py::RendezvousEnvTest::test_nominal, test/distributed/test_store.py::RendezvousFileTest::test_common_errors, test/distributed/test_store.py::RendezvousFileTest::test_nominal, test/distributed/test_store.py::RendezvousTCPTest::test_common_errors, test/distributed/test_store.py::RendezvousTCPTest::test_dns_timeout, test/distributed/test_store.py::RendezvousTCPTest::test_nominal, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_doest_break_client, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_set, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_url_with_libuv, test/distributed/test_store.py::TestPythonStore::test_append_roundtrip, test/distributed/test_store.py::TestPythonStore::test_extended_methods_fallbacks, test/distributed/test_store.py::TestPythonStore::test_has_extended_api_passthrough, test/distributed/test_store.py::TestPythonStore::test_has_extended_api_roundtrip, test/distributed/test_store.py::TestPythonStore::test_multi_get_roundtrip, test/distributed/test_store.py::TestPythonStore::test_multi_set_roundtrip, test/distributed/test_store.py::TestPythonStore::test_optional_methods_fail, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_0, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_1, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_2, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_3, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_4, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_5, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_6, test/distributed/test_store.py::TimeoutTest::test_interrupt_doesnt_break_wait, test/distributed/test_store.py::InitPgWithUvStore::test_with_env_var, test/distributed/test_store.py::InitPgWithUvStore::test_with_url_param 2024-04-03T17:34:00.6645856Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_append 2024-04-03T17:34:00.6646899Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_compare_set 2024-04-03T17:34:00.6648000Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_init_pg_and_rpc_with_same_file 2024-04-03T17:34:00.6649076Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_multi_get 2024-04-03T17:34:00.6650052Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_multi_set 2024-04-03T17:34:00.6651017Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_refcount 2024-04-03T17:34:00.6652015Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_set_get_check 2024-04-03T17:34:00.6653039Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_simple_wait 2024-04-03T17:34:00.6654006Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_append 2024-04-03T17:34:00.6654975Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_compare_set 2024-04-03T17:34:00.6655943Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_multi_get 2024-04-03T17:34:00.6656909Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_multi_set 2024-04-03T17:34:00.6657902Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_set_get_check 2024-04-03T17:34:00.6658884Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_simple_wait 2024-04-03T17:34:00.6660094Z Running 1 items in this shard: test/distributed/test_store.py::PrefixStoreTest::test_get_underlying_store 2024-04-03T17:34:00.6661169Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_append 2024-04-03T17:34:00.6662333Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_compare_set 2024-04-03T17:34:00.6663376Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_multi_get 2024-04-03T17:34:00.6664429Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_multi_set 2024-04-03T17:34:00.6665497Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_set_get_check 2024-04-03T17:34:00.6666560Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_simple_wait 2024-04-03T17:34:00.6667643Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_address_already_in_use 2024-04-03T17:34:00.6668656Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_append 2024-04-03T17:34:00.6669608Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_compare_set 2024-04-03T17:34:00.6670684Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_init_pg_and_rpc_with_same_socket 2024-04-03T17:34:00.6671763Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_get 2024-04-03T17:34:00.6672716Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_set 2024-04-03T17:34:00.6673790Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_fixed_world_size 2024-04-03T17:34:00.6675017Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_nonfixed_world_size 2024-04-03T17:34:00.6676309Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multitenancy 2024-04-03T17:34:00.6677322Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_numkeys_delkeys 2024-04-03T17:34:00.6678321Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_set_get_check 2024-04-03T17:34:00.6679310Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_simple_wait 2024-04-03T17:34:00.6680398Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_store_timeout_on_missing_clients 2024-04-03T17:34:00.6681544Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_take_over_listen_socket 2024-04-03T17:34:00.6682747Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_address_already_in_use 2024-04-03T17:34:00.6683806Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_append 2024-04-03T17:34:00.6684823Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_compare_set 2024-04-03T17:34:00.6685957Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_init_pg_and_rpc_with_same_socket 2024-04-03T17:34:00.6687094Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_get 2024-04-03T17:34:00.6688106Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_set 2024-04-03T17:34:00.6689245Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_fixed_world_size 2024-04-03T17:34:00.6690520Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_nonfixed_world_size 2024-04-03T17:34:00.6691701Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multitenancy 2024-04-03T17:34:00.6692772Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_numkeys_delkeys 2024-04-03T17:34:00.6693944Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_set_get_check 2024-04-03T17:34:00.6694996Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_simple_wait 2024-04-03T17:34:00.6696259Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_store_timeout_on_missing_clients 2024-04-03T17:34:00.6697470Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_take_over_listen_socket 2024-04-03T17:34:00.6698537Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_append 2024-04-03T17:34:00.6699565Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_compare_set 2024-04-03T17:34:00.6700608Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_get 2024-04-03T17:34:00.6701621Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_set 2024-04-03T17:34:00.6702670Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_set_get_check 2024-04-03T17:34:00.6703736Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_simple_wait 2024-04-03T17:34:00.6704869Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_underlying_non_prefix_store 2024-04-03T17:34:00.6705948Z Running 1 items in this shard: test/distributed/test_store.py::PythonStoreTest::test_set_get 2024-04-03T17:34:00.6706960Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTest::test_unknown_handler 2024-04-03T17:34:00.6708028Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTest::test_url_with_node_params 2024-04-03T17:34:00.6709061Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousEnvTest::test_nominal 2024-04-03T17:34:00.6710096Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousFileTest::test_common_errors 2024-04-03T17:34:00.6711134Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousFileTest::test_nominal 2024-04-03T17:34:00.6712172Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_common_errors 2024-04-03T17:34:00.6713205Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_dns_timeout 2024-04-03T17:34:00.6714217Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_nominal 2024-04-03T17:34:00.6715363Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_doest_break_client 2024-04-03T17:34:00.6716724Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_set 2024-04-03T17:34:00.6717882Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_url_with_libuv 2024-04-03T17:34:00.6719011Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_append_roundtrip 2024-04-03T17:34:00.6720130Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_extended_methods_fallbacks 2024-04-03T17:34:00.6721304Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_has_extended_api_passthrough 2024-04-03T17:34:00.6722475Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_has_extended_api_roundtrip 2024-04-03T17:34:00.6723674Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_multi_get_roundtrip 2024-04-03T17:34:00.6724773Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_multi_set_roundtrip 2024-04-03T17:34:00.6725863Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_optional_methods_fail 2024-04-03T17:34:00.6726954Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_0 2024-04-03T17:34:00.6728127Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_1 2024-04-03T17:34:00.6729245Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_2 2024-04-03T17:34:00.6730293Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_3 2024-04-03T17:34:00.6731348Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_4 2024-04-03T17:34:00.6732394Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_5 2024-04-03T17:34:00.6733436Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_i_6 2024-04-03T17:34:00.6734533Z Running 1 items in this shard: test/distributed/test_store.py::TimeoutTest::test_interrupt_doesnt_break_wait 2024-04-03T17:34:00.6735624Z Running 1 items in this shard: test/distributed/test_store.py::InitPgWithUvStore::test_with_env_var 2024-04-03T17:34:00.6736666Z Running 1 items in this shard: test/distributed/test_store.py::InitPgWithUvStore::test_with_url_param 2024-04-03T17:34:00.6737295Z 2024-04-03T17:34:00.6737943Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout 1/1 ... [2024-04-03 17:34:00.659189] 2024-04-03T17:34:00.6740142Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:34:00.659717] 2024-04-03T17:34:02.3809223Z 2024-04-03T17:34:02.3811183Z distributed/pipeline/sync/skip/test_inspect_skip_layout 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_inspect_skip_layout_1.1_69a68aecfa651e38_.log 2024-04-03T17:34:02.3816030Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace 2024-04-03T17:34:02.3819459Z 2024-04-03T17:34:02.3819985Z Running distributed/_shard/sharded_optim/test_sharded_optim 1/1 ... [2024-04-03 17:34:02.381376] 2024-04-03T17:34:02.3822094Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:34:02.381793] 2024-04-03T17:34:04.8033877Z 2024-04-03T17:34:04.8036173Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_optim.test_sharded_optim_1.1_b8caa0a304e3cdb7_.log 2024-04-03T17:34:04.8038729Z Running 2 items in this shard: test/distributed/_shard/sharded_optim/test_sharded_optim.py::TestShardedOptimizer::test_named_params_with_sharded_tensor, test/distributed/_shard/sharded_optim/test_sharded_optim.py::TestShardedOptimizer::test_sharded_optim 2024-04-03T17:34:04.8040513Z 2024-04-03T17:34:04.8040881Z Running distributed/test_launcher 1/1 ... [2024-04-03 17:34:04.803872] 2024-04-03T17:34:04.8046487Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_launcher.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:34:04.804310] 2024-04-03T17:34:08.6283715Z 2024-04-03T17:34:08.6285827Z distributed/test_launcher 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_launcher_1.1_ee11cd3a27fa1a5d_.log 2024-04-03T17:34:08.6287479Z Running 1 items in this shard: test/distributed/test_launcher.py::TestDistributedLaunch::test_launch_user_script 2024-04-03T17:34:08.6288263Z 2024-04-03T17:34:08.6289699Z Running distributed/optim/test_zero_redundancy_optimizer 1/1 ... [2024-04-03 17:34:08.628756] 2024-04-03T17:34:08.6294921Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/optim/test_zero_redundancy_optimizer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:34:08.629194] 2024-04-03T17:35:15.2547819Z 2024-04-03T17:35:15.2549500Z distributed/optim/test_zero_redundancy_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_zero_redundancy_optimizer_1.1_29c88b6d392601f0_.log 2024-04-03T17:35:15.2590227Z Running 42 items in this shard: test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_constructor, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_lr_scheduler, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_same_dense_param_type, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_state_dict, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_with_extra_inner_key, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_with_kwargs, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_without_closure, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_zero_grad, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_add_param_group, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_collect_shards, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_lr_scheduler, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_multiple_param_groups, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_nondefault_process_group, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_sharding, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_step, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_step_with_closure, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_join_cpu, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_join_gpu, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_model_parallel_parameters_as_bucket_view_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_model_parallel_parameters_as_bucket_view_True 2024-04-03T17:35:15.2628305Z 2024-04-03T17:35:15.2628882Z Running distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 ... [2024-04-03 17:35:15.255297] 2024-04-03T17:35:15.2631123Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:35:15.255733] 2024-04-03T17:35:23.2394091Z 2024-04-03T17:35:23.2396057Z distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_1.1_13ea9467df1ec5ca_.log 2024-04-03T17:35:23.2440635Z Running 67 items in this shard: test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorMetadata::test_serialize_and_deserialize, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestCreateTensorFromParams::test_empty, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardParameter::test_shard_parameter, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardParameter::test_shard_parameter_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardTensor::test_shard_tensor, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardTensor::test_shard_tensor_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardTensor::test_shard_tensor_with_empty_shard, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestModuleHookApi::test_collect_local_shard, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestModuleHookApi::test_reshard_output, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestLocalTensor::test_local_tensor, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestLocalTensor::test_local_tensor_error, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_cleanup, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_complete_world_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_like, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_full, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_ones, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_rand, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_zeros, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_gather_even, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_gather_uneven, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_insufficient_sharding_dims, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_invalid_pg_rpc_ranks, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_invalid_sharding, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_load_state_dict_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_multiple_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_partial_world_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_sharded_tensor_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_sharded_tensor_sizes, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_sharding_columns, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_state_dict, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_state_dict_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_state_dict_no_sharded_tensors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_create_sharded_tensor_with_ones, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_gather_even, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_gather_uneven, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_grid_sharding, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_multiple_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_partial_world_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_device, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_to_cpu, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_to_cuda, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_to_test, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_uneven_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_with_rpc_names, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalTensor::test_init_from_local_tensor, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalTensor::test_init_from_local_tensor_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_and_global_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_and_global_metadata_invalid_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_pin_memory, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_property_cross_ranks, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_shards_gaps, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_shards_overlap, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_st_base_init_from_local_shards_and_global_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorCustomOps::test_custom_op, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorCustomOps::test_custom_op_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorCustomOps::test_custom_op_override, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardMetadata::test_create_shard_with_no_placement, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardMetadata::test_shard_metadata_init, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestCreateTensorNoProcessGroupMode::test_init_from_local_shards_and_global_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestCreateTensorNoProcessGroupMode::test_non_contiguous_local_shards 2024-04-03T17:35:23.2483721Z 2024-04-03T17:35:23.2484230Z Running distributed/elastic/events/lib_test 1/1 ... [2024-04-03 17:35:23.239903] 2024-04-03T17:35:23.2486221Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/elastic/events/lib_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:35:23.240340] 2024-04-03T17:35:25.9629319Z 2024-04-03T17:35:25.9631264Z distributed/elastic/events/lib_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.events.lib_test_1.1_9e5f3925f7b4731d_.log 2024-04-03T17:35:25.9637888Z Running 8 items in this shard: test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_created, test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser, test/distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str 2024-04-03T17:35:25.9642675Z 2024-04-03T17:35:25.9643194Z Running distributed/elastic/metrics/api_test 1/1 ... [2024-04-03 17:35:25.963327] 2024-04-03T17:35:25.9645161Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:35:25.963788] 2024-04-03T17:35:28.6859611Z 2024-04-03T17:35:28.6862170Z distributed/elastic/metrics/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.metrics.api_test_1.1_15c298325372f8c2_.log 2024-04-03T17:35:28.6866817Z Running 3 items in this shard: test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_get_metric_name, test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_inheritance, test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_profile 2024-04-03T17:35:28.6869647Z 2024-04-03T17:35:28.6870513Z Running distributed/elastic/multiprocessing/api_test 1/1 ... [2024-04-03 17:35:28.686626] 2024-04-03T17:35:28.6876368Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:35:28.687178] 2024-04-03T17:35:49.9904378Z 2024-04-03T17:35:49.9906302Z distributed/elastic/multiprocessing/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.multiprocessing.api_test_1.1_652e64d633f42125_.log 2024-04-03T17:35:49.9930302Z Running 38 items in this shard: test/distributed/elastic/multiprocessing/api_test.py::RunProcResultsTest::test_get_failures, test/distributed/elastic/multiprocessing/api_test.py::RunProcResultsTest::test_is_failed, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_str_bad_input, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_value, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_value_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_args_env_len_mismatch, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_binary_exit, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_binary_incorrect_entrypoint, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_binary_raises, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_function_large_ret_val, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_function_raise, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_function_with_tensor, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_invalid_log_dir, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_multiprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_multiprocessing_context_poll_raises_exception, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_pcontext_wait, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_subprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_to_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_validate_full_rank, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_void_function, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_args_env_len_mismatch, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_binary, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_binary_exit, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_binary_incorrect_entrypoint, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_binary_raises, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_binary_redirect_and_tee, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_function, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_function_large_ret_val, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_function_raise, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_function_with_tensor, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_invalid_log_dir, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_multiprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_multiprocessing_context_poll_raises_exception, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_pcontext_wait, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_subprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_to_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_validate_full_rank, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_void_function 2024-04-03T17:35:49.9963853Z 2024-04-03T17:35:49.9964720Z Running distributed/elastic/timer/api_test 1/1 ... [2024-04-03 17:35:49.991073] 2024-04-03T17:35:49.9968445Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/elastic/timer/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:35:49.991485] 2024-04-03T17:35:51.2367477Z 2024-04-03T17:35:51.2369157Z distributed/elastic/timer/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.api_test_1.1_920c604bfdbfdd08_.log 2024-04-03T17:35:51.2370681Z 2024-04-03T17:35:51.2373418Z Running distributed/elastic/timer/local_timer_example 1/1 ... [2024-04-03 17:35:51.237148] 2024-04-03T17:35:51.2378456Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:35:51.237542] 2024-04-03T17:36:03.5296071Z 2024-04-03T17:36:03.5298437Z distributed/elastic/timer/local_timer_example 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.local_timer_example_1.1_2d6f6a70c4dfe384_.log 2024-04-03T17:36:03.5301312Z Running 2 items in this shard: test/distributed/elastic/timer/local_timer_example.py::LocalTimerExample::test_example_start_method_spawn, test/distributed/elastic/timer/local_timer_example.py::LocalTimerExample::test_torch_mp_example 2024-04-03T17:36:03.5302753Z 2024-04-03T17:36:03.5304010Z Running distributed/elastic/timer/local_timer_test 1/1 ... [2024-04-03 17:36:03.530132] 2024-04-03T17:36:03.5308725Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/elastic/timer/local_timer_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:36:03.530542] 2024-04-03T17:36:09.7079469Z 2024-04-03T17:36:09.7081440Z distributed/elastic/timer/local_timer_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.local_timer_test_1.1_e89f62ea197371e8_.log 2024-04-03T17:36:09.7090661Z Running 14 items in this shard: test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_client_interaction, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_exception_propagation, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_get_timer_recursive, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_happy_path, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_no_client, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_timer, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get_less_than_size, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get_size, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_acquire_release, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_expired_timers, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_valid_timers, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_watchdog_call_count, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_watchdog_empty_queue 2024-04-03T17:36:09.7098501Z 2024-04-03T17:36:09.7098980Z Running distributed/elastic/utils/distributed_test 1/1 ... [2024-04-03 17:36:09.708381] 2024-04-03T17:36:09.7101037Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/elastic/utils/distributed_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-04-03 17:36:09.708788] 2024-04-03T17:36:14.5343312Z 2024-04-03T17:36:14.5345596Z distributed/elastic/utils/distributed_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.distributed_test_1.1_1618c89267595fda_.log 2024-04-03T17:36:14.5352993Z Running 7 items in this shard: test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_multi, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_no_port_multi, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_single_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_timeout_on_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_timeout_on_worker, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_port_already_in_use_on_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_port_already_in_use_on_worker 2024-04-03T17:36:14.5363434Z 2024-04-03T17:36:15.0866990Z 2024-04-03T17:36:15.0867408Z real 89m28.307s 2024-04-03T17:36:15.0867916Z user 205m22.404s 2024-04-03T17:36:15.0869422Z sys 40m36.844s 2024-04-03T17:36:15.0869951Z + assert_git_not_dirty 2024-04-03T17:36:15.0870752Z + [[ linux-jammy-py3.8-gcc11 != *rocm* ]] 2024-04-03T17:36:15.0871591Z + [[ linux-jammy-py3.8-gcc11 != *xla* ]] 2024-04-03T17:36:15.0872770Z ++ git status --porcelain 2024-04-03T17:36:15.0873395Z ++ grep -v '?? third_party' 2024-04-03T17:36:24.5415289Z ++ true 2024-04-03T17:36:24.5427796Z + git_status= 2024-04-03T17:36:24.5428288Z + [[ -n '' ]] 2024-04-03T17:36:24.5428733Z + [[ linux-jammy-py3.8-gcc11 == *cuda* ]] 2024-04-03T17:36:24.5429220Z + [[ linux-jammy-py3.8-gcc11 == *rocm* ]] 2024-04-03T17:36:24.5429679Z + [[ 1 == 1 ]] 2024-04-03T17:36:24.5429937Z + test_rpc 2024-04-03T17:36:24.5430425Z + echo 'Testing RPC C++ tests' 2024-04-03T17:36:24.5430788Z Testing RPC C++ tests 2024-04-03T17:36:24.5434277Z + ln -sf /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-04-03T17:36:24.5447403Z + ln -sf /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-04-03T17:36:24.5455957Z + ln -sf '/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtbb*' /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-04-03T17:36:24.5463842Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-04-03T17:36:24.5464642Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc 2024-04-03T17:36:24.6309743Z test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-04-03T17:36:24.6310733Z import pkg_resources 2024-04-03T17:36:26.9035006Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-04-03T17:36:26.9036862Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-04-03T17:36:26.9150897Z Found test times from artifacts 2024-04-03T17:36:26.9499216Z Found test times from artifacts 2024-04-03T17:36:26.9509341Z Running all tests 2024-04-03T17:36:26.9511266Z Name: tests to run (est. time: 0.0min) 2024-04-03T17:36:26.9511810Z Serial tests (1): 2024-04-03T17:36:26.9512279Z cpp/test_cpp_rpc 1/1 2024-04-03T17:36:26.9512616Z Parallel tests (0): 2024-04-03T17:36:26.9512995Z Name: excluded (est. time: 0.0min) 2024-04-03T17:36:26.9513376Z Serial tests (0): 2024-04-03T17:36:26.9513678Z Parallel tests (0): 2024-04-03T17:36:26.9514359Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-04-03T17:36:26.9665954Z Running cpp/test_cpp_rpc 1/1 ... [2024-04-03 17:36:26.966070] 2024-04-03T17:36:26.9673716Z Executing ['pytest', '/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-e17bf4f5ac0e9417.xml', '-x', '--reruns=2'] ... [2024-04-03 17:36:26.966838] 2024-04-03T17:36:31.2099155Z 2024-04-03T17:36:31.2100919Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_91be82aa2ccd8a07_.log 2024-04-03T17:36:31.2102198Z 2024-04-03T17:36:31.7922777Z ##[group]Run cat test/**/*_toprint.log || true 2024-04-03T17:36:31.7923259Z cat test/**/*_toprint.log || true 2024-04-03T17:36:31.8240199Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T17:36:31.8240705Z env: 2024-04-03T17:36:31.8240966Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:36:31.8241554Z DOCKER_CONTAINER_ID: b0f53da97a9da663582739611fc2277e27fd12f5b71cdfe54025ecd3466cc388 2024-04-03T17:36:31.8242203Z ##[endgroup] 2024-04-03T17:36:31.8308879Z cat: test/**/*_toprint.log: No such file or directory 2024-04-03T17:36:31.8339615Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-04-03T17:36:31.8340049Z kill "$MONITOR_SCRIPT_PID" 2024-04-03T17:36:31.8347149Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T17:36:31.8347638Z env: 2024-04-03T17:36:31.8347897Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:36:31.8348483Z DOCKER_CONTAINER_ID: b0f53da97a9da663582739611fc2277e27fd12f5b71cdfe54025ecd3466cc388 2024-04-03T17:36:31.8349151Z MONITOR_SCRIPT_PID: 14390 2024-04-03T17:36:31.8349480Z ##[endgroup] 2024-04-03T17:36:31.8529346Z Prepare all required actions 2024-04-03T17:36:31.8529900Z Getting action download info 2024-04-03T17:36:32.0825681Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-04-03T17:36:32.2808188Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-04-03T17:36:32.2808651Z with: 2024-04-03T17:36:32.2808993Z file-suffix: test-distributed-1-2-linux.2xlarge_ 2024-04-03T17:36:32.2809458Z s3-bucket: gha-artifacts 2024-04-03T17:36:32.2809786Z env: 2024-04-03T17:36:32.2810032Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:36:32.2810611Z DOCKER_CONTAINER_ID: b0f53da97a9da663582739611fc2277e27fd12f5b71cdfe54025ecd3466cc388 2024-04-03T17:36:32.2811248Z ##[endgroup] 2024-04-03T17:36:32.2853349Z ##[group]Run # Remove any previous test jsons if they exist 2024-04-03T17:36:32.2853941Z # Remove any previous test jsons if they exist 2024-04-03T17:36:32.2854443Z rm -f test-jsons-*.zip 2024-04-03T17:36:32.2855115Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-04-03T17:36:32.2862817Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T17:36:32.2863289Z env: 2024-04-03T17:36:32.2863548Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:36:32.2864146Z DOCKER_CONTAINER_ID: b0f53da97a9da663582739611fc2277e27fd12f5b71cdfe54025ecd3466cc388 2024-04-03T17:36:32.2865107Z FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_ 2024-04-03T17:36:32.2865570Z ##[endgroup] 2024-04-03T17:36:32.3180904Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-04-03T17:36:32.3219551Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-04-03T17:36:32.3223765Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-04-03T17:36:32.3237591Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-04-03T17:36:32.3238833Z adding: test/.pytorch-slow-tests.json (deflated 59%) 2024-04-03T17:36:32.3245271Z adding: test/.pytorch-disabled-tests.json (deflated 84%) 2024-04-03T17:36:32.3274698Z ##[group]Run # Remove any previous test reports if they exist 2024-04-03T17:36:32.3275342Z # Remove any previous test reports if they exist 2024-04-03T17:36:32.3276062Z rm -f test-reports-*.zip 2024-04-03T17:36:32.3276636Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-04-03T17:36:32.3284407Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T17:36:32.3284885Z env: 2024-04-03T17:36:32.3285307Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:36:32.3285907Z DOCKER_CONTAINER_ID: b0f53da97a9da663582739611fc2277e27fd12f5b71cdfe54025ecd3466cc388 2024-04-03T17:36:32.3286637Z FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_ 2024-04-03T17:36:32.3287102Z ##[endgroup] 2024-04-03T17:36:32.3559580Z adding: test/test-reports/python-pytest/distributed._tensor.debug.test_op_coverage/distributed._tensor.debug.test_op_coverage-0a6c89e339dc3852.xml (deflated 35%) 2024-04-03T17:36:32.3571526Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor/distributed._tensor.test_dtensor-6b28b147e81ca1ce.xml (deflated 91%) 2024-04-03T17:36:32.3575149Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_tp_checkpoint/distributed.checkpoint.test_tp_checkpoint-a9596c04102612ee.xml (deflated 65%) 2024-04-03T17:36:32.3579183Z adding: test/test-reports/python-pytest/distributed._tensor.test_tensor_ops/distributed._tensor.test_tensor_ops-bc76340fd8e171a1.xml (deflated 85%) 2024-04-03T17:36:32.3584051Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_init/distributed._composable.fsdp.test_fully_shard_init-4c919d00ec9213e4.xml (deflated 91%) 2024-04-03T17:36:32.3588174Z adding: test/test-reports/python-pytest/distributed._tensor.test_xla_integration/distributed._tensor.test_xla_integration-bfc13eea7bb7d7fb.xml (deflated 74%) 2024-04-03T17:36:32.3591831Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_format_utils/distributed.checkpoint.test_format_utils-05f72783cd2b0d3e.xml (deflated 59%) 2024-04-03T17:36:32.3604557Z adding: test/test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-228c5c328ca8fe91.xml (deflated 91%) 2024-04-03T17:36:32.3611258Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_optim_state/distributed.checkpoint.test_fsdp_optim_state-e27cee402b9bebef.xml (deflated 66%) 2024-04-03T17:36:32.3614939Z adding: test/test-reports/python-pytest/distributed._tensor.test_api/distributed._tensor.test_api-afb49458789d1823.xml (deflated 78%) 2024-04-03T17:36:32.3619626Z adding: test/test-reports/python-pytest/distributed._tensor.test_random_ops/distributed._tensor.test_random_ops-11f2f40ce2a680b4.xml (deflated 82%) 2024-04-03T17:36:32.3627523Z adding: test/test-reports/python-pytest/distributed.test_device_mesh/distributed.test_device_mesh-966d10cd449c9b7f.xml (deflated 91%) 2024-04-03T17:36:32.3644477Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dtensor_resharding/distributed.checkpoint.test_dtensor_resharding-3d5b95a14e41b80a.xml (deflated 80%) 2024-04-03T17:36:32.3652359Z adding: test/test-reports/python-pytest/distributed._tensor.experimental.test_tp_transform/distributed._tensor.experimental.test_tp_transform-264bc3caebb0f847.xml (deflated 77%) 2024-04-03T17:36:32.3669511Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_autograd/distributed._composable.fsdp.test_fully_shard_autograd-e703a445f7c9d261.xml (deflated 61%) 2024-04-03T17:36:32.3672884Z adding: test/test-reports/python-pytest/distributed.test_compute_comm_reordering/distributed.test_compute_comm_reordering-0e46ac65b8580d46.xml (deflated 84%) 2024-04-03T17:36:32.3678041Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_ddp_2d_parallel/distributed.tensor.parallel.test_ddp_2d_parallel-fbf9598ffe8bd52a.xml (deflated 46%) 2024-04-03T17:36:32.3684174Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsspec/distributed.checkpoint.test_fsspec-00e534aab0ee1f3a.xml (deflated 60%) 2024-04-03T17:36:32.3688116Z adding: test/test-reports/python-pytest/distributed._tensor.test_experimental_ops/distributed._tensor.test_experimental_ops-053b4552b0c65501.xml (deflated 59%) 2024-04-03T17:36:32.3692994Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_memory/distributed.fsdp.test_fsdp_memory-8619ffcc127bd21b.xml (deflated 65%) 2024-04-03T17:36:32.3695078Z adding: test/test-reports/python-pytest/distributed._tensor.test_convolution_ops/distributed._tensor.test_convolution_ops-0ad0e42d6c55dcf8.xml (deflated 52%) 2024-04-03T17:36:32.3697359Z adding: test/test-reports/python-pytest/distributed._tools.test_memory_tracker/distributed._tools.test_memory_tracker-54b976465b06af48.xml (deflated 44%) 2024-04-03T17:36:32.3699369Z adding: test/test-reports/python-pytest/distributed._tensor.test_pointwise_ops/distributed._tensor.test_pointwise_ops-86f9443079a9d3e7.xml (deflated 70%) 2024-04-03T17:36:32.3701408Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_style/distributed.tensor.parallel.test_tp_style-b0d153464a2979bd.xml (deflated 79%) 2024-04-03T17:36:32.3703504Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dtensor_checkpoint/distributed.checkpoint.test_dtensor_checkpoint-d1e903bd9a02e76f.xml (deflated 47%) 2024-04-03T17:36:32.3705768Z adding: test/test-reports/python-pytest/distributed._tensor.test_op_strategy/distributed._tensor.test_op_strategy-bb4263dd5bed0eeb.xml (deflated 84%) 2024-04-03T17:36:32.3707804Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_memory/distributed._composable.fsdp.test_fully_shard_memory-4a4c137f2b8f1fbf.xml (deflated 38%) 2024-04-03T17:36:32.3710059Z adding: test/test-reports/python-pytest/distributed.checkpoint.e2e.test_pipeline/distributed.checkpoint.e2e.test_pipeline-800b0526a56a6b5a.xml (deflated 38%) 2024-04-03T17:36:32.3712211Z adding: test/test-reports/python-pytest/distributed._tensor.test_embedding_ops/distributed._tensor.test_embedding_ops-ecda458cd4c9cd9f.xml (deflated 61%) 2024-04-03T17:36:32.3714349Z adding: test/test-reports/python-pytest/distributed.checkpoint.e2e.test_e2e_save_and_load/distributed.checkpoint.e2e.test_e2e_save_and_load-6ae000d34298c089.xml (deflated 88%) 2024-04-03T17:36:32.3716809Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_state_dict/distributed._composable.fsdp.test_fully_shard_state_dict-5a80cd9b1b78acff.xml (deflated 55%) 2024-04-03T17:36:32.3719300Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_mixed_precision/distributed._composable.fsdp.test_fully_shard_mixed_precision-289c728dae282c84.xml (deflated 82%) 2024-04-03T17:36:32.3721920Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_mixed_precision/distributed._composable.fully_shard.test_fully_shard_mixed_precision-f75e5d15e4b13ad6.xml (deflated 48%) 2024-04-03T17:36:32.3724393Z adding: test/test-reports/python-pytest/distributed.fsdp.test_shard_utils/distributed.fsdp.test_shard_utils-273975e0a6a3a9c3.xml (deflated 63%) 2024-04-03T17:36:32.3726748Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-1fa816ee69ab28fc.xml (deflated 84%) 2024-04-03T17:36:32.3729144Z adding: test/test-reports/python-pytest/distributed._composable.test_compose/distributed._composable.test_compose-859bd52294a3735a.xml (deflated 86%) 2024-04-03T17:36:32.3731135Z adding: test/test-reports/python-pytest/distributed.test_fake_pg/distributed.test_fake_pg-514341a236a797be.xml (deflated 81%) 2024-04-03T17:36:32.3733280Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_parallelize_api/distributed.tensor.parallel.test_parallelize_api-0cb9528640a9a279.xml (deflated 76%) 2024-04-03T17:36:32.3735909Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_util/distributed._composable.fully_shard.test_fully_shard_util-fac3d7b30defb1ec.xml (deflated 47%) 2024-04-03T17:36:32.3738367Z adding: test/test-reports/python-pytest/distributed.test_inductor_collectives/distributed.test_inductor_collectives-95f80791c2dd3aaf.xml (deflated 97%) 2024-04-03T17:36:32.3740665Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_backward_prefetch/distributed.fsdp.test_fsdp_backward_prefetch-b7d591597e193458.xml (deflated 47%) 2024-04-03T17:36:32.3743125Z adding: test/test-reports/python-pytest/distributed._composable.test_replicate/distributed._composable.test_replicate-3dbcebc9692aef94.xml (deflated 80%) 2024-04-03T17:36:32.3745548Z adding: test/test-reports/python-pytest/distributed._shard.sharding_plan.test_sharding_plan/distributed._shard.sharding_plan.test_sharding_plan-db99cd408058d94b.xml (deflated 75%) 2024-04-03T17:36:32.3748242Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_clip_grad_norm_/distributed._composable.fsdp.test_fully_shard_clip_grad_norm_-5244d521a35dcde8.xml (deflated 53%) 2024-04-03T17:36:32.3751130Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion-c001bda3f9b5f63a.xml (deflated 48%) 2024-04-03T17:36:32.3753577Z adding: test/test-reports/python-pytest/distributed._tensor.test_optimizers/distributed._tensor.test_optimizers-c876891619823410.xml (deflated 82%) 2024-04-03T17:36:32.3755804Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_pure_fp16/distributed.fsdp.test_fsdp_pure_fp16-99ebb94aceef5d61.xml (deflated 63%) 2024-04-03T17:36:32.3758033Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_optim_state/distributed.fsdp.test_fsdp_optim_state-fa20f6578aeebdaf.xml (deflated 96%) 2024-04-03T17:36:32.3760282Z adding: test/test-reports/python-pytest/distributed._shard.test_sharder/distributed._shard.test_sharder-8e94929be0bf8e29.xml (deflated 66%) 2024-04-03T17:36:32.3762358Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_overlap/distributed.fsdp.test_fsdp_overlap-39f9d354ebd28142.xml (deflated 64%) 2024-04-03T17:36:32.3764511Z adding: test/test-reports/python-pytest/distributed.test_c10d_logger/distributed.test_c10d_logger-8e11e22e63b2cc3b.xml (deflated 65%) 2024-04-03T17:36:32.3766743Z adding: test/test-reports/python-pytest/distributed._shard.sharding_spec.test_sharding_spec/distributed._shard.sharding_spec.test_sharding_spec-d77b651c2b1204c4.xml (deflated 83%) 2024-04-03T17:36:32.3769227Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_dtensor_state_dict/distributed.fsdp.test_fsdp_dtensor_state_dict-fd275da0e63ca6c3.xml (deflated 92%) 2024-04-03T17:36:32.3771566Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_checkpoint/distributed.checkpoint.test_checkpoint-4be9167c556a1c7c.xml (deflated 84%) 2024-04-03T17:36:32.3773751Z adding: test/test-reports/python-pytest/distributed.test_multi_threaded_pg/distributed.test_multi_threaded_pg-429131796ab92813.xml (deflated 79%) 2024-04-03T17:36:32.3775832Z adding: test/test-reports/python-pytest/distributed.nn.jit.test_instantiator/distributed.nn.jit.test_instantiator-2b14a0bfc159c461.xml (deflated 61%) 2024-04-03T17:36:32.3778210Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_file_system_checkpoint/distributed.checkpoint.test_file_system_checkpoint-fe961fec18ae8e86.xml (deflated 77%) 2024-04-03T17:36:32.3780533Z adding: test/test-reports/python-pytest/distributed.rpc.test_share_memory/distributed.rpc.test_share_memory-b030da832469cce9.xml (deflated 48%) 2024-04-03T17:36:32.3782772Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_freezing_weights/distributed.fsdp.test_fsdp_freezing_weights-5a14ad85e8290e60.xml (deflated 96%) 2024-04-03T17:36:32.3785047Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_pipe/distributed.pipeline.sync.test_pipe-2786a2166d4b1b1f.xml (deflated 88%) 2024-04-03T17:36:32.3787234Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_fine_tune/distributed.fsdp.test_fsdp_fine_tune-2a185e629ab420ef.xml (deflated 64%) 2024-04-03T17:36:32.3789349Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-e53849905266e86b.xml (deflated 95%) 2024-04-03T17:36:32.3791428Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_meta/distributed.fsdp.test_fsdp_meta-3c295ced74399d24.xml (deflated 90%) 2024-04-03T17:36:32.3793746Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-f69edd83ac3ab8a1.xml (deflated 93%) 2024-04-03T17:36:32.3796068Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_worker/distributed.pipeline.sync.test_worker-d313dd665f61ed6e.xml (deflated 71%) 2024-04-03T17:36:32.3798473Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_transparency/distributed.pipeline.sync.test_transparency-02598d9c1f67c10c.xml (deflated 41%) 2024-04-03T17:36:32.3801204Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_optim_checkpoint/distributed._composable.fully_shard.test_fully_shard_optim_checkpoint-5f0782155fd6a9ce.xml (deflated 67%) 2024-04-03T17:36:32.3803916Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_unshard_params/distributed.fsdp.test_fsdp_unshard_params-936e58c459e6eb56.xml (deflated 89%) 2024-04-03T17:36:32.3806097Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_misc/distributed.fsdp.test_fsdp_misc-bce3128de28408da.xml (deflated 89%) 2024-04-03T17:36:32.3808278Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_hybrid_shard/distributed.fsdp.test_fsdp_hybrid_shard-0cfa2dfb1c6426b9.xml (deflated 81%) 2024-04-03T17:36:32.3810256Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-e17bf4f5ac0e9417.xml (deflated 65%) 2024-04-03T17:36:32.3812239Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_microbatch/distributed.pipeline.sync.test_microbatch-c775a21a9e8e8a4b.xml (deflated 81%) 2024-04-03T17:36:32.3814661Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_checkpoint/distributed.pipeline.sync.test_checkpoint-d7d6a568eeccbd34.xml (deflated 74%) 2024-04-03T17:36:32.3816901Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-0e629e029dbe25b3.xml (deflated 36%) 2024-04-03T17:36:32.3818999Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-04930392ee21e0fe.xml (deflated 36%) 2024-04-03T17:36:32.3821060Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-82ed85f288a914db.xml (deflated 36%) 2024-04-03T17:36:32.3823160Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-cf13fae0cc77f5c2.xml (deflated 36%) 2024-04-03T17:36:32.3825264Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-48ea24bdc032861b.xml (deflated 36%) 2024-04-03T17:36:32.3827323Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-0d78e20bf6d3f5fa.xml (deflated 36%) 2024-04-03T17:36:32.3829395Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-31a926a49b3b562c.xml (deflated 36%) 2024-04-03T17:36:32.3831467Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-f3c668b727ad05a0.xml (deflated 36%) 2024-04-03T17:36:32.3833517Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-4a9bf07e4acf117a.xml (deflated 36%) 2024-04-03T17:36:32.3835729Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-f36954e882b832f5.xml (deflated 36%) 2024-04-03T17:36:32.3837491Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-a3f5087febb17c8e.xml (deflated 36%) 2024-04-03T17:36:32.3839518Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-ce80d7d618465077.xml (deflated 36%) 2024-04-03T17:36:32.3841549Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-eaa64f5acf4a9417.xml (deflated 36%) 2024-04-03T17:36:32.3843814Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-e103b59b49e45a6a.xml (deflated 36%) 2024-04-03T17:36:32.3845856Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-c463513c8da07fcc.xml (deflated 36%) 2024-04-03T17:36:32.3847976Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-d42a5b7ba5eb59f6.xml (deflated 36%) 2024-04-03T17:36:32.3849775Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-76224a47b06178f9.xml (deflated 36%) 2024-04-03T17:36:32.3851464Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-712eb6adfcfd046f.xml (deflated 36%) 2024-04-03T17:36:32.3853148Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-d676989809d306fb.xml (deflated 36%) 2024-04-03T17:36:32.3854805Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-2880e6cf52983846.xml (deflated 35%) 2024-04-03T17:36:32.3856470Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-6cacf0050467b6ec.xml (deflated 36%) 2024-04-03T17:36:32.3858268Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-7c7081f21edd39d1.xml (deflated 35%) 2024-04-03T17:36:32.3860060Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-5bd01c38566764a0.xml (deflated 35%) 2024-04-03T17:36:32.3861730Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-705bbd2b446efdfd.xml (deflated 35%) 2024-04-03T17:36:32.3863426Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-d7bfde4e94f2461d.xml (deflated 35%) 2024-04-03T17:36:32.3865123Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-753d5ef320729a95.xml (deflated 35%) 2024-04-03T17:36:32.3866794Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-c38614e12fb8027c.xml (deflated 35%) 2024-04-03T17:36:32.3868459Z adding: test/test-reports/python-pytest/distributed.rpc.test_faulty_agent/distributed.rpc.test_faulty_agent-91efb7fe284db330.xml (deflated 35%) 2024-04-03T17:36:32.3870444Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_portal/distributed.pipeline.sync.skip.test_portal-c12c0bb8e610029d.xml (deflated 78%) 2024-04-03T17:36:32.3872440Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_init/distributed._shard.sharded_tensor.ops.test_init-69e112067caf5534.xml (deflated 75%) 2024-04-03T17:36:32.3874337Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c33199d7174ddf59.xml (deflated 36%) 2024-04-03T17:36:32.3876248Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e7468d55eb170c52.xml (deflated 35%) 2024-04-03T17:36:32.3878054Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-5a94ce2f89239af7.xml (deflated 35%) 2024-04-03T17:36:32.3879842Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-35d3af3490066681.xml (deflated 35%) 2024-04-03T17:36:32.3881630Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1c2577358af8918c.xml (deflated 36%) 2024-04-03T17:36:32.3883628Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-cba1bf922159422c.xml (deflated 36%) 2024-04-03T17:36:32.3885402Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-cdb4fa66f04bd919.xml (deflated 36%) 2024-04-03T17:36:32.3887193Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-58ba2b86a087b1c1.xml (deflated 35%) 2024-04-03T17:36:32.3888981Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-40382b71c8e44d7a.xml (deflated 36%) 2024-04-03T17:36:32.3890768Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-bd920b9a015636bb.xml (deflated 35%) 2024-04-03T17:36:32.3892549Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b0ade7edd8fa3c04.xml (deflated 35%) 2024-04-03T17:36:32.3894355Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6731e6f5b9c9a87e.xml (deflated 36%) 2024-04-03T17:36:32.3896130Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-5e501bb373ba3f9b.xml (deflated 35%) 2024-04-03T17:36:32.3898014Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-ae7ac3b98682a675.xml (deflated 35%) 2024-04-03T17:36:32.3899791Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d8dfc84bdeba346d.xml (deflated 35%) 2024-04-03T17:36:32.3901578Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4b466616ae177311.xml (deflated 35%) 2024-04-03T17:36:32.3903377Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f9e8a5763dfd1ea1.xml (deflated 35%) 2024-04-03T17:36:32.3905167Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-34fbe1b449db5ba3.xml (deflated 35%) 2024-04-03T17:36:32.3906927Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-25322b81057cf713.xml (deflated 35%) 2024-04-03T17:36:32.3908701Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-dc4a98971f91fe99.xml (deflated 35%) 2024-04-03T17:36:32.3910477Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8e9d2ace3a9d04a0.xml (deflated 35%) 2024-04-03T17:36:32.3912259Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-bdcb509882fece9d.xml (deflated 35%) 2024-04-03T17:36:32.3914031Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1b80d69aa0770ba9.xml (deflated 35%) 2024-04-03T17:36:32.3915903Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c1fecbc2772e0b42.xml (deflated 36%) 2024-04-03T17:36:32.3917688Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-ecce17a5c5844456.xml (deflated 35%) 2024-04-03T17:36:32.3919471Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-73267dcfa0be571a.xml (deflated 35%) 2024-04-03T17:36:32.3921238Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-58c01b5babeac76b.xml (deflated 36%) 2024-04-03T17:36:32.3923094Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d8c5e894288f9a46.xml (deflated 35%) 2024-04-03T17:36:32.3925006Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-dde9868eb8b114c7.xml (deflated 36%) 2024-04-03T17:36:32.3926793Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d5179e7a2207bf81.xml (deflated 36%) 2024-04-03T17:36:32.3928570Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6ea0f405e226e6b1.xml (deflated 37%) 2024-04-03T17:36:32.3930353Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b233dc20e7d26b6b.xml (deflated 35%) 2024-04-03T17:36:32.3932130Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f10829184ae5e10f.xml (deflated 35%) 2024-04-03T17:36:32.3933930Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3d2fce7b08dace0e.xml (deflated 35%) 2024-04-03T17:36:32.3935702Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-29e2fa287bc3bfd1.xml (deflated 35%) 2024-04-03T17:36:32.3937482Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a2f853866497ab30.xml (deflated 36%) 2024-04-03T17:36:32.3939342Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4809242ee422711c.xml (deflated 36%) 2024-04-03T17:36:32.3941121Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-de375dd728ce18fe.xml (deflated 37%) 2024-04-03T17:36:32.3942892Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-51dda4728a09e41c.xml (deflated 35%) 2024-04-03T17:36:32.3944676Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-ffcc458b26facbfd.xml (deflated 36%) 2024-04-03T17:36:32.3946462Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d3c95b7dcf75cffe.xml (deflated 36%) 2024-04-03T17:36:32.3948252Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1d840e125b79123f.xml (deflated 36%) 2024-04-03T17:36:32.3950017Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e7ba03455112be6e.xml (deflated 36%) 2024-04-03T17:36:32.3951796Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a8ca434642b99aa4.xml (deflated 35%) 2024-04-03T17:36:32.3953573Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3176ede2905792bc.xml (deflated 35%) 2024-04-03T17:36:32.3955355Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8fe3809ee89d67f8.xml (deflated 35%) 2024-04-03T17:36:32.3957210Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-049b5cdd6045a387.xml (deflated 35%) 2024-04-03T17:36:32.3959011Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b033dabaad404189.xml (deflated 36%) 2024-04-03T17:36:32.3960797Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4338ca02fb16beb6.xml (deflated 36%) 2024-04-03T17:36:32.3962582Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c49144698bd9b516.xml (deflated 36%) 2024-04-03T17:36:32.3964531Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c1396a820797b584.xml (deflated 35%) 2024-04-03T17:36:32.3966307Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a82fcb0a1ef1a12c.xml (deflated 36%) 2024-04-03T17:36:32.3968337Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d5d783f9a8410c13.xml (deflated 36%) 2024-04-03T17:36:32.3970122Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-000df89527c6d001.xml (deflated 36%) 2024-04-03T17:36:32.3971904Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-36d74df60d41b91e.xml (deflated 35%) 2024-04-03T17:36:32.3973665Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-35df36722f0d610d.xml (deflated 35%) 2024-04-03T17:36:32.3975453Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a86eb61cdd81b936.xml (deflated 35%) 2024-04-03T17:36:32.3977235Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b31766f170043f3c.xml (deflated 35%) 2024-04-03T17:36:32.3979098Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-9efe1ebd60c8510e.xml (deflated 35%) 2024-04-03T17:36:32.3980867Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c4e59bd98fb96ce3.xml (deflated 36%) 2024-04-03T17:36:32.3982653Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-685de951842d7f91.xml (deflated 36%) 2024-04-03T17:36:32.3984433Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-514f2f25726e5f6f.xml (deflated 35%) 2024-04-03T17:36:32.3986200Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1955cb0906397ae2.xml (deflated 35%) 2024-04-03T17:36:32.3987954Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7120de59051eeb03.xml (deflated 35%) 2024-04-03T17:36:32.3989736Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7cc3440090f5fe71.xml (deflated 36%) 2024-04-03T17:36:32.3991512Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-28fb94c8fba7d4ce.xml (deflated 36%) 2024-04-03T17:36:32.3993305Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f7068b075ef8fdd1.xml (deflated 36%) 2024-04-03T17:36:32.3995073Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f3de79cce1920281.xml (deflated 35%) 2024-04-03T17:36:32.3996950Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3f328956898d544b.xml (deflated 35%) 2024-04-03T17:36:32.3998730Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-651787788a3bac07.xml (deflated 35%) 2024-04-03T17:36:32.4000498Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-fa175191921b6445.xml (deflated 36%) 2024-04-03T17:36:32.4002264Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-fdb3bcd3b905ca40.xml (deflated 35%) 2024-04-03T17:36:32.4004120Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-eaa3c027feda1763.xml (deflated 37%) 2024-04-03T17:36:32.4006050Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-57bb78dd8a5bebb6.xml (deflated 36%) 2024-04-03T17:36:32.4007834Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3fe5848c2f17fe79.xml (deflated 36%) 2024-04-03T17:36:32.4009605Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-72ed0633a8183869.xml (deflated 36%) 2024-04-03T17:36:32.4011370Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-2d3012f3c3e01386.xml (deflated 35%) 2024-04-03T17:36:32.4013128Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6a5c4e503dbb9069.xml (deflated 35%) 2024-04-03T17:36:32.4014923Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0bea38fdcf7eab6f.xml (deflated 35%) 2024-04-03T17:36:32.4016700Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8eb6f082f30750e1.xml (deflated 35%) 2024-04-03T17:36:32.4018554Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-9d8992ed2e8947c5.xml (deflated 35%) 2024-04-03T17:36:32.4020337Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f2667d6136bafe6d.xml (deflated 35%) 2024-04-03T17:36:32.4022110Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3557e9b55d2e4b06.xml (deflated 35%) 2024-04-03T17:36:32.4023873Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-33c2cea7d9098b47.xml (deflated 35%) 2024-04-03T17:36:32.4025656Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-ff49c72bc06f2e0d.xml (deflated 35%) 2024-04-03T17:36:32.4027442Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-99eb4b2a9b961ca7.xml (deflated 36%) 2024-04-03T17:36:32.4029233Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0e42a108bf007a10.xml (deflated 35%) 2024-04-03T17:36:32.4030991Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8ab6e567db202fe5.xml (deflated 35%) 2024-04-03T17:36:32.4032768Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c872ca4c6a99e9ee.xml (deflated 34%) 2024-04-03T17:36:32.4034557Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-9723d3ce719abf6a.xml (deflated 35%) 2024-04-03T17:36:32.4036427Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8cecc4b59ded1b1e.xml (deflated 34%) 2024-04-03T17:36:32.4038216Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4baf91360b24fa28.xml (deflated 35%) 2024-04-03T17:36:32.4039997Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-ac4bfd48c9e2fe91.xml (deflated 35%) 2024-04-03T17:36:32.4041780Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1c667b41b5502da1.xml (deflated 35%) 2024-04-03T17:36:32.4043628Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d06bd0c6c1a67c52.xml (deflated 34%) 2024-04-03T17:36:32.4045497Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6421450bf9742da9.xml (deflated 35%) 2024-04-03T17:36:32.4047249Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-9df675f20bdbb787.xml (deflated 35%) 2024-04-03T17:36:32.4049040Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0b383d973f5d0168.xml (deflated 35%) 2024-04-03T17:36:32.4050819Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0f91485c98e766dd.xml (deflated 35%) 2024-04-03T17:36:32.4052601Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-83e8d03e9c180b1b.xml (deflated 35%) 2024-04-03T17:36:32.4054377Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c8fe72ea65d9846d.xml (deflated 35%) 2024-04-03T17:36:32.4056161Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-69eecfb634450fc1.xml (deflated 35%) 2024-04-03T17:36:32.4057938Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-24f999cf06e77756.xml (deflated 35%) 2024-04-03T17:36:32.4059814Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b0e05637e7d879ec.xml (deflated 35%) 2024-04-03T17:36:32.4061599Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0f8b9388defba75c.xml (deflated 36%) 2024-04-03T17:36:32.4063384Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-2bee5f7e8da7ca9a.xml (deflated 36%) 2024-04-03T17:36:32.4065174Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e6283e41bfd0aa9d.xml (deflated 36%) 2024-04-03T17:36:32.4066940Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b346862190b4ac2d.xml (deflated 35%) 2024-04-03T17:36:32.4068715Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-cbc3fd203e96d57c.xml (deflated 35%) 2024-04-03T17:36:32.4070510Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8dd6b5b79fae3c01.xml (deflated 35%) 2024-04-03T17:36:32.4072294Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4706a7179dc22142.xml (deflated 35%) 2024-04-03T17:36:32.4074064Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-5ea10e3a16ceef50.xml (deflated 35%) 2024-04-03T17:36:32.4075915Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-448c6d03e389ee51.xml (deflated 35%) 2024-04-03T17:36:32.4077697Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-02844e0ddf747a6f.xml (deflated 35%) 2024-04-03T17:36:32.4079486Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-fed4317eb7b96ff8.xml (deflated 35%) 2024-04-03T17:36:32.4081266Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3fa1c97026b5740e.xml (deflated 35%) 2024-04-03T17:36:32.4083093Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-41a84ac4a46a61e2.xml (deflated 36%) 2024-04-03T17:36:32.4084973Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8f36aa58ad0197b8.xml (deflated 36%) 2024-04-03T17:36:32.4086752Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7ac9805d53610fa8.xml (deflated 35%) 2024-04-03T17:36:32.4088540Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3129b0467e2cddd0.xml (deflated 35%) 2024-04-03T17:36:32.4090312Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d11cb8e62d2fb69a.xml (deflated 35%) 2024-04-03T17:36:32.4092097Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-cd6efa0b45853254.xml (deflated 35%) 2024-04-03T17:36:32.4093874Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-405758ec457190cb.xml (deflated 35%) 2024-04-03T17:36:32.4095651Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1309e00b5cd9cdc1.xml (deflated 35%) 2024-04-03T17:36:32.4097404Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-2739270aed3711e8.xml (deflated 35%) 2024-04-03T17:36:32.4099254Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a5d4036c5dd40dfc.xml (deflated 36%) 2024-04-03T17:36:32.4101033Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e09fab52f3d25847.xml (deflated 35%) 2024-04-03T17:36:32.4102806Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-90e2447afe61a656.xml (deflated 35%) 2024-04-03T17:36:32.4104563Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0399d17f297a6121.xml (deflated 36%) 2024-04-03T17:36:32.4106335Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-dd539dced45a3671.xml (deflated 35%) 2024-04-03T17:36:32.4108123Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-be46df2b4eb234d8.xml (deflated 37%) 2024-04-03T17:36:32.4109912Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1674f3cf28e7e405.xml (deflated 35%) 2024-04-03T17:36:32.4111676Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8ff30ae4d1c89586.xml (deflated 55%) 2024-04-03T17:36:32.4113444Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-16405599769c6c39.xml (deflated 55%) 2024-04-03T17:36:32.4115214Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-46a68ac8fb140de6.xml (deflated 36%) 2024-04-03T17:36:32.4117081Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c85d86c11d9ab51b.xml (deflated 35%) 2024-04-03T17:36:32.4118866Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-be3569ffb3637419.xml (deflated 41%) 2024-04-03T17:36:32.4120631Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-ec9febdc24720caf.xml (deflated 37%) 2024-04-03T17:36:32.4122407Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-2a805436d288df92.xml (deflated 37%) 2024-04-03T17:36:32.4124281Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8fa46598819279de.xml (deflated 37%) 2024-04-03T17:36:32.4126157Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-bc1099bd9d8131e6.xml (deflated 37%) 2024-04-03T17:36:32.4127928Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a2ee8056805054cc.xml (deflated 37%) 2024-04-03T17:36:32.4129712Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-82dc9894384893b6.xml (deflated 37%) 2024-04-03T17:36:32.4131491Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-fb79b6a106efa531.xml (deflated 37%) 2024-04-03T17:36:32.4133279Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-91ccd626c836bd12.xml (deflated 36%) 2024-04-03T17:36:32.4135062Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6b3b686828434ef6.xml (deflated 36%) 2024-04-03T17:36:32.4136841Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-bf4909a51ac9d395.xml (deflated 37%) 2024-04-03T17:36:32.4138617Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-5ca1c802624ada2e.xml (deflated 37%) 2024-04-03T17:36:32.4140473Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-2f2369c8643090b9.xml (deflated 37%) 2024-04-03T17:36:32.4142242Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-428c84b1ebe9af5a.xml (deflated 37%) 2024-04-03T17:36:32.4144023Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-945084d8a421204a.xml (deflated 36%) 2024-04-03T17:36:32.4145805Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8984bee29413edcf.xml (deflated 37%) 2024-04-03T17:36:32.4147588Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-af347c8d92b0650d.xml (deflated 37%) 2024-04-03T17:36:32.4149356Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e7a81d173ca7cfd7.xml (deflated 35%) 2024-04-03T17:36:32.4151131Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-18bb01b59918e70e.xml (deflated 36%) 2024-04-03T17:36:32.4152909Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1997d9bc0390cdd2.xml (deflated 36%) 2024-04-03T17:36:32.4154680Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-724d2a9020878f19.xml (deflated 35%) 2024-04-03T17:36:32.4156544Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-84b5b4bfd76576e4.xml (deflated 35%) 2024-04-03T17:36:32.4158330Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7f6885905b704eaa.xml (deflated 35%) 2024-04-03T17:36:32.4160109Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6594f66d45707511.xml (deflated 36%) 2024-04-03T17:36:32.4161881Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-12e268e45ca9b4c8.xml (deflated 36%) 2024-04-03T17:36:32.4163713Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7b597b760b11ee5b.xml (deflated 35%) 2024-04-03T17:36:32.4165589Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-378dfc6abf1b574c.xml (deflated 36%) 2024-04-03T17:36:32.4167364Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-99fc4c4019927228.xml (deflated 36%) 2024-04-03T17:36:32.4169141Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-32dc20455a83c53f.xml (deflated 36%) 2024-04-03T17:36:32.4170906Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-123d32f726f97319.xml (deflated 35%) 2024-04-03T17:36:32.4172694Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b2444e88ac1edff0.xml (deflated 35%) 2024-04-03T17:36:32.4174466Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-641238276a290f46.xml (deflated 35%) 2024-04-03T17:36:32.4176242Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0a8423ce0e5dcd96.xml (deflated 35%) 2024-04-03T17:36:32.4178004Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-19e6f8b9d8833b88.xml (deflated 35%) 2024-04-03T17:36:32.4179864Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-5d155044162429d2.xml (deflated 36%) 2024-04-03T17:36:32.4181643Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-49afcae3dbd5f727.xml (deflated 36%) 2024-04-03T17:36:32.4183421Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4e0a9a53f4c5aaa1.xml (deflated 36%) 2024-04-03T17:36:32.4185193Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-249e75988cfce79a.xml (deflated 35%) 2024-04-03T17:36:32.4186964Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6df87bd79c31d89f.xml (deflated 35%) 2024-04-03T17:36:32.4188757Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f8f2d18c2d486112.xml (deflated 35%) 2024-04-03T17:36:32.4190526Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-45305745692e1070.xml (deflated 36%) 2024-04-03T17:36:32.4192278Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-db473822718d022c.xml (deflated 35%) 2024-04-03T17:36:32.4194042Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7dc1ec49738640bf.xml (deflated 35%) 2024-04-03T17:36:32.4195908Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6b34c9a9599c79c9.xml (deflated 35%) 2024-04-03T17:36:32.4197702Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-efee819e42ddd66a.xml (deflated 35%) 2024-04-03T17:36:32.4199479Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-35d9a5ced151b4f0.xml (deflated 35%) 2024-04-03T17:36:32.4201272Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d2db9ed4ec02a6f4.xml (deflated 36%) 2024-04-03T17:36:32.4203131Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-5bff533562148ebb.xml (deflated 36%) 2024-04-03T17:36:32.4204917Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-56353f9585684ee5.xml (deflated 36%) 2024-04-03T17:36:32.4206800Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8d9c7eb3c78f640c.xml (deflated 35%) 2024-04-03T17:36:32.4208579Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-01f3dd072f99f00e.xml (deflated 35%) 2024-04-03T17:36:32.4210375Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e922e3f3804d788b.xml (deflated 35%) 2024-04-03T17:36:32.4212155Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-855ae5be79aa7e64.xml (deflated 36%) 2024-04-03T17:36:32.4213933Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3f95e24bf05e4e59.xml (deflated 36%) 2024-04-03T17:36:32.4215696Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b591ea89f8c84f0a.xml (deflated 36%) 2024-04-03T17:36:32.4217484Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c8094c7c3bfb543a.xml (deflated 35%) 2024-04-03T17:36:32.4219355Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7c01a0f9d745e69a.xml (deflated 35%) 2024-04-03T17:36:32.4221141Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c0b5bbf3f8022111.xml (deflated 35%) 2024-04-03T17:36:32.4222906Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b9ed93404c76a26b.xml (deflated 35%) 2024-04-03T17:36:32.4224686Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d73e610bca697749.xml (deflated 35%) 2024-04-03T17:36:32.4226475Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-eca98c6923c86b3f.xml (deflated 35%) 2024-04-03T17:36:32.4228256Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b694f23002d25115.xml (deflated 36%) 2024-04-03T17:36:32.4230022Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f25c9d81e0b8f701.xml (deflated 36%) 2024-04-03T17:36:32.4231800Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b26cd5f9311ec5fa.xml (deflated 35%) 2024-04-03T17:36:32.4233578Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-913d4fa41104fdbb.xml (deflated 35%) 2024-04-03T17:36:32.4235363Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-396ba1905ecfa2dc.xml (deflated 35%) 2024-04-03T17:36:32.4237217Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-afd538a720f40fcc.xml (deflated 36%) 2024-04-03T17:36:32.4239004Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c25db5833c3c05c6.xml (deflated 36%) 2024-04-03T17:36:32.4240794Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c0122c6a5d3f1027.xml (deflated 36%) 2024-04-03T17:36:32.4242576Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-75f35b8bdeb4504e.xml (deflated 36%) 2024-04-03T17:36:32.4244420Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-76b17e15d4a69f65.xml (deflated 35%) 2024-04-03T17:36:32.4246318Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1dff328414bdaa30.xml (deflated 36%) 2024-04-03T17:36:32.4248108Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-aa4b9152a5383361.xml (deflated 38%) 2024-04-03T17:36:32.4249888Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-070329bfda41522b.xml (deflated 35%) 2024-04-03T17:36:32.4251650Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-2876865f0de7aa99.xml (deflated 36%) 2024-04-03T17:36:32.4253421Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3605596f835741b5.xml (deflated 36%) 2024-04-03T17:36:32.4255195Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-37f51aa0be04b223.xml (deflated 35%) 2024-04-03T17:36:32.4256977Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-2778f06a06a310b0.xml (deflated 35%) 2024-04-03T17:36:32.4258733Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-909cee5a7e31bd49.xml (deflated 35%) 2024-04-03T17:36:32.4260592Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3a9e06bd7d9b1294.xml (deflated 36%) 2024-04-03T17:36:32.4262375Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e046f5390186d55e.xml (deflated 36%) 2024-04-03T17:36:32.4264154Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f9d6e5a3ce78b3d5.xml (deflated 35%) 2024-04-03T17:36:32.4265922Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1df129ccbde16a53.xml (deflated 36%) 2024-04-03T17:36:32.4267693Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e5cbe720434e4714.xml (deflated 36%) 2024-04-03T17:36:32.4269476Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-debbc0df3d2f86f0.xml (deflated 36%) 2024-04-03T17:36:32.4271261Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-58aeb0f7fce64b6d.xml (deflated 36%) 2024-04-03T17:36:32.4273023Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-56efd4b37d6cbcbe.xml (deflated 35%) 2024-04-03T17:36:32.4274795Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7565a5c0c8706b35.xml (deflated 36%) 2024-04-03T17:36:32.4276663Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-61ad8e19e525f20a.xml (deflated 38%) 2024-04-03T17:36:32.4278442Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b259105557c24e0d.xml (deflated 36%) 2024-04-03T17:36:32.4280211Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a183a596283fa46e.xml (deflated 36%) 2024-04-03T17:36:32.4281996Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a11fcaaa252d6ec6.xml (deflated 36%) 2024-04-03T17:36:32.4283832Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4705200afadc0100.xml (deflated 35%) 2024-04-03T17:36:32.4285711Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-aca29dfc2e8c09b6.xml (deflated 36%) 2024-04-03T17:36:32.4287474Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-24c210adf27c0995.xml (deflated 35%) 2024-04-03T17:36:32.4289258Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-ed2d37f8f27c5370.xml (deflated 35%) 2024-04-03T17:36:32.4291040Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-fa5865d9163a6782.xml (deflated 35%) 2024-04-03T17:36:32.4292829Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8edffdc88d299977.xml (deflated 35%) 2024-04-03T17:36:32.4294598Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-00745140c916b764.xml (deflated 35%) 2024-04-03T17:36:32.4296350Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-72a64804efabba51.xml (deflated 37%) 2024-04-03T17:36:32.4298129Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3d39ccabf3cef630.xml (deflated 35%) 2024-04-03T17:36:32.4299997Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d33091d933b4a14e.xml (deflated 36%) 2024-04-03T17:36:32.4301787Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4b504a549eb866c1.xml (deflated 34%) 2024-04-03T17:36:32.4303554Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-43cd8b7d978de8b6.xml (deflated 35%) 2024-04-03T17:36:32.4305337Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8e142608fe1d8b6b.xml (deflated 35%) 2024-04-03T17:36:32.4307117Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-fbf697743c645923.xml (deflated 35%) 2024-04-03T17:36:32.4308889Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8c7aacafdd29b633.xml (deflated 35%) 2024-04-03T17:36:32.4310658Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4a8805a44ccb1270.xml (deflated 35%) 2024-04-03T17:36:32.4312445Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8c745e64eb848b29.xml (deflated 34%) 2024-04-03T17:36:32.4314227Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-74300af4c18d2a4d.xml (deflated 36%) 2024-04-03T17:36:32.4316089Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-15910749e27ba663.xml (deflated 34%) 2024-04-03T17:36:32.4317861Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b05cab1eccbd7dc6.xml (deflated 35%) 2024-04-03T17:36:32.4319646Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7f95900e0c75ed8a.xml (deflated 35%) 2024-04-03T17:36:32.4321422Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b788f14823b6c20b.xml (deflated 36%) 2024-04-03T17:36:32.4323266Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-148505c2105099c8.xml (deflated 35%) 2024-04-03T17:36:32.4325014Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-346cef6d83190c40.xml (deflated 35%) 2024-04-03T17:36:32.4326904Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-9de600ddc156b53b.xml (deflated 35%) 2024-04-03T17:36:32.4328686Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f07ce57c68748037.xml (deflated 35%) 2024-04-03T17:36:32.4330478Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4cd5bab230db66f8.xml (deflated 35%) 2024-04-03T17:36:32.4332237Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6fb969a3489f3603.xml (deflated 36%) 2024-04-03T17:36:32.4334011Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-69f44c57493bddf2.xml (deflated 35%) 2024-04-03T17:36:32.4335795Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f0adf23d1226f6ae.xml (deflated 35%) 2024-04-03T17:36:32.4337580Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c6e537e9c6fa2f6e.xml (deflated 35%) 2024-04-03T17:36:32.4339474Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-ee1e0fd2d6ed4af6.xml (deflated 36%) 2024-04-03T17:36:32.4341272Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-49d25505aaf650ac.xml (deflated 35%) 2024-04-03T17:36:32.4343051Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-409cd09610f01d7b.xml (deflated 34%) 2024-04-03T17:36:32.4344834Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-be4ecbc97bca2d58.xml (deflated 35%) 2024-04-03T17:36:32.4346607Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-67473d73414049cb.xml (deflated 35%) 2024-04-03T17:36:32.4348382Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c7b3110d8af7e883.xml (deflated 35%) 2024-04-03T17:36:32.4350159Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c1d1e969798fe12b.xml (deflated 35%) 2024-04-03T17:36:32.4351933Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-95f44c9e8750a4fb.xml (deflated 35%) 2024-04-03T17:36:32.4353684Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-f44ac86b8db84a31.xml (deflated 35%) 2024-04-03T17:36:32.4355555Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0f6d8ee1fd0146d7.xml (deflated 35%) 2024-04-03T17:36:32.4357337Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-09cc3df653f01c34.xml (deflated 36%) 2024-04-03T17:36:32.4359116Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-649bff760fbd2e9d.xml (deflated 36%) 2024-04-03T17:36:32.4361033Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3f0e2c269ac2b36f.xml (deflated 36%) 2024-04-03T17:36:32.4362898Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d66cf6d956768f4a.xml (deflated 35%) 2024-04-03T17:36:32.4364682Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0de4490e8d3fcfb8.xml (deflated 35%) 2024-04-03T17:36:32.4366576Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-6241155bdc2ee544.xml (deflated 35%) 2024-04-03T17:36:32.4368342Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-142319204d716f7a.xml (deflated 35%) 2024-04-03T17:36:32.4370121Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-179e252a8e0155e4.xml (deflated 35%) 2024-04-03T17:36:32.4372133Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-353af725b8b314e0.xml (deflated 36%) 2024-04-03T17:36:32.4373919Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-3df6bd0dd8af0046.xml (deflated 37%) 2024-04-03T17:36:32.4375709Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-42964e13ff81f431.xml (deflated 35%) 2024-04-03T17:36:32.4377485Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-552faa3f92f8d72e.xml (deflated 35%) 2024-04-03T17:36:32.4379273Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-5ea356f43726aadd.xml (deflated 34%) 2024-04-03T17:36:32.4381145Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-456b040bf51bdfad.xml (deflated 35%) 2024-04-03T17:36:32.4382933Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-020f1e3b51e81d11.xml (deflated 35%) 2024-04-03T17:36:32.4384692Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-769a2c220f146310.xml (deflated 35%) 2024-04-03T17:36:32.4386482Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-71e7a15f0ae21b90.xml (deflated 35%) 2024-04-03T17:36:32.4388261Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-9e50563063713f89.xml (deflated 35%) 2024-04-03T17:36:32.4390046Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-e1bcb30662118f20.xml (deflated 36%) 2024-04-03T17:36:32.4391805Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-86d0913af755099a.xml (deflated 36%) 2024-04-03T17:36:32.4393584Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7ba6effd9f71d558.xml (deflated 36%) 2024-04-03T17:36:32.4395371Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-390b948774b6d0e0.xml (deflated 36%) 2024-04-03T17:36:32.4397240Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-cd8c0e505105005d.xml (deflated 36%) 2024-04-03T17:36:32.4399006Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-d8cdf502651d5544.xml (deflated 37%) 2024-04-03T17:36:32.4400785Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7bf99cacbfe73ea4.xml (deflated 37%) 2024-04-03T17:36:32.4402572Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7bcf331d9aa2f91f.xml (deflated 36%) 2024-04-03T17:36:32.4404416Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1d583de5bd1ce77e.xml (deflated 37%) 2024-04-03T17:36:32.4406269Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-38d3bc5b9c718bae.xml (deflated 37%) 2024-04-03T17:36:32.4408050Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-19e3554f5b971785.xml (deflated 36%) 2024-04-03T17:36:32.4409831Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-719f8bba08ef68fe.xml (deflated 38%) 2024-04-03T17:36:32.4411623Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-4d262cef5c4483a5.xml (deflated 38%) 2024-04-03T17:36:32.4413376Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-02778c292e2761aa.xml (deflated 37%) 2024-04-03T17:36:32.4415162Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-b2e55baa1865e661.xml (deflated 38%) 2024-04-03T17:36:32.4416938Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7c37cf022d08eacc.xml (deflated 37%) 2024-04-03T17:36:32.4418703Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-475e13e524052606.xml (deflated 37%) 2024-04-03T17:36:32.4420522Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-1e5881159d09f176.xml (deflated 39%) 2024-04-03T17:36:32.4422292Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0e766148fe0ec2b5.xml (deflated 38%) 2024-04-03T17:36:32.4424058Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8fc5423649586ed3.xml (deflated 37%) 2024-04-03T17:36:32.4425835Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8ca1447a94c092e0.xml (deflated 37%) 2024-04-03T17:36:32.4427609Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-8d96f5dbf662295c.xml (deflated 37%) 2024-04-03T17:36:32.4429390Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-a641a86c3e55ad49.xml (deflated 36%) 2024-04-03T17:36:32.4431184Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-9d0b9ffdf6d57f09.xml (deflated 37%) 2024-04-03T17:36:32.4432964Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-0f4d51ec12add94c.xml (deflated 37%) 2024-04-03T17:36:32.4434725Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-2ad983db604df3b6.xml (deflated 37%) 2024-04-03T17:36:32.4436597Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-2d22d8d01be03366.xml (deflated 37%) 2024-04-03T17:36:32.4438371Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-7a278d61bb493116.xml (deflated 37%) 2024-04-03T17:36:32.4440159Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-c9cd0e6eb0ef0466.xml (deflated 37%) 2024-04-03T17:36:32.4442033Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-55c7d65a8bcc369f.xml (deflated 37%) 2024-04-03T17:36:32.4443880Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-64bcb37b6850b083.xml (deflated 37%) 2024-04-03T17:36:32.4445653Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-fa3460851b282df7.xml (deflated 37%) 2024-04-03T17:36:32.4447562Z adding: test/test-reports/python-pytest/distributed.rpc.test_tensorpipe_agent/distributed.rpc.test_tensorpipe_agent-197a97da6e93d507.xml (deflated 37%) 2024-04-03T17:36:32.4449135Z adding: test/test-reports/python-pytest/distributed.test_nccl/distributed.test_nccl-35bf2d42c769c2a6.xml (deflated 27%) 2024-04-03T17:36:32.4450991Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_verify_skippables/distributed.pipeline.sync.skip.test_verify_skippables-0d95eabf8811b914.xml (deflated 80%) 2024-04-03T17:36:32.4453080Z adding: test/test-reports/python-pytest/distributed.test_data_parallel/distributed.test_data_parallel-dd51d6781032f10b.xml (deflated 94%) 2024-04-03T17:36:32.4454621Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-ccfae0b9b0704b1e.xml (deflated 44%) 2024-04-03T17:36:32.4456061Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-0bad4a6cc08326a6.xml (deflated 37%) 2024-04-03T17:36:32.4457503Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-b0043d1087e44bef.xml (deflated 40%) 2024-04-03T17:36:32.4458922Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-09705549bcf37c38.xml (deflated 43%) 2024-04-03T17:36:32.4460446Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-8439db3e86dfbec3.xml (deflated 44%) 2024-04-03T17:36:32.4461874Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e6cc7324cac1ca4e.xml (deflated 37%) 2024-04-03T17:36:32.4463311Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-9957ad05206e95fb.xml (deflated 36%) 2024-04-03T17:36:32.4464854Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-de911890665b1325.xml (deflated 37%) 2024-04-03T17:36:32.4466283Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-9b04801811711f4d.xml (deflated 37%) 2024-04-03T17:36:32.4467696Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-a6ba0b0619989852.xml (deflated 37%) 2024-04-03T17:36:32.4469125Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-c5d2d11af446f2c5.xml (deflated 37%) 2024-04-03T17:36:32.4470551Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-1e95757475c40f77.xml (deflated 37%) 2024-04-03T17:36:32.4471984Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-47d07ccfa2f90f68.xml (deflated 37%) 2024-04-03T17:36:32.4473396Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-3f44b2027615498d.xml (deflated 37%) 2024-04-03T17:36:32.4474823Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5eef70665798b6ec.xml (deflated 37%) 2024-04-03T17:36:32.4476354Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-28b292e3f1e26d06.xml (deflated 43%) 2024-04-03T17:36:32.4477774Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-22e4e22de2650efe.xml (deflated 36%) 2024-04-03T17:36:32.4479200Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-3841de2e2c7f0ce9.xml (deflated 43%) 2024-04-03T17:36:32.4480629Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-54b3341b3cd8825e.xml (deflated 43%) 2024-04-03T17:36:32.4482067Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-293a7fb725875a3f.xml (deflated 36%) 2024-04-03T17:36:32.4483551Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-b82bd99340fde3d7.xml (deflated 36%) 2024-04-03T17:36:32.4484981Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-9a1dcb0125730fb6.xml (deflated 36%) 2024-04-03T17:36:32.4486611Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-c16ab9fa4b742305.xml (deflated 36%) 2024-04-03T17:36:32.4488078Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e7ce35f360651f80.xml (deflated 36%) 2024-04-03T17:36:32.4489560Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-3d1962cdf9a037e0.xml (deflated 40%) 2024-04-03T17:36:32.4491051Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-6a6b66941e919124.xml (deflated 36%) 2024-04-03T17:36:32.4492481Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-fc1c6b71c666c556.xml (deflated 36%) 2024-04-03T17:36:32.4493909Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-8fa3844ca707304e.xml (deflated 35%) 2024-04-03T17:36:32.4495333Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-c47c16a71044e186.xml (deflated 34%) 2024-04-03T17:36:32.4496753Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-2c20565804e49f1e.xml (deflated 36%) 2024-04-03T17:36:32.4498177Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-f97466846074282e.xml (deflated 37%) 2024-04-03T17:36:32.4499674Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-3d28ed40c90b063f.xml (deflated 36%) 2024-04-03T17:36:32.4501104Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5345cd584d553a8a.xml (deflated 36%) 2024-04-03T17:36:32.4502534Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-c77e9bdb0215871d.xml (deflated 36%) 2024-04-03T17:36:32.4503963Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-7104fcd0f99459ff.xml (deflated 36%) 2024-04-03T17:36:32.4505386Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-4bf63e5b1097b6be.xml (deflated 34%) 2024-04-03T17:36:32.4506811Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-0e205c7a63254350.xml (deflated 34%) 2024-04-03T17:36:32.4508235Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-5a47c2096be1691d.xml (deflated 35%) 2024-04-03T17:36:32.4509670Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-6d6b6b5ad6d1d768.xml (deflated 40%) 2024-04-03T17:36:32.4511085Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-480a9b4d005e4ef7.xml (deflated 34%) 2024-04-03T17:36:32.4512512Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-768cebc3a3e3b1cd.xml (deflated 34%) 2024-04-03T17:36:32.4513959Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d087bbfb7ce2ee98.xml (deflated 34%) 2024-04-03T17:36:32.4515402Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-f2d46ae1ca74f373.xml (deflated 34%) 2024-04-03T17:36:32.4516905Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-6b5c40a1e3888d12.xml (deflated 35%) 2024-04-03T17:36:32.4518332Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-545429c60222c215.xml (deflated 35%) 2024-04-03T17:36:32.4519763Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-14fd36ec57a9b710.xml (deflated 35%) 2024-04-03T17:36:32.4521189Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-afbe9c6feb0f3f9e.xml (deflated 35%) 2024-04-03T17:36:32.4522696Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-1a65398aafba9a89.xml (deflated 35%) 2024-04-03T17:36:32.4524134Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-153252d812ec0f6f.xml (deflated 34%) 2024-04-03T17:36:32.4525657Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-b77fab84c4172338.xml (deflated 35%) 2024-04-03T17:36:32.4527083Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-f00cb5fefeaae0fb.xml (deflated 36%) 2024-04-03T17:36:32.4528525Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-2bb8050bba737b8f.xml (deflated 36%) 2024-04-03T17:36:32.4530025Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-834fe75330ddbf6c.xml (deflated 36%) 2024-04-03T17:36:32.4531464Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-0d37d946362b528a.xml (deflated 36%) 2024-04-03T17:36:32.4532879Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-1f1c768b66d83adc.xml (deflated 36%) 2024-04-03T17:36:32.4534303Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-0b494121aae14037.xml (deflated 35%) 2024-04-03T17:36:32.4535732Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-66409ec431a90b48.xml (deflated 37%) 2024-04-03T17:36:32.4537158Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d344962206e117af.xml (deflated 36%) 2024-04-03T17:36:32.4538651Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-19c6b8d29996c93e.xml (deflated 36%) 2024-04-03T17:36:32.4540089Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-6ae6898414d80eab.xml (deflated 36%) 2024-04-03T17:36:32.4541522Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-a068f4db08f38f73.xml (deflated 36%) 2024-04-03T17:36:32.4542955Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-ec7aff7e6a37d1d5.xml (deflated 35%) 2024-04-03T17:36:32.4544394Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-2ae1447d69aff284.xml (deflated 35%) 2024-04-03T17:36:32.4545821Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-363e6848e9b58adc.xml (deflated 35%) 2024-04-03T17:36:32.4547249Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-de248ff59f3fb3d6.xml (deflated 35%) 2024-04-03T17:36:32.4548667Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-be336c81c6da5eb8.xml (deflated 35%) 2024-04-03T17:36:32.4550098Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-117b8a9a9ab181c9.xml (deflated 35%) 2024-04-03T17:36:32.4551526Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-51e0041e2640f205.xml (deflated 34%) 2024-04-03T17:36:32.4552947Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-e42d341f3b4dca28.xml (deflated 36%) 2024-04-03T17:36:32.4554378Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-834332b992f868a1.xml (deflated 35%) 2024-04-03T17:36:32.4555893Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-56c2ac9d4e9d5421.xml (deflated 35%) 2024-04-03T17:36:32.4557340Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-c4186c77ea7f10c3.xml (deflated 35%) 2024-04-03T17:36:32.4558777Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-d25dd7df0bf46705.xml (deflated 35%) 2024-04-03T17:36:32.4560202Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-a83a7e123d3a19b7.xml (deflated 36%) 2024-04-03T17:36:32.4561626Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-95b030a034f1d87d.xml (deflated 36%) 2024-04-03T17:36:32.4563221Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-a494903dd6fc3792.xml (deflated 35%) 2024-04-03T17:36:32.4564646Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-cd0c5c3e1a812e6d.xml (deflated 35%) 2024-04-03T17:36:32.4566090Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-a500b890d5430e40.xml (deflated 35%) 2024-04-03T17:36:32.4567531Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-1659acf5f8e9c92d.xml (deflated 57%) 2024-04-03T17:36:32.4569010Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-4962a665624160c0.xml (deflated 57%) 2024-04-03T17:36:32.4570491Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-716c21226eec6609.xml (deflated 57%) 2024-04-03T17:36:32.4571920Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-f7915dda6e4283dc.xml (deflated 57%) 2024-04-03T17:36:32.4573356Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-7e4aca2471c0e9ea.xml (deflated 37%) 2024-04-03T17:36:32.4574780Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-8cd1c090a91b18a2.xml (deflated 34%) 2024-04-03T17:36:32.4576209Z adding: test/test-reports/python-pytest/distributed.test_store/distributed.test_store-cb783ffeaefd0541.xml (deflated 33%) 2024-04-03T17:36:32.4578169Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_inspect_skip_layout/distributed.pipeline.sync.skip.test_inspect_skip_layout-61a2e47cef8e27e1.xml (deflated 74%) 2024-04-03T17:36:32.4580332Z adding: test/test-reports/python-pytest/distributed._shard.sharded_optim.test_sharded_optim/distributed._shard.sharded_optim.test_sharded_optim-19fb771d19d82daf.xml (deflated 67%) 2024-04-03T17:36:32.4582100Z adding: test/test-reports/python-pytest/distributed.test_launcher/distributed.test_launcher-bec971677d877245.xml (deflated 39%) 2024-04-03T17:36:32.4583891Z adding: test/test-reports/python-pytest/distributed.optim.test_zero_redundancy_optimizer/distributed.optim.test_zero_redundancy_optimizer-d98078424ba3e0c8.xml (deflated 94%) 2024-04-03T17:36:32.4585978Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_sharded_tensor/distributed._shard.sharded_tensor.test_sharded_tensor-67bd98d923722db1.xml (deflated 95%) 2024-04-03T17:36:32.4587919Z adding: test/test-reports/python-pytest/distributed.elastic.events.lib_test/distributed.elastic.events.lib_test-07802a648d6e3712.xml (deflated 76%) 2024-04-03T17:36:32.4589733Z adding: test/test-reports/python-pytest/distributed.elastic.metrics.api_test/distributed.elastic.metrics.api_test-4fbe4f144f7b9154.xml (deflated 60%) 2024-04-03T17:36:32.4591671Z adding: test/test-reports/python-pytest/distributed.elastic.multiprocessing.api_test/distributed.elastic.multiprocessing.api_test-503b426b24307211.xml (deflated 88%) 2024-04-03T17:36:32.4593716Z adding: test/test-reports/python-pytest/distributed.elastic.timer.local_timer_example/distributed.elastic.timer.local_timer_example-2fda7db38a066de3.xml (deflated 51%) 2024-04-03T17:36:32.4595794Z adding: test/test-reports/python-pytest/distributed.elastic.timer.local_timer_test/distributed.elastic.timer.local_timer_test-5033b72c40ebdb75.xml (deflated 80%) 2024-04-03T17:36:32.4597771Z adding: test/test-reports/python-pytest/distributed.elastic.utils.distributed_test/distributed.elastic.utils.distributed_test-137bfafef97c7548.xml (deflated 77%) 2024-04-03T17:36:32.4599582Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3b272cddda384082.xml (deflated 57%) 2024-04-03T17:36:32.4601237Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e281636d6543725e.xml (deflated 42%) 2024-04-03T17:36:32.4602952Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3687f7e61521bf43.xml (deflated 41%) 2024-04-03T17:36:32.4604705Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c3257bfd8e553063.xml (deflated 42%) 2024-04-03T17:36:32.4606338Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3922c2c0c59b535a.xml (deflated 43%) 2024-04-03T17:36:32.4607992Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17c5e36b00e8980b.xml (deflated 42%) 2024-04-03T17:36:32.4609635Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-23a9ac2f502d9f1d.xml (deflated 35%) 2024-04-03T17:36:32.4611276Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c76502b7476f421b.xml (deflated 36%) 2024-04-03T17:36:32.4612901Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-640d028990c6ed8d.xml (deflated 35%) 2024-04-03T17:36:32.4614552Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca58e8101e79247b.xml (deflated 36%) 2024-04-03T17:36:32.4616195Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-72ba929199615342.xml (deflated 35%) 2024-04-03T17:36:32.4617926Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-147dab9bf3a6732b.xml (deflated 43%) 2024-04-03T17:36:32.4619563Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-73ca35f3a99ab364.xml (deflated 44%) 2024-04-03T17:36:32.4621205Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3a94cef54c28de4c.xml (deflated 36%) 2024-04-03T17:36:32.4622934Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0169dc3e6e61629a.xml (deflated 35%) 2024-04-03T17:36:32.4624587Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f494bd4abbe8dd5.xml (deflated 35%) 2024-04-03T17:36:32.4626231Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5975137fc572b8b0.xml (deflated 35%) 2024-04-03T17:36:32.4627853Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-778754d04ea33099.xml (deflated 36%) 2024-04-03T17:36:32.4629489Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7306947e79c89f8c.xml (deflated 42%) 2024-04-03T17:36:32.4631123Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-76200d21895cf2f7.xml (deflated 43%) 2024-04-03T17:36:32.4632765Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dbf7a9af645d175c.xml (deflated 43%) 2024-04-03T17:36:32.4634396Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-37d68d163ea9ce8c.xml (deflated 43%) 2024-04-03T17:36:32.4636125Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ec413d0029a5d02.xml (deflated 44%) 2024-04-03T17:36:32.4637781Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-53f40c91e1745e80.xml (deflated 45%) 2024-04-03T17:36:32.4639424Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2384bbc17394e92d.xml (deflated 44%) 2024-04-03T17:36:32.4641043Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-109edaac52e64e0a.xml (deflated 44%) 2024-04-03T17:36:32.4642837Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5181562501950eeb.xml (deflated 44%) 2024-04-03T17:36:32.4644471Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-383335fdd3cbb6d3.xml (deflated 44%) 2024-04-03T17:36:32.4646112Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-826158a048e218bf.xml (deflated 44%) 2024-04-03T17:36:32.4647732Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0442a74012869d61.xml (deflated 43%) 2024-04-03T17:36:32.4649362Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c50ce4068a251327.xml (deflated 36%) 2024-04-03T17:36:32.4651000Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-373c3ed31f336b20.xml (deflated 50%) 2024-04-03T17:36:32.4652661Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c98984c7ba1fe983.xml (deflated 42%) 2024-04-03T17:36:32.4654292Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6af29ae3ebe153ab.xml (deflated 42%) 2024-04-03T17:36:32.4655931Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ce70e23de3c86240.xml (deflated 42%) 2024-04-03T17:36:32.4657666Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-11724b4c07c47fa2.xml (deflated 36%) 2024-04-03T17:36:32.4659314Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5b316e1e6782c2ef.xml (deflated 42%) 2024-04-03T17:36:32.4660948Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2cd9dd622ed0b9b3.xml (deflated 56%) 2024-04-03T17:36:32.4662615Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f6520c356be7f77e.xml (deflated 42%) 2024-04-03T17:36:32.4664259Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e7d26b02e7e498b.xml (deflated 42%) 2024-04-03T17:36:32.4665901Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f29f220482816ff8.xml (deflated 42%) 2024-04-03T17:36:32.4667533Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5d0ce0d001885741.xml (deflated 43%) 2024-04-03T17:36:32.4669173Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-56f18bda5be43c87.xml (deflated 42%) 2024-04-03T17:36:32.4670817Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ec36501cf21071cc.xml (deflated 48%) 2024-04-03T17:36:32.4672473Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fb189f4844ea97b4.xml (deflated 42%) 2024-04-03T17:36:32.4674107Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cb232d6b7481ac2e.xml (deflated 41%) 2024-04-03T17:36:32.4675828Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9116eaf5d9188f00.xml (deflated 42%) 2024-04-03T17:36:32.4677485Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2df88bfa0bf98c45.xml (deflated 42%) 2024-04-03T17:36:32.4679141Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-113b5fc1fbab8484.xml (deflated 42%) 2024-04-03T17:36:32.4680777Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ffcf376103dca32.xml (deflated 42%) 2024-04-03T17:36:32.4682509Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-545fc72153c136b3.xml (deflated 42%) 2024-04-03T17:36:32.4684218Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a19b44ed9d8b4c09.xml (deflated 44%) 2024-04-03T17:36:32.4685861Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5bd7d93a082c05ea.xml (deflated 43%) 2024-04-03T17:36:32.4687480Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-325b8959b4df7d05.xml (deflated 47%) 2024-04-03T17:36:32.4689126Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-85cc9f268b8b554c.xml (deflated 42%) 2024-04-03T17:36:32.4690773Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2ddf09e41201e0f9.xml (deflated 42%) 2024-04-03T17:36:32.4692415Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dec500857e9988da.xml (deflated 42%) 2024-04-03T17:36:32.4694036Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6359cab41facc308.xml (deflated 35%) 2024-04-03T17:36:32.4695742Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4626935b66562567.xml (deflated 36%) 2024-04-03T17:36:32.4697392Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca927adef997bdfc.xml (deflated 42%) 2024-04-03T17:36:32.4699152Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c8f44a95c747b82.xml (deflated 35%) 2024-04-03T17:36:32.4700799Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0dc8caa19f5201d0.xml (deflated 43%) 2024-04-03T17:36:32.4702473Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4315687616e91af.xml (deflated 35%) 2024-04-03T17:36:32.4704597Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-58755a7f45dbef3c.xml (deflated 36%) 2024-04-03T17:36:32.4706262Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-706f23ca6d0cfb53.xml (deflated 35%) 2024-04-03T17:36:32.4707887Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c29440bf5dd6e86d.xml (deflated 37%) 2024-04-03T17:36:32.4709535Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1bfad1e293bc1d69.xml (deflated 36%) 2024-04-03T17:36:32.4711195Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-47ec96ce63024b89.xml (deflated 42%) 2024-04-03T17:36:32.4712838Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5632e2d42ee783d3.xml (deflated 35%) 2024-04-03T17:36:32.4714465Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-48a27979fe96ad3d.xml (deflated 45%) 2024-04-03T17:36:32.4716263Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d0941b195e26c28a.xml (deflated 43%) 2024-04-03T17:36:32.4718722Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9cd51ddb3f5a72ab.xml (deflated 43%) 2024-04-03T17:36:32.4721090Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c83eafd7cf70d60a.xml (deflated 43%) 2024-04-03T17:36:32.4722849Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d8b240031346a3da.xml (deflated 42%) 2024-04-03T17:36:32.4724661Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d4ec47b0222f1739.xml (deflated 42%) 2024-04-03T17:36:32.4726759Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1438ca7aba79eff.xml (deflated 42%) 2024-04-03T17:36:32.4728422Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1abbd5f8aaae5c1a.xml (deflated 42%) 2024-04-03T17:36:32.4730059Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a831d86bd66d1b9.xml (deflated 36%) 2024-04-03T17:36:32.4731781Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4dc49d88b2641e88.xml (deflated 36%) 2024-04-03T17:36:32.4733763Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-734a2607df586c0f.xml (deflated 36%) 2024-04-03T17:36:32.4735408Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5e90c0f4842a050.xml (deflated 36%) 2024-04-03T17:36:32.4737153Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9687a8b3f15c063e.xml (deflated 45%) 2024-04-03T17:36:32.4739200Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-43891f89aef2a0e3.xml (deflated 36%) 2024-04-03T17:36:32.4740857Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a15a9b52e1e85836.xml (deflated 36%) 2024-04-03T17:36:32.4742933Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4d9dd31560c323bf.xml (deflated 36%) 2024-04-03T17:36:32.4744557Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-020e6f18b9504459.xml (deflated 36%) 2024-04-03T17:36:32.4746213Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f17d51a317b069e6.xml (deflated 36%) 2024-04-03T17:36:32.4748156Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-de3b2dc39104f1b3.xml (deflated 36%) 2024-04-03T17:36:32.4749919Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-db185b44208a6522.xml (deflated 42%) 2024-04-03T17:36:32.4751541Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-338cf69557f06084.xml (deflated 43%) 2024-04-03T17:36:32.4753178Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8dfbd8304a893531.xml (deflated 57%) 2024-04-03T17:36:32.4754812Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9321db7cd98583ca.xml (deflated 41%) 2024-04-03T17:36:32.4756614Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-22a7a5bc4b70b017.xml (deflated 41%) 2024-04-03T17:36:32.4758243Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c0cd331eaccc5a4.xml (deflated 42%) 2024-04-03T17:36:32.4759898Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3803002630a7d76.xml (deflated 43%) 2024-04-03T17:36:32.4761538Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a792eea28361a079.xml (deflated 42%) 2024-04-03T17:36:32.4763690Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e56f3ebc3f79978.xml (deflated 36%) 2024-04-03T17:36:32.4765463Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-11d8cd6d87c84b6b.xml (deflated 35%) 2024-04-03T17:36:32.4767428Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d983d02823a416e1.xml (deflated 36%) 2024-04-03T17:36:32.4769165Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a10561b96a33c4db.xml (deflated 36%) 2024-04-03T17:36:32.4770810Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d2d508cb6473ef7f.xml (deflated 35%) 2024-04-03T17:36:32.4772864Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32bc5e519178e481.xml (deflated 43%) 2024-04-03T17:36:32.4774505Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0caa6cec1b16bff4.xml (deflated 44%) 2024-04-03T17:36:32.4776580Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f29552ac2f3af05e.xml (deflated 36%) 2024-04-03T17:36:32.4778229Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fe981e70ec0f4b67.xml (deflated 35%) 2024-04-03T17:36:32.4779857Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-07fb84745c8d14b4.xml (deflated 35%) 2024-04-03T17:36:32.4782012Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b71ed2768ff8d678.xml (deflated 35%) 2024-04-03T17:36:32.4783915Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-06aebeabc27d96fb.xml (deflated 36%) 2024-04-03T17:36:32.4786626Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b0aa6c366cd6f4a2.xml (deflated 42%) 2024-04-03T17:36:32.4789732Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-23f53d445c7e5da6.xml (deflated 43%) 2024-04-03T17:36:32.4792832Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-638b2f9bd584cff9.xml (deflated 43%) 2024-04-03T17:36:32.4812663Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0d44479e417cae92.xml (deflated 44%) 2024-04-03T17:36:32.4816407Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c78374223edeea88.xml (deflated 44%) 2024-04-03T17:36:32.4818547Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-661925294bb7ed9c.xml (deflated 45%) 2024-04-03T17:36:32.4820201Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1b530fff50c28c2.xml (deflated 44%) 2024-04-03T17:36:32.4821877Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0da2804faa005fe5.xml (deflated 44%) 2024-04-03T17:36:32.4823945Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fee21004b9b630d0.xml (deflated 44%) 2024-04-03T17:36:32.4825595Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d58ca56f9c635537.xml (deflated 44%) 2024-04-03T17:36:32.4827337Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-35e689f707dc9699.xml (deflated 44%) 2024-04-03T17:36:32.4829615Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6b370b65fe066587.xml (deflated 42%) 2024-04-03T17:36:32.4831261Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4eccd5eab7e896c9.xml (deflated 34%) 2024-04-03T17:36:32.4833447Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bac74dfceb519c77.xml (deflated 36%) 2024-04-03T17:36:32.4835142Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b4172e4b2545e2d5.xml (deflated 42%) 2024-04-03T17:36:32.4836993Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2d081478015cd875.xml (deflated 42%) 2024-04-03T17:36:32.4839065Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fb0c0ba5857cded8.xml (deflated 42%) 2024-04-03T17:36:32.4840808Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-546ed4a3247b670c.xml (deflated 36%) 2024-04-03T17:36:32.4842511Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8c7487d35d43b597.xml (deflated 42%) 2024-04-03T17:36:32.4844260Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c5f9396cbbcaef7a.xml (deflated 56%) 2024-04-03T17:36:32.4845900Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf2edb3895285596.xml (deflated 42%) 2024-04-03T17:36:32.4847654Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e59389a4a74276a.xml (deflated 42%) 2024-04-03T17:36:32.4849308Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f671a5195977342a.xml (deflated 42%) 2024-04-03T17:36:32.4850955Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cd4231aba4b95273.xml (deflated 42%) 2024-04-03T17:36:32.4852581Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-62d049a0fd84be94.xml (deflated 41%) 2024-04-03T17:36:32.4854228Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5242a7de1332823e.xml (deflated 48%) 2024-04-03T17:36:32.4855869Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c02a460ac2b9336.xml (deflated 42%) 2024-04-03T17:36:32.4857545Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f7f04f6d32762f04.xml (deflated 42%) 2024-04-03T17:36:32.4859192Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d4c16541931fe08c.xml (deflated 42%) 2024-04-03T17:36:32.4861269Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30fc462c0d9f1380.xml (deflated 42%) 2024-04-03T17:36:32.4862905Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-27c6604844069661.xml (deflated 42%) 2024-04-03T17:36:32.4864970Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cc8acb3576f74647.xml (deflated 42%) 2024-04-03T17:36:32.4866610Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fe7161c608bdd018.xml (deflated 41%) 2024-04-03T17:36:32.4868661Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c042d13622811f72.xml (deflated 44%) 2024-04-03T17:36:32.4870300Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-95066ab4f9a142a2.xml (deflated 43%) 2024-04-03T17:36:32.4871997Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-10115ba538fdd051.xml (deflated 47%) 2024-04-03T17:36:32.4874018Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-686c67edfd8cbeb9.xml (deflated 42%) 2024-04-03T17:36:32.4875928Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3c285c68432a68ff.xml (deflated 42%) 2024-04-03T17:36:32.4877580Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a79b416ddfbda94e.xml (deflated 42%) 2024-04-03T17:36:32.4879236Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6c4da03370e5e04f.xml (deflated 35%) 2024-04-03T17:36:32.4881050Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-07ba0644a1ba506a.xml (deflated 36%) 2024-04-03T17:36:32.4882754Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70fda6948fb17879.xml (deflated 42%) 2024-04-03T17:36:32.4884399Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-99d6aa7c20fda29a.xml (deflated 35%) 2024-04-03T17:36:32.4886057Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3047612a1f165cd5.xml (deflated 43%) 2024-04-03T17:36:32.4887690Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e88455c083093445.xml (deflated 35%) 2024-04-03T17:36:32.4889409Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6f5a707519f7462c.xml (deflated 36%) 2024-04-03T17:36:32.4891055Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d048b6a295a5470a.xml (deflated 35%) 2024-04-03T17:36:32.4892697Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2061a426e184b474.xml (deflated 36%) 2024-04-03T17:36:32.4894338Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-25337328be8769d6.xml (deflated 36%) 2024-04-03T17:36:32.4895966Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6bd8c2f430852303.xml (deflated 42%) 2024-04-03T17:36:32.4897614Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8553b3b075ea5dcb.xml (deflated 35%) 2024-04-03T17:36:32.4899254Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2fdd5dcbc9226189.xml (deflated 45%) 2024-04-03T17:36:32.4900895Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-195617ae64a3a293.xml (deflated 43%) 2024-04-03T17:36:32.4902513Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc0047029381649c.xml (deflated 43%) 2024-04-03T17:36:32.4904157Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e980543aab96a719.xml (deflated 43%) 2024-04-03T17:36:32.4905798Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b33506c21e33015.xml (deflated 42%) 2024-04-03T17:36:32.4907440Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0962fbd926b6f13a.xml (deflated 42%) 2024-04-03T17:36:32.4909492Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e16888ec7c0c9cd8.xml (deflated 42%) 2024-04-03T17:36:32.4911130Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dd228fb40b92fea6.xml (deflated 42%) 2024-04-03T17:36:32.4912771Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f319f3be8780d43.xml (deflated 35%) 2024-04-03T17:36:32.4914501Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1771f65b3e5e030f.xml (deflated 35%) 2024-04-03T17:36:32.4916256Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-31e3962f583a56c6.xml (deflated 35%) 2024-04-03T17:36:32.4917904Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f09132db81f636a.xml (deflated 36%) 2024-04-03T17:36:32.4919563Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bc63540f842e0807.xml (deflated 45%) 2024-04-03T17:36:32.4921210Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ac7179d8037d4aaf.xml (deflated 36%) 2024-04-03T17:36:32.4922915Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fe80fe6a7ee7ca55.xml (deflated 36%) 2024-04-03T17:36:32.4924573Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c439f97d5d68a6e2.xml (deflated 36%) 2024-04-03T17:36:32.4926216Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-067eed542628c494.xml (deflated 36%) 2024-04-03T17:36:32.4927877Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e07ac46ecd46bf1.xml (deflated 35%) 2024-04-03T17:36:32.4929601Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-244eb5723f663741.xml (deflated 35%) 2024-04-03T17:36:32.4931246Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c836fbab8b63c4f4.xml (deflated 42%) 2024-04-03T17:36:32.4932898Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd1fd6ec37164439.xml (deflated 42%) 2024-04-03T17:36:32.4958146Z ##[group]Run # Remove any previous test reports if they exist 2024-04-03T17:36:32.4958783Z # Remove any previous test reports if they exist 2024-04-03T17:36:32.4959266Z rm -f logs-*.zip 2024-04-03T17:36:32.4959924Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-04-03T17:36:32.4960702Z # so check to see if the file exists first 2024-04-03T17:36:32.4961170Z if [ -f 'usage_log.txt' ]; then 2024-04-03T17:36:32.4961686Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-04-03T17:36:32.4962152Z fi 2024-04-03T17:36:32.4962471Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-04-03T17:36:32.4963111Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-04-03T17:36:32.4963585Z fi 2024-04-03T17:36:32.4970985Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T17:36:32.4971454Z env: 2024-04-03T17:36:32.4971722Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:36:32.4972323Z DOCKER_CONTAINER_ID: b0f53da97a9da663582739611fc2277e27fd12f5b71cdfe54025ecd3466cc388 2024-04-03T17:36:32.4973046Z FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_ 2024-04-03T17:36:32.4973504Z ##[endgroup] 2024-04-03T17:36:32.5303554Z adding: usage_log.txt (deflated 96%) 2024-04-03T17:36:32.5564118Z adding: test/test-reports/distributed._tensor.test_pointwise_ops_1.1_a6806cb85994eafd_.log (deflated 69%) 2024-04-03T17:36:32.5569148Z adding: test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_fa314eb63de0fd16_.log (deflated 54%) 2024-04-03T17:36:32.5570529Z adding: test/test-reports/distributed.test_fake_pg_1.1_d32589e6657fb4db_.log (deflated 73%) 2024-04-03T17:36:32.5571620Z adding: test/test-reports/distributed._spmd.test_graph_utils_1.1_5d5d3bef216be4c0_.log (stored 0%) 2024-04-03T17:36:32.5572750Z adding: test/test-reports/distributed._shard.test_sharder_1.1_83f6f5faaddf25bd_.log (deflated 59%) 2024-04-03T17:36:32.5579198Z adding: test/test-reports/distributed._tensor.test_dtensor_1.1_33e020d2158f5785_.log (deflated 94%) 2024-04-03T17:36:32.5580539Z adding: test/test-reports/distributed.tensor.parallel.test_tp_style_1.1_11c8558d029f4ef3_.log (deflated 90%) 2024-04-03T17:36:32.5581752Z adding: test/test-reports/distributed._spmd.test_data_parallel_1.1_789eae00e80cee81_.log (stored 0%) 2024-04-03T17:36:32.5582917Z adding: test/test-reports/distributed.fsdp.test_fsdp_overlap_1.1_820a1f0405dce9cc_.log (deflated 59%) 2024-04-03T17:36:32.5588600Z adding: test/test-reports/distributed.test_distributed_spawn_2.3_4e662b051c6bb258_.log (deflated 12%) 2024-04-03T17:36:32.5589727Z adding: test/test-reports/distributed.test_c10d_logger_1.1_ee87361701f26332_.log (deflated 60%) 2024-04-03T17:36:32.5596948Z adding: test/test-reports/distributed.test_distributed_spawn_2.3_1f5cef3fdf6ecb33_.log (deflated 12%) 2024-04-03T17:36:32.5598254Z adding: test/test-reports/distributed.test_multi_threaded_pg_1.1_e722a700d2b2e7ee_.log (deflated 85%) 2024-04-03T17:36:32.5629973Z adding: test/test-reports/distributed.test_distributed_spawn_2.3_08865f72cafa1d62_.log (deflated 95%) 2024-04-03T17:36:32.5632000Z adding: test/test-reports/distributed.rpc.test_share_memory_1.1_73b66a8d4cdff0d9_.log (deflated 57%) 2024-04-03T17:36:32.5677532Z adding: test/test-reports/distributed.test_distributed_spawn_2.3_fd23c646647f4173_.log (deflated 95%) 2024-04-03T17:36:32.5679073Z adding: test/test-reports/distributed._tensor.test_op_strategy_1.1_65c9f6b3ba596f9e_.log (deflated 79%) 2024-04-03T17:36:32.5692726Z adding: test/test-reports/distributed.checkpoint.test_tp_checkpoint_1.1_c032f8f6c8cb2683_.log (deflated 61%) 2024-04-03T17:36:32.5694740Z adding: test/test-reports/distributed.fsdp.test_fsdp_core_1.1_e93302c253b9101e_.log (deflated 92%) 2024-04-03T17:36:32.5718010Z adding: test/test-reports/distributed._tensor.test_tensor_ops_1.1_1cac935f4887e855_.log (deflated 90%) 2024-04-03T17:36:32.5719243Z adding: test/test-reports/distributed.checkpoint.test_fsspec_1.1_8646094d777e2c5b_.log (deflated 56%) 2024-04-03T17:36:32.5729768Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_init_1.1_bf64ce7a055b8394_.log (deflated 85%) 2024-04-03T17:36:32.5731091Z adding: test/test-reports/distributed.checkpoint.e2e.test_pipeline_1.1_a266e90144a3ed76_.log (deflated 52%) 2024-04-03T17:36:32.5735356Z adding: test/test-reports/distributed._tensor.test_xla_integration_1.1_36479aca781c3e15_.log (deflated 66%) 2024-04-03T17:36:32.5736664Z adding: test/test-reports/distributed._tensor.test_embedding_ops_1.1_904e8eee66d6f76b_.log (deflated 82%) 2024-04-03T17:36:32.5742891Z adding: test/test-reports/distributed.checkpoint.test_format_utils_1.1_a6010019ccb53782_.log (deflated 85%) 2024-04-03T17:36:32.5744369Z adding: test/test-reports/distributed.fsdp.test_fsdp_meta_1.1_f974e25cb575ac40_.log (deflated 84%) 2024-04-03T17:36:32.5751335Z adding: test/test-reports/distributed.test_dynamo_distributed_1.1_d88d9e2c9d7eb77b_.log (deflated 86%) 2024-04-03T17:36:32.5752965Z adding: test/test-reports/distributed._tensor.test_experimental_ops_1.1_d083603fe615f616_.log (deflated 82%) 2024-04-03T17:36:32.5757645Z adding: test/test-reports/distributed.checkpoint.test_fsdp_optim_state_1.1_227ae436e16e9c78_.log (deflated 63%) 2024-04-03T17:36:32.5759290Z adding: test/test-reports/distributed.fsdp.test_fsdp_misc_1.1_eb18615a869e9360_.log (deflated 84%) 2024-04-03T17:36:32.5783043Z adding: test/test-reports/distributed._tensor.test_api_1.1_f7565b272946c558_.log (deflated 89%) 2024-04-03T17:36:32.5789963Z adding: test/test-reports/distributed.rpc.test_faulty_agent_1.1_5f7695819a10f268_.log (deflated 96%) 2024-04-03T17:36:32.5799027Z adding: test/test-reports/distributed._tensor.test_random_ops_1.1_27332ff31679919e_.log (deflated 75%) 2024-04-03T17:36:32.5800161Z adding: test/test-reports/distributed.test_nccl_1.1_797bae420066f849_.log (deflated 49%) 2024-04-03T17:36:32.5818356Z adding: test/test-reports/distributed.test_device_mesh_1.1_041398b27a05e6ca_.log (deflated 92%) 2024-04-03T17:36:32.5819759Z adding: test/test-reports/distributed.fsdp.test_fsdp_memory_1.1_ac72d4849852c254_.log (deflated 60%) 2024-04-03T17:36:32.5823115Z adding: test/test-reports/distributed.checkpoint.test_dtensor_resharding_1.1_9d8ede7695361473_.log (deflated 72%) 2024-04-03T17:36:32.5824398Z adding: test/test-reports/distributed._tensor.test_convolution_ops_1.1_3f2f0898e35e09cf_.log (deflated 69%) 2024-04-03T17:36:32.5827236Z adding: test/test-reports/distributed._tensor.experimental.test_tp_transform_1.1_f38eb6c192607b12_.log (deflated 78%) 2024-04-03T17:36:32.5831514Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_autograd_1.1_f8665121d4d73bf0_.log (deflated 67%) 2024-04-03T17:36:32.5832797Z adding: test/test-reports/distributed.fsdp.test_shard_utils_1.1_7d0dad92f6e0e480_.log (deflated 59%) 2024-04-03T17:36:32.5836551Z adding: test/test-reports/distributed.test_compute_comm_reordering_1.1_ca74324cfcf30973_.log (deflated 75%) 2024-04-03T17:36:32.5837814Z adding: test/test-reports/distributed._tools.test_memory_tracker_1.1_8b750102f351db5a_.log (deflated 54%) 2024-04-03T17:36:32.5839921Z adding: test/test-reports/distributed.tensor.parallel.test_ddp_2d_parallel_1.1_cd2542b98e34b99f_.log (deflated 56%) 2024-04-03T17:36:32.5842420Z adding: test/test-reports/distributed.checkpoint.test_dtensor_checkpoint_1.1_e7c87f9e13dc86ef_.log (deflated 55%) 2024-04-03T17:36:32.5845112Z adding: test/test-reports/distributed._composable.test_compose_1.1_0f88bf4630f41b64_.log (deflated 78%) 2024-04-03T17:36:32.5846302Z adding: test/test-reports/distributed.optim.test_named_optimizer_1.1_d9e65fdc400309c1_.log (stored 0%) 2024-04-03T17:36:32.5847376Z adding: test/test-reports/cpp.test_cpp_rpc_1.1_91be82aa2ccd8a07_.log (deflated 78%) 2024-04-03T17:36:32.5848556Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_memory_1.1_0dc73789d534c1e5_.log (deflated 55%) 2024-04-03T17:36:32.5849937Z adding: test/test-reports/distributed.checkpoint.e2e.test_e2e_save_and_load_1.1_fe2c4d2d84e6026e_.log (deflated 83%) 2024-04-03T17:36:32.5851327Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_state_dict_1.1_f7fa839b5fd5e915_.log (deflated 63%) 2024-04-03T17:36:32.5853376Z adding: test/test-reports/distributed._shard.sharded_optim.test_sharded_optim_1.1_b8caa0a304e3cdb7_.log (deflated 62%) 2024-04-03T17:36:32.5855884Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_f008010bcd5b5ae1_.log (deflated 74%) 2024-04-03T17:36:32.5857129Z adding: test/test-reports/distributed.test_store_1.1_a2edbf29f2c8642f_.log (deflated 95%) 2024-04-03T17:36:32.5859230Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_mixed_precision_1.1_cbbc96b825c99539_.log (deflated 58%) 2024-04-03T17:36:32.5861954Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_3601819a77c13780_.log (deflated 83%) 2024-04-03T17:36:32.5864008Z adding: test/test-reports/distributed.tensor.parallel.test_parallelize_api_1.1_b154395c4836d606_.log (deflated 88%) 2024-04-03T17:36:32.5865553Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_util_1.1_fdc8c5f5885e449f_.log (deflated 57%) 2024-04-03T17:36:32.5867957Z adding: test/test-reports/distributed.test_inductor_collectives_1.1_41b9cd5a6ae916a9_.log (deflated 94%) 2024-04-03T17:36:32.5870299Z adding: test/test-reports/distributed.fsdp.test_fsdp_backward_prefetch_1.1_c243a53aa6046137_.log (deflated 55%) 2024-04-03T17:36:32.5871911Z adding: test/test-reports/distributed._composable.test_replicate_1.1_e986aeb3d978d0d6_.log (deflated 74%) 2024-04-03T17:36:32.5873632Z adding: test/test-reports/distributed._shard.sharding_plan.test_sharding_plan_1.1_6227847af6ad7149_.log (deflated 66%) 2024-04-03T17:36:32.5875887Z adding: test/test-reports/distributed.test_launcher_1.1_ee11cd3a27fa1a5d_.log (deflated 58%) 2024-04-03T17:36:32.5878421Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_clip_grad_norm__1.1_7d59e50b4c6f60eb_.log (deflated 63%) 2024-04-03T17:36:32.5881314Z adding: test/test-reports/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion_1.1_bfa7bf9286935c1e_.log (deflated 57%) 2024-04-03T17:36:32.5883717Z adding: test/test-reports/distributed._tensor.test_optimizers_1.1_b7a9f8258a370dc6_.log (deflated 92%) 2024-04-03T17:36:32.5884894Z adding: test/test-reports/distributed.fsdp.test_fsdp_pure_fp16_1.1_63671cd20dfb8e33_.log (deflated 61%) 2024-04-03T17:36:32.5886117Z adding: test/test-reports/distributed.fsdp.test_fsdp_optim_state_1.1_bb2ec51a96d40a77_.log (deflated 93%) 2024-04-03T17:36:32.5887420Z adding: test/test-reports/distributed._shard.sharding_spec.test_sharding_spec_1.1_61c905b36aa2fbc1_.log (deflated 77%) 2024-04-03T17:36:32.5888769Z adding: test/test-reports/distributed.fsdp.test_fsdp_dtensor_state_dict_1.1_92eea54192d92e0e_.log (deflated 86%) 2024-04-03T17:36:32.5890026Z adding: test/test-reports/distributed.checkpoint.test_checkpoint_1.1_bc3bf59fd091d0f3_.log (deflated 81%) 2024-04-03T17:36:32.5891263Z adding: test/test-reports/distributed.nn.jit.test_instantiator_1.1_5681b5ed2dc74aaa_.log (deflated 63%) 2024-04-03T17:36:32.5892548Z adding: test/test-reports/distributed.checkpoint.test_file_system_checkpoint_1.1_7837f255faea5980_.log (deflated 74%) 2024-04-03T17:36:32.5893850Z adding: test/test-reports/distributed.fsdp.test_fsdp_freezing_weights_1.1_11fa00d472c76dd6_.log (deflated 93%) 2024-04-03T17:36:32.5895201Z adding: test/test-reports/distributed.pipeline.sync.test_pipe_1.1_9ddb0f0081239f3f_.log (deflated 83%) 2024-04-03T17:36:32.5896416Z adding: test/test-reports/distributed.fsdp.test_fsdp_fine_tune_1.1_c31aaddc70e2d398_.log (deflated 68%) 2024-04-03T17:36:32.5897682Z adding: test/test-reports/distributed.optim.test_apply_optimizer_in_backward_1.1_a37fc83f4cbabb8c_.log (stored 0%) 2024-04-03T17:36:32.5898933Z adding: test/test-reports/distributed.fsdp.test_fsdp_checkpoint_1.1_d2423d4fbd029049_.log (deflated 88%) 2024-04-03T17:36:32.5900165Z adding: test/test-reports/distributed.pipeline.sync.test_worker_1.1_452ddf41804cba69_.log (deflated 64%) 2024-04-03T17:36:32.5901436Z adding: test/test-reports/distributed.pipeline.sync.test_transparency_1.1_61ea1258edadc26f_.log (deflated 53%) 2024-04-03T17:36:32.5902809Z adding: test/test-reports/distributed.pipeline.sync.skip.test_inspect_skip_layout_1.1_69a68aecfa651e38_.log (deflated 67%) 2024-04-03T17:36:32.5904320Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_optim_checkpoint_1.1_bd80261e2889144b_.log (deflated 65%) 2024-04-03T17:36:32.5905706Z adding: test/test-reports/distributed.fsdp.test_fsdp_unshard_params_1.1_846a6e45b89ae422_.log (deflated 82%) 2024-04-03T17:36:32.5906943Z adding: test/test-reports/distributed.fsdp.test_fsdp_hybrid_shard_1.1_542bf96b73b55345_.log (deflated 73%) 2024-04-03T17:36:32.5908197Z adding: test/test-reports/distributed.pipeline.sync.test_microbatch_1.1_22981cfa9dded8ac_.log (deflated 70%) 2024-04-03T17:36:32.5909673Z adding: test/test-reports/distributed.pipeline.sync.test_checkpoint_1.1_49627a3d14d60058_.log (deflated 67%) 2024-04-03T17:36:32.5911967Z adding: test/test-reports/distributed.pipeline.sync.skip.test_portal_1.1_b3d81fa8d1acf95a_.log (deflated 71%) 2024-04-03T17:36:32.5913982Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_init_1.1_c4edff395dec7829_.log (deflated 67%) 2024-04-03T17:36:32.5927072Z adding: test/test-reports/distributed.rpc.test_tensorpipe_agent_1.3_4d2b8ceccdba7fab_.log (deflated 97%) 2024-04-03T17:36:32.5974987Z adding: test/test-reports/distributed.rpc.test_tensorpipe_agent_3.3_3f194c313be972fb_.log (deflated 97%) 2024-04-03T17:36:32.5976942Z adding: test/test-reports/distributed.pipeline.sync.skip.test_verify_skippables_1.1_8de2ff51501c1f1d_.log (deflated 71%) 2024-04-03T17:36:32.5978655Z adding: test/test-reports/distributed.test_data_parallel_1.1_c66b2111ea83ccb8_.log (deflated 88%) 2024-04-03T17:36:32.5980024Z adding: test/test-reports/distributed.optim.test_zero_redundancy_optimizer_1.1_29c88b6d392601f0_.log (deflated 91%) 2024-04-03T17:36:32.5982399Z adding: test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_1.1_13ea9467df1ec5ca_.log (deflated 91%) 2024-04-03T17:36:32.5983884Z adding: test/test-reports/distributed.elastic.events.lib_test_1.1_9e5f3925f7b4731d_.log (deflated 72%) 2024-04-03T17:36:32.5986205Z adding: test/test-reports/distributed.elastic.metrics.api_test_1.1_15c298325372f8c2_.log (deflated 62%) 2024-04-03T17:36:32.5987674Z adding: test/test-reports/distributed.elastic.multiprocessing.api_test_1.1_652e64d633f42125_.log (deflated 88%) 2024-04-03T17:36:32.5988925Z adding: test/test-reports/distributed.elastic.timer.api_test_1.1_920c604bfdbfdd08_.log (stored 0%) 2024-04-03T17:36:32.5990156Z adding: test/test-reports/distributed.elastic.timer.local_timer_example_1.1_2d6f6a70c4dfe384_.log (deflated 68%) 2024-04-03T17:36:32.5991490Z adding: test/test-reports/distributed.elastic.timer.local_timer_test_1.1_e89f62ea197371e8_.log (deflated 77%) 2024-04-03T17:36:32.5992899Z adding: test/test-reports/distributed.elastic.utils.distributed_test_1.1_1618c89267595fda_.log (deflated 72%) 2024-04-03T17:36:32.6045742Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-04-03T17:36:32.6046150Z with: 2024-04-03T17:36:32.6046417Z s3-bucket: gha-artifacts 2024-04-03T17:36:32.6046825Z s3-prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T17:36:32.6047262Z retention-days: 14 2024-04-03T17:36:32.6047703Z if-no-files-found: warn 2024-04-03T17:36:32.6048047Z path: test-jsons-*.zip 2024-04-03T17:36:32.6048352Z name: artifact 2024-04-03T17:36:32.6048634Z region: us-east-1 2024-04-03T17:36:32.6048912Z env: 2024-04-03T17:36:32.6049152Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:36:32.6049746Z DOCKER_CONTAINER_ID: b0f53da97a9da663582739611fc2277e27fd12f5b71cdfe54025ecd3466cc388 2024-04-03T17:36:32.6050394Z ##[endgroup] 2024-04-03T17:36:33.3530313Z NOTE: s3-prefix specified, ignoring name parameter 2024-04-03T17:36:33.3531142Z With the provided path, there will be 1 file uploaded 2024-04-03T17:36:33.3531746Z Uploading to s3 prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T17:36:33.5006307Z Starting upload of test-jsons-test-distributed-1-2-linux.2xlarge_.zip 2024-04-03T17:36:33.7095447Z Finished upload of test-jsons-test-distributed-1-2-linux.2xlarge_.zip 2024-04-03T17:36:33.7247425Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-04-03T17:36:33.7247848Z with: 2024-04-03T17:36:33.7248124Z s3-bucket: gha-artifacts 2024-04-03T17:36:33.7248529Z s3-prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T17:36:33.7248965Z retention-days: 14 2024-04-03T17:36:33.7249282Z if-no-files-found: error 2024-04-03T17:36:33.7249625Z path: test-reports-*.zip 2024-04-03T17:36:33.7249936Z name: artifact 2024-04-03T17:36:33.7250213Z region: us-east-1 2024-04-03T17:36:33.7250489Z env: 2024-04-03T17:36:33.7250727Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:36:33.7251314Z DOCKER_CONTAINER_ID: b0f53da97a9da663582739611fc2277e27fd12f5b71cdfe54025ecd3466cc388 2024-04-03T17:36:33.7251975Z ##[endgroup] 2024-04-03T17:36:34.0650796Z NOTE: s3-prefix specified, ignoring name parameter 2024-04-03T17:36:34.0651473Z With the provided path, there will be 1 file uploaded 2024-04-03T17:36:34.0652064Z Uploading to s3 prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T17:36:34.0686800Z Starting upload of test-reports-test-distributed-1-2-linux.2xlarge_.zip 2024-04-03T17:36:34.2446442Z Finished upload of test-reports-test-distributed-1-2-linux.2xlarge_.zip 2024-04-03T17:36:34.2600108Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-04-03T17:36:34.2600532Z with: 2024-04-03T17:36:34.2600798Z s3-bucket: gha-artifacts 2024-04-03T17:36:34.2601200Z s3-prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T17:36:34.2601640Z retention-days: 14 2024-04-03T17:36:34.2601959Z if-no-files-found: ignore 2024-04-03T17:36:34.2602295Z path: logs-*.zip 2024-04-03T17:36:34.2602566Z name: artifact 2024-04-03T17:36:34.2603070Z region: us-east-1 2024-04-03T17:36:34.2603348Z env: 2024-04-03T17:36:34.2603586Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:36:34.2604176Z DOCKER_CONTAINER_ID: b0f53da97a9da663582739611fc2277e27fd12f5b71cdfe54025ecd3466cc388 2024-04-03T17:36:34.2604824Z ##[endgroup] 2024-04-03T17:36:34.6020247Z NOTE: s3-prefix specified, ignoring name parameter 2024-04-03T17:36:34.6020904Z With the provided path, there will be 1 file uploaded 2024-04-03T17:36:34.6021534Z Uploading to s3 prefix: pytorch/pytorch/8536950919/2/artifact 2024-04-03T17:36:34.6055800Z Starting upload of logs-test-distributed-1-2-linux.2xlarge_.zip 2024-04-03T17:36:34.7866836Z Finished upload of logs-test-distributed-1-2-linux.2xlarge_.zip 2024-04-03T17:36:34.8020210Z ##[group]Run # shellcheck disable=SC2156 2024-04-03T17:36:34.8020664Z # shellcheck disable=SC2156 2024-04-03T17:36:34.8021453Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-04-03T17:36:34.8029108Z shell: /usr/bin/bash -e {0} 2024-04-03T17:36:34.8029422Z env: 2024-04-03T17:36:34.8029679Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:36:34.8030277Z DOCKER_CONTAINER_ID: b0f53da97a9da663582739611fc2277e27fd12f5b71cdfe54025ecd3466cc388 2024-04-03T17:36:34.8030910Z ##[endgroup] 2024-04-03T17:36:36.2576898Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-04-03T17:36:36.2577427Z with: 2024-04-03T17:36:36.2577660Z env: 2024-04-03T17:36:36.2577913Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:36:36.2578497Z DOCKER_CONTAINER_ID: b0f53da97a9da663582739611fc2277e27fd12f5b71cdfe54025ecd3466cc388 2024-04-03T17:36:36.2579139Z ##[endgroup] 2024-04-03T17:36:36.2599292Z ##[group]Run set -eou pipefail 2024-04-03T17:36:36.2599683Z set -eou pipefail 2024-04-03T17:36:36.2600014Z  2024-04-03T17:36:36.2600509Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-04-03T17:36:36.2601130Z for _ in $(seq 1440); do 2024-04-03T17:36:36.2601590Z  # Break if no ssh session exists anymore 2024-04-03T17:36:36.2602071Z  if [ "$(who)" = "" ]; then 2024-04-03T17:36:36.2602442Z  break 2024-04-03T17:36:36.2602817Z  fi 2024-04-03T17:36:36.2603130Z  echo "." 2024-04-03T17:36:36.2603437Z  sleep 5 2024-04-03T17:36:36.2603726Z done 2024-04-03T17:36:36.2611292Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T17:36:36.2611763Z env: 2024-04-03T17:36:36.2612017Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:36:36.2612609Z DOCKER_CONTAINER_ID: b0f53da97a9da663582739611fc2277e27fd12f5b71cdfe54025ecd3466cc388 2024-04-03T17:36:36.2613239Z ##[endgroup] 2024-04-03T17:36:36.2634737Z Holding runner for 2 hours until all ssh sessions have logged out 2024-04-03T17:36:36.2719741Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T17:36:36.2720502Z # ignore expansion of "docker ps -q" since it could be empty 2024-04-03T17:36:36.2721094Z # shellcheck disable=SC2046 2024-04-03T17:36:36.2721528Z docker stop $(docker ps -q) || true 2024-04-03T17:36:36.2721973Z # Prune all of the docker images 2024-04-03T17:36:36.2722396Z docker system prune -af 2024-04-03T17:36:36.2729902Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-03T17:36:36.2730382Z env: 2024-04-03T17:36:36.2730634Z GIT_DEFAULT_BRANCH: main 2024-04-03T17:36:36.2731213Z DOCKER_CONTAINER_ID: b0f53da97a9da663582739611fc2277e27fd12f5b71cdfe54025ecd3466cc388 2024-04-03T17:36:36.2731858Z ##[endgroup] 2024-04-03T17:36:46.8300905Z b0f53da97a9d 2024-04-03T17:36:47.3426884Z Deleted Containers: 2024-04-03T17:36:47.3427561Z b0f53da97a9da663582739611fc2277e27fd12f5b71cdfe54025ecd3466cc388 2024-04-03T17:36:47.3428009Z 2024-04-03T17:36:54.6978205Z Deleted Images: 2024-04-03T17:36:54.6979822Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11:16884143090f148aa56809ceba26eeb699b5dc49 2024-04-03T17:36:54.6981887Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-jammy-py3.8-gcc11@sha256:66934e8a2aa7cad8e06a1fbc5a6f0e239ee3d124c7929e4418f76d11beb80c7b 2024-04-03T17:36:54.6983338Z deleted: sha256:edab37a6fecd099844f4888cdb58ab57de8cc1ffb0964933664a10a6b9fa73a8 2024-04-03T17:36:54.6984186Z deleted: sha256:905eb3165d72389fe2a77b6b0b9b42979fec6d4762ba5d6372b004485a7887f2 2024-04-03T17:36:54.6985030Z deleted: sha256:b52147393ef7954d559c54e5e8bdf5c9362923c6bc5e5c1ac7b416ae3bc50f1b 2024-04-03T17:36:54.6985883Z deleted: sha256:33129cdc36b6b17d0d59f4913c861e2ab55cec6b1f0bcf729807fced51ee458e 2024-04-03T17:36:54.6986728Z deleted: sha256:905ee334605b68ef01e63105bb18f2ea2b4fc1b7cb7feabfb7a0eb65023cfbaa 2024-04-03T17:36:54.6987582Z deleted: sha256:538bde79919a681e153c980201fba870a6f7ceb5f0cf38c531bdb7d8808da5fe 2024-04-03T17:36:54.6988427Z deleted: sha256:db649c126d92a59d7f3777fa9cda59a19fc09346e3f2da46637f769b001da08d 2024-04-03T17:36:54.6989258Z deleted: sha256:401aa540d7013a2fec18292b710a4b7b704ff0e5191a6409bfa7f5105c03bf0f 2024-04-03T17:36:54.6990080Z deleted: sha256:3b53de43223804a7efad8f916999035e775d90036803f86521e9be97539f40ad 2024-04-03T17:36:54.6991017Z deleted: sha256:b2f44ae198a8e733062bc965e9f8f0b237e493f74113f343e75e9f11c1ffd82d 2024-04-03T17:36:54.6991867Z deleted: sha256:512d4e1ee86372c20c627cfcbd49474ff2417e1cbe160861e19b0a6b612858b7 2024-04-03T17:36:54.6992716Z deleted: sha256:40f2d30dfb73d70ffeaccc94d76abf117254af30c7ae8d05b869cd1e02072c56 2024-04-03T17:36:54.6993580Z deleted: sha256:30cbd3d349b325ad324c7222b09f06461d7d5ba9c2f0d2d47902f95e4eb735e9 2024-04-03T17:36:54.6994407Z deleted: sha256:84e3a88d823a68472d445767c8677811a579b76d421ddc260d06de3a3d20254a 2024-04-03T17:36:54.6995223Z deleted: sha256:74be4e310354a4d7f3b6f81aa1701ef3f988e96e7c8ce0d1accc035bcebb7903 2024-04-03T17:36:54.6996251Z deleted: sha256:6a4cb11588f457a23f797cb6c3c63ff04ca813557f1a5199a4b56bb96f5a400f 2024-04-03T17:36:54.6997088Z deleted: sha256:154b4926ace46a1bc9dd4859a007e72e05c53455362fa4d1b9c6a7a88658e2ad 2024-04-03T17:36:54.6997924Z deleted: sha256:83cca981aeb36a816e3f7961b4db68f6d1083a1a485359e27002dd05551a0faf 2024-04-03T17:36:54.6998759Z deleted: sha256:bd99ecf15c76d11726541058b9988b00b0448f5572e29581f8fe4d1beb6f2f04 2024-04-03T17:36:54.6999594Z deleted: sha256:c2306f98283138f1e8a0eb86821c56a874a3a0e64f3c5bd78af77e3d4dc4cea2 2024-04-03T17:36:54.7000436Z deleted: sha256:8d3cbc73dd42c894ad477153964abbd1db5ff1a85639c576f3dd15aa49fc1e32 2024-04-03T17:36:54.7001274Z deleted: sha256:93bca5d18fc95e7b538f7b011f199ea47d4c4a24d92e37befdb55c65315daef3 2024-04-03T17:36:54.7002129Z deleted: sha256:ca03cea0e25997d85fc876c81c3ec9a2d5280b5b9efa04b3f5dde9482894735b 2024-04-03T17:36:54.7003070Z deleted: sha256:2868da78fde9fe36f01a3bdf87ff3af36ec30cf36787d0a5cbad1573f9eb72a7 2024-04-03T17:36:54.7003907Z deleted: sha256:861a2e61cf7edf4a2d567f8f91a0a1d6720b2804696e455566653519def5eed6 2024-04-03T17:36:54.7004756Z deleted: sha256:ae20e25a200dfe3331cb1623aaa4d94fda1da70ca1457bece6ee0736406c477b 2024-04-03T17:36:54.7005596Z deleted: sha256:b2c2049f6254580f321ddb8e344a6f30920295714920d542f50895964494bcaa 2024-04-03T17:36:54.7006407Z deleted: sha256:6528c514c7f21f80242cf6a7bdc1c8d915649cfafdb940bc43a730b92c055395 2024-04-03T17:36:54.7007241Z deleted: sha256:661270485835c1ad00ddc00a41d55afcdf2f15326873b6564685568e51502f0b 2024-04-03T17:36:54.7008070Z deleted: sha256:c620865dc9b9190a1d5020fa279930e88aea7cc0fba3fce9a23526d59fb91b8b 2024-04-03T17:36:54.7008930Z deleted: sha256:6bbb1aecdf7e92a8f542cf0d72e09fb3ebb5d3e54e026c45fc6005479c973812 2024-04-03T17:36:54.7009871Z deleted: sha256:c38992a57858cfe5482c0f6a222dd2ec1365febf3634116368e0224766b8eabf 2024-04-03T17:36:54.7010702Z deleted: sha256:a806e3a5b978f751b2d24fb75b0ec1531609aa5438e969b553e35eae9420291d 2024-04-03T17:36:54.7011534Z deleted: sha256:f977093d1313dfd0c96542aec48775e1c6153e6d4da0e44e4019a1f33bc7ccbc 2024-04-03T17:36:54.7012339Z deleted: sha256:d87f19303d93852e7dec32bda379838586b85592647ae5fb83e9145395922c52 2024-04-03T17:36:54.7013302Z deleted: sha256:30fab3365ed7c6036db2bd494146182269ab08e5f076a9f05cc782ae2a1be6f2 2024-04-03T17:36:54.7014166Z deleted: sha256:dc7d5459e2773dac5bd936b3e0f2712f59e2f2efb6fdd9fa58b1dad13e7ba709 2024-04-03T17:36:54.7015010Z deleted: sha256:3c9770b6c5083d2b47eb7cb18e40f73377ad8c85193c48d10dc53453942a58ca 2024-04-03T17:36:54.7015851Z deleted: sha256:6037d0815881f7ca8e604b2cd0ba9c02a7b51d1dfaf9b45f72a9690593f4522e 2024-04-03T17:36:54.7016696Z deleted: sha256:4c5ec0d9046384b56009a78bd08ea8c4d53566560bfc1a5f3cbbf29fb5d28e78 2024-04-03T17:36:54.7017530Z deleted: sha256:831fd71d6c6e601a049d95d96f3b86602c60014f66c4a8ba995274930ec6360d 2024-04-03T17:36:54.7018354Z deleted: sha256:7f853fb34a14d00c3cc35edb4c6c2d139853c00531d5c0e947aaf0af32fc8c6d 2024-04-03T17:36:54.7019204Z deleted: sha256:6216695cd3743e80cf171dc625b35bdd8c875eb38ccb1c74de9ecaf70ef12cf7 2024-04-03T17:36:54.7020048Z deleted: sha256:a3987ee214e96c8a4c394439202fdebc1742b5c9b7c49c795eb5b02873caf5c0 2024-04-03T17:36:54.7020884Z deleted: sha256:ec36b2dfaf0780fc77239a6442a03d3fe12344a62fd9f160706ea8fb31827c31 2024-04-03T17:36:54.7021741Z deleted: sha256:e74fdfc92a654609e8a0e58038f355ac20ca6e48f9740e1f8eeb4258745664e6 2024-04-03T17:36:54.7022605Z deleted: sha256:213a5ea0dfcd1c6d1a1b663bfb48666e21fa4ccfb4e50814fe80178c59877bf8 2024-04-03T17:36:54.7023551Z deleted: sha256:2eeecd229060f5607471a8b1ae0eb47c1ef360d0bde7b0b918fe58a471d07632 2024-04-03T17:36:54.7024399Z deleted: sha256:e06952350caa9d97c27e0cf877ec331c95ccd08f9d101338b3f2430048435795 2024-04-03T17:36:54.7025240Z deleted: sha256:3f092d21eb8319fde82a9b1457866cfe4b510bf67241420ca209e8bf7102059e 2024-04-03T17:36:54.7026079Z deleted: sha256:2e04267fe055da334722fb21dbc887d8f07a04fdf09a51938de789a08b291cfe 2024-04-03T17:36:54.7026912Z deleted: sha256:a8917fb72475f126ca60bfa82d4cf4f13f68faf76b35caca0a08278683a28bf8 2024-04-03T17:36:54.7027768Z deleted: sha256:90cc1b8908359659cd6d3452ac6ebe7a6c136fea9a08dbce8895e7dbdb6207a5 2024-04-03T17:36:54.7028641Z deleted: sha256:0fff0f3eae6cbddf08197e2d51fdba3a40ec153315bb94e5bf8cfd4b46f98419 2024-04-03T17:36:54.7029500Z deleted: sha256:24ab507a199fbc79871ceb0443245e24c0458d10faade9c24593b0623f78cb03 2024-04-03T17:36:54.7030324Z deleted: sha256:cd88b61c700f6c253b46448dff97698516945970d56776e20e894d80d190b904 2024-04-03T17:36:54.7031166Z deleted: sha256:8d4170f13f8a43f7f191b1c5a5f5b1bdd6530ddd457a80a7b69de301801550e9 2024-04-03T17:36:54.7032007Z deleted: sha256:eeb6d5b1e91efc779ec4a6422b4d87e49967833e726c2108c9fdfff3a9febf43 2024-04-03T17:36:54.7032852Z deleted: sha256:c93041a9e83bfb126e457b0603544f90579dda9c6a13a915849645eb85a68502 2024-04-03T17:36:54.7033684Z deleted: sha256:2fb3495a30e7eb0864d26a8a93d8c78b86b8a64cd41619a1361a08b799d11685 2024-04-03T17:36:54.7034551Z deleted: sha256:b66ee3c97bcad9d55cb39a67ccd6b3beb626b4187bedceff060d6df941cc837d 2024-04-03T17:36:54.7035416Z deleted: sha256:392ced3ad906438bdead3c86295eb143f2fb85264af147e6f1f35d01c1ced362 2024-04-03T17:36:54.7036361Z deleted: sha256:87c81bac51502ae15d854b5a0e091a15ee10bc2b6d418a5705ca4a3abfcd0d28 2024-04-03T17:36:54.7037220Z deleted: sha256:d4f429e424bdec09e61d24c39f65c9c3933c4dfeb4b6310f4987a1799657ad77 2024-04-03T17:36:54.7038054Z deleted: sha256:f5efbd022127e1175e36f2c5fd792ac1f7c47418c93f79846df2412bdb7430bd 2024-04-03T17:36:54.7038891Z deleted: sha256:e2fad19928d02016347bc999f793c1912a2a4c979511f4e87b49db49233eed88 2024-04-03T17:36:54.7039719Z deleted: sha256:ee0f573586ebb0580982f974c3101c73d10d53c9e5523cb283bb5fd9cfb45017 2024-04-03T17:36:54.7040544Z deleted: sha256:aec57c0a36a59d850a778188d86004b271d7ea7c7f3099b69cd0964c23df31f3 2024-04-03T17:36:54.7041393Z deleted: sha256:97e7daacc3e32cc73f0a634ac185f46b236145f7adc160db674efdf892c1766f 2024-04-03T17:36:54.7042244Z deleted: sha256:e5ff4155a5c530732299da68cf28a5a52028fa5a25899de16a686b43d6c04e63 2024-04-03T17:36:54.7043166Z deleted: sha256:6145ed59ce48c64268ca5c98a5fbe2fbff17157e38af09e12bff68b74c79e74c 2024-04-03T17:36:54.7044010Z deleted: sha256:8ecde50858cbc91165551bcd309f8cd00e49b8ae1f7f864070b5551fc199ec46 2024-04-03T17:36:54.7044866Z deleted: sha256:c0c28dfb686a210566daab4fafef59cb4720c8036d3011179056cd059e034bce 2024-04-03T17:36:54.7045809Z deleted: sha256:d5d2024b69813e69f912057b0af512c77952d0e780a84868328829a7c3b1a5b7 2024-04-03T17:36:54.7046626Z deleted: sha256:1d15b0497ee6d16bddf0294f7a72aea8a230ef252d8a736c3084de3bd448f007 2024-04-03T17:36:54.7047475Z deleted: sha256:6f9321552d3cd0357443910af819fc1080aae134db4db2cebc9fbe8950f5970e 2024-04-03T17:36:54.7048434Z deleted: sha256:10406cf5503ff7215e2fa6a948121f26252776c8a77d59c03ea960d181953e33 2024-04-03T17:36:54.7049564Z deleted: sha256:edacda4f3bed0017ee8e30644e3ca826770e4fa9594617f67f67d8b04566c4d6 2024-04-03T17:36:54.7050404Z deleted: sha256:2734737e3cd948c91fd4003c5ee35820d4115a112c0478e0f031365431c1a2f6 2024-04-03T17:36:54.7051225Z deleted: sha256:eb1b20984d5deb9d3d1743e16a10683d787ce12f6b88133174ec4999344f362c 2024-04-03T17:36:54.7052061Z deleted: sha256:549d11624b0cbbb68984f5d6a33498bfb076fc901e4f23ea53b32971c7aaed02 2024-04-03T17:36:54.7052878Z deleted: sha256:26f2869e3ab075b907969de9aa179979f4fc9f8cc864025a646497ed71e2f0bc 2024-04-03T17:36:54.7053719Z deleted: sha256:222a48fb23cb13e5340360dee15485a1e547c730e8b0f79be58b38b443f96a32 2024-04-03T17:36:54.7054561Z deleted: sha256:81ef77bb993ba4fb3978141df789aba6d6754bb86c51c859d717367d9c37a65f 2024-04-03T17:36:54.7055502Z deleted: sha256:7bb82f4b0c6491c4b4cafaadd5377931be37d828ac2899475ac8abe9d0e520af 2024-04-03T17:36:54.7056348Z deleted: sha256:5498e8c22f6996f25ef193ee58617d5b37e2a96decf22e72de13c3b34e147591 2024-04-03T17:36:54.7056847Z 2024-04-03T17:36:54.7057006Z Total reclaimed space: 14.3GB 2024-04-03T17:36:54.7152328Z Post job cleanup. 2024-04-03T17:36:54.7227439Z Post job cleanup. 2024-04-03T17:36:54.8137485Z [command]/usr/bin/git version 2024-04-03T17:36:54.8440261Z git version 2.40.1 2024-04-03T17:36:54.8479368Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/2e433f93-a347-465b-9a68-a6ad1a3f3191' before making global git config changes 2024-04-03T17:36:54.8480695Z Adding repository directory to the temporary git global config as a safe directory 2024-04-03T17:36:54.8484771Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-04-03T17:36:54.8539607Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-04-03T17:36:54.8572459Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-04-03T17:36:54.8971335Z Entering 'android/libs/fbjni' 2024-04-03T17:36:54.9020977Z Entering 'third_party/FP16' 2024-04-03T17:36:54.9069660Z Entering 'third_party/FXdiv' 2024-04-03T17:36:54.9117156Z Entering 'third_party/NNPACK' 2024-04-03T17:36:54.9166803Z Entering 'third_party/QNNPACK' 2024-04-03T17:36:54.9216903Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T17:36:54.9259548Z Entering 'third_party/XNNPACK' 2024-04-03T17:36:54.9408139Z Entering 'third_party/benchmark' 2024-04-03T17:36:54.9459947Z Entering 'third_party/cpuinfo' 2024-04-03T17:36:54.9511329Z Entering 'third_party/cub' 2024-04-03T17:36:54.9560705Z Entering 'third_party/cudnn_frontend' 2024-04-03T17:36:54.9602795Z Entering 'third_party/cutlass' 2024-04-03T17:36:54.9654159Z Entering 'third_party/eigen' 2024-04-03T17:36:54.9722441Z Entering 'third_party/fbgemm' 2024-04-03T17:36:54.9792392Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T17:36:54.9853492Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T17:36:54.9913400Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T17:36:55.0031602Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T17:36:55.0095552Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T17:36:55.0146269Z Entering 'third_party/flatbuffers' 2024-04-03T17:36:55.0192472Z Entering 'third_party/fmt' 2024-04-03T17:36:55.0246939Z Entering 'third_party/foxi' 2024-04-03T17:36:55.0292531Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T17:36:55.0339488Z Entering 'third_party/gloo' 2024-04-03T17:36:55.0388774Z Entering 'third_party/googletest' 2024-04-03T17:36:55.0439368Z Entering 'third_party/ideep' 2024-04-03T17:36:55.0502136Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T17:36:55.0591994Z Entering 'third_party/ios-cmake' 2024-04-03T17:36:55.0640166Z Entering 'third_party/ittapi' 2024-04-03T17:36:55.0684007Z Entering 'third_party/kineto' 2024-04-03T17:36:55.0726331Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T17:36:55.0766714Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T17:36:55.0808456Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T17:36:55.0849644Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T17:36:55.0890539Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T17:36:55.0930294Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T17:36:55.0970808Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T17:36:55.1011657Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T17:36:55.1052292Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T17:36:55.1094874Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T17:36:55.1136671Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T17:36:55.1177671Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T17:36:55.1219796Z Entering 'third_party/mimalloc' 2024-04-03T17:36:55.1263238Z Entering 'third_party/nccl/nccl' 2024-04-03T17:36:55.1319337Z Entering 'third_party/neon2sse' 2024-04-03T17:36:55.1368005Z Entering 'third_party/nlohmann' 2024-04-03T17:36:55.1411168Z Entering 'third_party/onnx' 2024-04-03T17:36:55.1529763Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T17:36:55.1603781Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T17:36:55.1661854Z Entering 'third_party/onnx-tensorrt' 2024-04-03T17:36:55.1718243Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T17:36:55.1832238Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T17:36:55.1919875Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T17:36:55.1979971Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T17:36:55.2048681Z Entering 'third_party/pocketfft' 2024-04-03T17:36:55.2091467Z Entering 'third_party/protobuf' 2024-04-03T17:36:55.2177939Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T17:36:55.2240910Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T17:36:55.2295574Z Entering 'third_party/psimd' 2024-04-03T17:36:55.2343837Z Entering 'third_party/pthreadpool' 2024-04-03T17:36:55.2391282Z Entering 'third_party/pybind11' 2024-04-03T17:36:55.2439417Z Entering 'third_party/python-peachpy' 2024-04-03T17:36:55.2489550Z Entering 'third_party/sleef' 2024-04-03T17:36:55.2546485Z Entering 'third_party/tbb' 2024-04-03T17:36:55.2590573Z Entering 'third_party/tensorpipe' 2024-04-03T17:36:55.2668147Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T17:36:55.2733884Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T17:36:55.2805301Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T17:36:55.2881206Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T17:36:55.2966673Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T17:36:55.3025505Z Entering 'third_party/zstd' 2024-04-03T17:36:55.3091130Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-04-03T17:36:55.3118028Z http.https://github.com/.extraheader 2024-04-03T17:36:55.3127237Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-04-03T17:36:55.3159440Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-04-03T17:36:55.3422423Z Entering 'android/libs/fbjni' 2024-04-03T17:36:55.3449516Z http.https://github.com/.extraheader 2024-04-03T17:36:55.3477634Z Entering 'third_party/FP16' 2024-04-03T17:36:55.3504012Z http.https://github.com/.extraheader 2024-04-03T17:36:55.3531598Z Entering 'third_party/FXdiv' 2024-04-03T17:36:55.3558316Z http.https://github.com/.extraheader 2024-04-03T17:36:55.3585816Z Entering 'third_party/NNPACK' 2024-04-03T17:36:55.3612527Z http.https://github.com/.extraheader 2024-04-03T17:36:55.3640558Z Entering 'third_party/QNNPACK' 2024-04-03T17:36:55.3667383Z http.https://github.com/.extraheader 2024-04-03T17:36:55.3696205Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-03T17:36:55.3723273Z http.https://github.com/.extraheader 2024-04-03T17:36:55.3750857Z Entering 'third_party/XNNPACK' 2024-04-03T17:36:55.3777514Z http.https://github.com/.extraheader 2024-04-03T17:36:55.3822860Z Entering 'third_party/benchmark' 2024-04-03T17:36:55.3850460Z http.https://github.com/.extraheader 2024-04-03T17:36:55.3878234Z Entering 'third_party/cpuinfo' 2024-04-03T17:36:55.3905050Z http.https://github.com/.extraheader 2024-04-03T17:36:55.3932788Z Entering 'third_party/cub' 2024-04-03T17:36:55.3960229Z http.https://github.com/.extraheader 2024-04-03T17:36:55.3988422Z Entering 'third_party/cudnn_frontend' 2024-04-03T17:36:55.4015332Z http.https://github.com/.extraheader 2024-04-03T17:36:55.4043613Z Entering 'third_party/cutlass' 2024-04-03T17:36:55.4069899Z http.https://github.com/.extraheader 2024-04-03T17:36:55.4106074Z Entering 'third_party/eigen' 2024-04-03T17:36:55.4133303Z http.https://github.com/.extraheader 2024-04-03T17:36:55.4163826Z Entering 'third_party/fbgemm' 2024-04-03T17:36:55.4190269Z http.https://github.com/.extraheader 2024-04-03T17:36:55.4219786Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-03T17:36:55.4245373Z http.https://github.com/.extraheader 2024-04-03T17:36:55.4273314Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-03T17:36:55.4298970Z http.https://github.com/.extraheader 2024-04-03T17:36:55.4327848Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-03T17:36:55.4353570Z http.https://github.com/.extraheader 2024-04-03T17:36:55.4389369Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-03T17:36:55.4415183Z http.https://github.com/.extraheader 2024-04-03T17:36:55.4444310Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-03T17:36:55.4470045Z http.https://github.com/.extraheader 2024-04-03T17:36:55.4498490Z Entering 'third_party/flatbuffers' 2024-04-03T17:36:55.4526319Z http.https://github.com/.extraheader 2024-04-03T17:36:55.4557594Z Entering 'third_party/fmt' 2024-04-03T17:36:55.4583827Z http.https://github.com/.extraheader 2024-04-03T17:36:55.4611720Z Entering 'third_party/foxi' 2024-04-03T17:36:55.4638973Z http.https://github.com/.extraheader 2024-04-03T17:36:55.4666368Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-03T17:36:55.4693391Z http.https://github.com/.extraheader 2024-04-03T17:36:55.4721596Z Entering 'third_party/gloo' 2024-04-03T17:36:55.4748125Z http.https://github.com/.extraheader 2024-04-03T17:36:55.4775787Z Entering 'third_party/googletest' 2024-04-03T17:36:55.4805388Z http.https://github.com/.extraheader 2024-04-03T17:36:55.4832698Z Entering 'third_party/ideep' 2024-04-03T17:36:55.4860651Z http.https://github.com/.extraheader 2024-04-03T17:36:55.4886460Z Entering 'third_party/ideep/mkl-dnn' 2024-04-03T17:36:55.4912487Z http.https://github.com/.extraheader 2024-04-03T17:36:55.4946539Z Entering 'third_party/ios-cmake' 2024-04-03T17:36:55.4973713Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5000676Z Entering 'third_party/ittapi' 2024-04-03T17:36:55.5026817Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5054435Z Entering 'third_party/kineto' 2024-04-03T17:36:55.5081335Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5109105Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-03T17:36:55.5134773Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5162609Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-03T17:36:55.5187755Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5216524Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-03T17:36:55.5242758Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5270154Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-03T17:36:55.5296098Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5324108Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-03T17:36:55.5349153Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5376312Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-03T17:36:55.5401848Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5430750Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-03T17:36:55.5456150Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5482890Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-03T17:36:55.5507999Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5536033Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-03T17:36:55.5563592Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5592843Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-03T17:36:55.5618553Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5647483Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-03T17:36:55.5672604Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5700942Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-03T17:36:55.5726505Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5754786Z Entering 'third_party/mimalloc' 2024-04-03T17:36:55.5782582Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5809667Z Entering 'third_party/nccl/nccl' 2024-04-03T17:36:55.5836618Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5863771Z Entering 'third_party/neon2sse' 2024-04-03T17:36:55.5890550Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5916642Z Entering 'third_party/nlohmann' 2024-04-03T17:36:55.5943547Z http.https://github.com/.extraheader 2024-04-03T17:36:55.5971187Z Entering 'third_party/onnx' 2024-04-03T17:36:55.5999058Z http.https://github.com/.extraheader 2024-04-03T17:36:55.6041888Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-03T17:36:55.6067723Z http.https://github.com/.extraheader 2024-04-03T17:36:55.6094477Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-03T17:36:55.6121715Z http.https://github.com/.extraheader 2024-04-03T17:36:55.6151368Z Entering 'third_party/onnx-tensorrt' 2024-04-03T17:36:55.6177690Z http.https://github.com/.extraheader 2024-04-03T17:36:55.6204158Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-03T17:36:55.6230045Z http.https://github.com/.extraheader 2024-04-03T17:36:55.6261100Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-03T17:36:55.6288504Z http.https://github.com/.extraheader 2024-04-03T17:36:55.6316657Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-03T17:36:55.6341666Z http.https://github.com/.extraheader 2024-04-03T17:36:55.6368452Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-03T17:36:55.6393952Z http.https://github.com/.extraheader 2024-04-03T17:36:55.6426475Z Entering 'third_party/pocketfft' 2024-04-03T17:36:55.6454006Z http.https://github.com/.extraheader 2024-04-03T17:36:55.6481634Z Entering 'third_party/protobuf' 2024-04-03T17:36:55.6508321Z http.https://github.com/.extraheader 2024-04-03T17:36:55.6539461Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-03T17:36:55.6565442Z http.https://github.com/.extraheader 2024-04-03T17:36:55.6593314Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-03T17:36:55.6620099Z http.https://github.com/.extraheader 2024-04-03T17:36:55.6650686Z Entering 'third_party/psimd' 2024-04-03T17:36:55.6678069Z http.https://github.com/.extraheader 2024-04-03T17:36:55.6705967Z Entering 'third_party/pthreadpool' 2024-04-03T17:36:55.6733003Z http.https://github.com/.extraheader 2024-04-03T17:36:55.6761237Z Entering 'third_party/pybind11' 2024-04-03T17:36:55.6788661Z http.https://github.com/.extraheader 2024-04-03T17:36:55.6816349Z Entering 'third_party/python-peachpy' 2024-04-03T17:36:55.6844848Z http.https://github.com/.extraheader 2024-04-03T17:36:55.6873159Z Entering 'third_party/sleef' 2024-04-03T17:36:55.6901148Z http.https://github.com/.extraheader 2024-04-03T17:36:55.6929270Z Entering 'third_party/tbb' 2024-04-03T17:36:55.6956483Z http.https://github.com/.extraheader 2024-04-03T17:36:55.6985877Z Entering 'third_party/tensorpipe' 2024-04-03T17:36:55.7012928Z http.https://github.com/.extraheader 2024-04-03T17:36:55.7040776Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-03T17:36:55.7065954Z http.https://github.com/.extraheader 2024-04-03T17:36:55.7094165Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-03T17:36:55.7120368Z http.https://github.com/.extraheader 2024-04-03T17:36:55.7147672Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-03T17:36:55.7173474Z http.https://github.com/.extraheader 2024-04-03T17:36:55.7201680Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-03T17:36:55.7227365Z http.https://github.com/.extraheader 2024-04-03T17:36:55.7253921Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-03T17:36:55.7287045Z http.https://github.com/.extraheader 2024-04-03T17:36:55.7316890Z Entering 'third_party/zstd' 2024-04-03T17:36:55.7343244Z http.https://github.com/.extraheader 2024-04-03T17:36:55.7628888Z Cleaning up orphan processes