2023-11-14T04:44:50.4293251Z Requested labels: linux.8xlarge.nvidia.gpu 2023-11-14T04:44:50.4293398Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/112098/merge 2023-11-14T04:44:50.4293618Z Reusable workflow chain: 2023-11-14T04:44:50.4293698Z pytorch/pytorch/.github/workflows/pull.yml@refs/pull/112098/merge (35b9c00c6974fbcfd0eb53491c53c61a6c439831) 2023-11-14T04:44:50.4293797Z -> pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/112098/merge (35b9c00c6974fbcfd0eb53491c53c61a6c439831) 2023-11-14T04:44:50.4293899Z Waiting for a runner to pick up this job... 2023-11-14T04:44:50.6500381Z Job is about to start running on the runner: i-0bfddc9f582173679 (organization) 2023-11-14T04:44:56.3402658Z Current runner version: '2.311.0' 2023-11-14T04:44:56.3412317Z Runner name: 'i-0bfddc9f582173679' 2023-11-14T04:44:56.3413560Z Runner group name: 'Default' 2023-11-14T04:44:56.3414759Z Machine name: 'ip-10-0-49-138' 2023-11-14T04:44:56.3420871Z ##[group]GITHUB_TOKEN Permissions 2023-11-14T04:44:56.3423733Z Actions: write 2023-11-14T04:44:56.3424589Z Checks: write 2023-11-14T04:44:56.3425213Z Contents: write 2023-11-14T04:44:56.3425924Z Deployments: write 2023-11-14T04:44:56.3426691Z Discussions: write 2023-11-14T04:44:56.3427336Z Issues: write 2023-11-14T04:44:56.3428078Z Metadata: read 2023-11-14T04:44:56.3428815Z Packages: write 2023-11-14T04:44:56.3429445Z Pages: write 2023-11-14T04:44:56.3430153Z PullRequests: write 2023-11-14T04:44:56.3430977Z RepositoryProjects: write 2023-11-14T04:44:56.3431746Z SecurityEvents: write 2023-11-14T04:44:56.3432536Z Statuses: write 2023-11-14T04:44:56.3433272Z ##[endgroup] 2023-11-14T04:44:56.3438349Z Secret source: Actions 2023-11-14T04:44:56.3439730Z Prepare workflow directory 2023-11-14T04:44:56.7118550Z Prepare all required actions 2023-11-14T04:44:56.7385088Z Getting action download info 2023-11-14T04:44:56.9108928Z Download action repository 'pytorch/test-infra@main' (SHA:5fbbad20d6fc6779f80c83977e319b918387cbc0) 2023-11-14T04:44:57.2909642Z Download action repository 'pytorch/pytorch@main' (SHA:b3a76ccc12a0a3cb56fd32a8a14feccf0e2cb918) 2023-11-14T04:45:01.5467603Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2023-11-14T04:45:01.8622787Z Getting action download info 2023-11-14T04:45:01.9837371Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2023-11-14T04:45:02.2027730Z Getting action download info 2023-11-14T04:45:02.3139960Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2023-11-14T04:45:02.5151502Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/112098/merge (35b9c00c6974fbcfd0eb53491c53c61a6c439831) 2023-11-14T04:45:02.5155687Z ##[group] Inputs 2023-11-14T04:45:02.5156351Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2023-11-14T04:45:02.5158554Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2023-11-14T04:45:02.5161409Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:45:02.5162847Z sync-tag: 2023-11-14T04:45:02.5164244Z timeout-minutes: 360 2023-11-14T04:45:02.5164789Z use-gha: 2023-11-14T04:45:02.5165262Z dashboard-tag: 2023-11-14T04:45:02.5165735Z ##[endgroup] 2023-11-14T04:45:02.5167190Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2023-11-14T04:45:02.6657368Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2023-11-14T04:45:02.6658232Z with: 2023-11-14T04:45:02.6659173Z github-secret: *** 2023-11-14T04:45:02.6660452Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2023-11-14T04:45:02.6661825Z activate-with-label: false 2023-11-14T04:45:02.6662369Z label: with-ssh 2023-11-14T04:45:02.6662833Z remove-existing-keys: true 2023-11-14T04:45:02.6663360Z env: 2023-11-14T04:45:02.6663800Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:45:02.6664296Z ##[endgroup] 2023-11-14T04:45:02.8007942Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2023-11-14T04:45:03.2892618Z Grabbing public ssh keys from https://github.com/clee2000.keys 2023-11-14T04:45:03.4309691Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2023-11-14T04:45:03.4332581Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2023-11-14T04:45:03.4379136Z Login using: ssh ec2-user@ec2-18-234-146-91.compute-1.amazonaws.com 2023-11-14T04:45:03.4380151Z All testing is done inside the container, to start an interactive session run: 2023-11-14T04:45:03.4381205Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2023-11-14T04:45:03.4751443Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2023-11-14T04:45:03.4752207Z with: 2023-11-14T04:45:03.4752574Z submodules: recursive 2023-11-14T04:45:03.4752994Z fetch-depth: 0 2023-11-14T04:45:03.4753370Z env: 2023-11-14T04:45:03.4753699Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:45:03.4754147Z ##[endgroup] 2023-11-14T04:45:03.5069391Z ##[group]Run retry () { 2023-11-14T04:45:03.5069927Z retry () { 2023-11-14T04:45:03.5070551Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2023-11-14T04:45:03.5071301Z } 2023-11-14T04:45:03.5071706Z echo "${GITHUB_WORKSPACE}" 2023-11-14T04:45:03.5072225Z if [ -z "${NO_SUDO}" ]; then 2023-11-14T04:45:03.5072844Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2023-11-14T04:45:03.5073429Z else 2023-11-14T04:45:03.5073859Z  retry rm -rf "${GITHUB_WORKSPACE}" 2023-11-14T04:45:03.5074408Z fi 2023-11-14T04:45:03.5075200Z mkdir "${GITHUB_WORKSPACE}" 2023-11-14T04:45:03.5106133Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:45:03.5106782Z env: 2023-11-14T04:45:03.5107151Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:45:03.5107596Z NO_SUDO: 2023-11-14T04:45:03.5107931Z ##[endgroup] 2023-11-14T04:45:03.5244942Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2023-11-14T04:45:07.1273094Z ##[group]Run malfet/checkout@silent-checkout 2023-11-14T04:45:07.1273741Z with: 2023-11-14T04:45:07.1274171Z ref: fe705c0d795479e12d728ea79f708a42b978ffae 2023-11-14T04:45:07.1275308Z fetch-depth: 0 2023-11-14T04:45:07.1275724Z submodules: recursive 2023-11-14T04:45:07.1276165Z quiet-checkout: true 2023-11-14T04:45:07.1276594Z repository: pytorch/pytorch 2023-11-14T04:45:07.1277226Z token: *** 2023-11-14T04:45:07.1277610Z ssh-strict: true 2023-11-14T04:45:07.1278016Z persist-credentials: true 2023-11-14T04:45:07.1278481Z clean: true 2023-11-14T04:45:07.1278922Z sparse-checkout-cone-mode: true 2023-11-14T04:45:07.1279399Z lfs: false 2023-11-14T04:45:07.1279792Z set-safe-directory: true 2023-11-14T04:45:07.1280222Z env: 2023-11-14T04:45:07.1280551Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:45:07.1280995Z ##[endgroup] 2023-11-14T04:45:07.2628221Z Syncing repository: pytorch/pytorch 2023-11-14T04:45:07.2630407Z ##[group]Getting Git version info 2023-11-14T04:45:07.2631424Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2023-11-14T04:45:07.2632537Z [command]/usr/bin/git version 2023-11-14T04:45:07.2632933Z git version 2.40.1 2023-11-14T04:45:07.2634963Z ##[endgroup] 2023-11-14T04:45:07.2655850Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/23db6ada-eb61-47e2-b21b-446cb876344d' before making global git config changes 2023-11-14T04:45:07.2657469Z Adding repository directory to the temporary git global config as a safe directory 2023-11-14T04:45:07.2663308Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2023-11-14T04:45:07.2709660Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2023-11-14T04:45:07.2715893Z ##[group]Initializing the repository 2023-11-14T04:45:07.2720020Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2023-11-14T04:45:07.2753558Z hint: Using 'master' as the name for the initial branch. This default branch name 2023-11-14T04:45:07.2754972Z hint: is subject to change. To configure the initial branch name to use in all 2023-11-14T04:45:07.2756020Z hint: of your new repositories, which will suppress this warning, call: 2023-11-14T04:45:07.2757166Z hint: 2023-11-14T04:45:07.2757970Z hint: git config --global init.defaultBranch 2023-11-14T04:45:07.2758556Z hint: 2023-11-14T04:45:07.2759195Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2023-11-14T04:45:07.2760256Z hint: 'development'. The just-created branch can be renamed via this command: 2023-11-14T04:45:07.2761211Z hint: 2023-11-14T04:45:07.2761664Z hint: git branch -m 2023-11-14T04:45:07.2762754Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2023-11-14T04:45:07.2767528Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2023-11-14T04:45:07.2803869Z ##[endgroup] 2023-11-14T04:45:07.2804475Z ##[group]Disabling automatic garbage collection 2023-11-14T04:45:07.2808181Z [command]/usr/bin/git config --local gc.auto 0 2023-11-14T04:45:07.2842499Z ##[endgroup] 2023-11-14T04:45:07.2843031Z ##[group]Setting up auth 2023-11-14T04:45:07.2852243Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2023-11-14T04:45:07.2888865Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2023-11-14T04:45:07.3197335Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2023-11-14T04:45:07.3226617Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2023-11-14T04:45:07.3529714Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2023-11-14T04:45:07.3577602Z ##[endgroup] 2023-11-14T04:45:07.3578299Z ##[group]Fetching the repository 2023-11-14T04:45:07.3586413Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2023-11-14T04:45:15.9215558Z remote: Enumerating objects: 1197588 2023-11-14T04:45:15.9216261Z remote: Enumerating objects: 1200035, done. 2023-11-14T04:45:15.9217573Z remote: Counting objects: 0% (1/2447) 2023-11-14T04:45:15.9218224Z remote: Counting objects: 1% (25/2447) 2023-11-14T04:45:15.9219164Z remote: Counting objects: 2% (49/2447) 2023-11-14T04:45:15.9221426Z remote: Counting objects: 3% (74/2447) 2023-11-14T04:45:15.9224235Z remote: Counting objects: 4% (98/2447) 2023-11-14T04:45:15.9224883Z remote: Counting objects: 5% (123/2447) 2023-11-14T04:45:15.9225370Z remote: Counting objects: 6% (147/2447) 2023-11-14T04:45:15.9226011Z remote: Counting objects: 7% (172/2447) 2023-11-14T04:45:15.9226626Z remote: Counting objects: 8% (196/2447) 2023-11-14T04:45:15.9229397Z remote: Counting objects: 9% (221/2447) 2023-11-14T04:45:15.9230116Z remote: Counting objects: 10% (245/2447) 2023-11-14T04:45:15.9230750Z remote: Counting objects: 11% (270/2447) 2023-11-14T04:45:15.9231352Z remote: Counting objects: 12% (294/2447) 2023-11-14T04:45:15.9231992Z remote: Counting objects: 13% (319/2447) 2023-11-14T04:45:15.9232600Z remote: Counting objects: 14% (343/2447) 2023-11-14T04:45:15.9233236Z remote: Counting objects: 15% (368/2447) 2023-11-14T04:45:15.9233874Z remote: Counting objects: 16% (392/2447) 2023-11-14T04:45:15.9234559Z remote: Counting objects: 17% (416/2447) 2023-11-14T04:45:15.9235441Z remote: Counting objects: 18% (441/2447) 2023-11-14T04:45:15.9236094Z remote: Counting objects: 19% (465/2447) 2023-11-14T04:45:15.9236679Z remote: Counting objects: 20% (490/2447) 2023-11-14T04:45:15.9237316Z remote: Counting objects: 21% (514/2447) 2023-11-14T04:45:15.9237980Z remote: Counting objects: 22% (539/2447) 2023-11-14T04:45:15.9238820Z remote: Counting objects: 23% (563/2447) 2023-11-14T04:45:15.9239518Z remote: Counting objects: 24% (588/2447) 2023-11-14T04:45:15.9240189Z remote: Counting objects: 25% (612/2447) 2023-11-14T04:45:15.9240871Z remote: Counting objects: 26% (637/2447) 2023-11-14T04:45:15.9241502Z remote: Counting objects: 27% (661/2447) 2023-11-14T04:45:15.9242292Z remote: Counting objects: 28% (686/2447) 2023-11-14T04:45:15.9242854Z remote: Counting objects: 29% (710/2447) 2023-11-14T04:45:15.9243525Z remote: Counting objects: 30% (735/2447) 2023-11-14T04:45:15.9244053Z remote: Counting objects: 31% (759/2447) 2023-11-14T04:45:15.9244855Z remote: Counting objects: 32% (784/2447) 2023-11-14T04:45:15.9245477Z remote: Counting objects: 33% (808/2447) 2023-11-14T04:45:15.9245953Z remote: Counting objects: 34% (832/2447) 2023-11-14T04:45:15.9246588Z remote: Counting objects: 35% (857/2447) 2023-11-14T04:45:15.9247252Z remote: Counting objects: 36% (881/2447) 2023-11-14T04:45:15.9247993Z remote: Counting objects: 37% (906/2447) 2023-11-14T04:45:15.9248595Z remote: Counting objects: 38% (930/2447) 2023-11-14T04:45:15.9249278Z remote: Counting objects: 39% (955/2447) 2023-11-14T04:45:15.9249910Z remote: Counting objects: 40% (979/2447) 2023-11-14T04:45:15.9250376Z remote: Counting objects: 41% (1004/2447) 2023-11-14T04:45:15.9251220Z remote: Counting objects: 42% (1028/2447) 2023-11-14T04:45:15.9251885Z remote: Counting objects: 43% (1053/2447) 2023-11-14T04:45:15.9252441Z remote: Counting objects: 44% (1077/2447) 2023-11-14T04:45:15.9253118Z remote: Counting objects: 45% (1102/2447) 2023-11-14T04:45:15.9253676Z remote: Counting objects: 46% (1126/2447) 2023-11-14T04:45:15.9254388Z remote: Counting objects: 47% (1151/2447) 2023-11-14T04:45:15.9255041Z remote: Counting objects: 48% (1175/2447) 2023-11-14T04:45:15.9255747Z remote: Counting objects: 49% (1200/2447) 2023-11-14T04:45:15.9256328Z remote: Counting objects: 50% (1224/2447) 2023-11-14T04:45:15.9257008Z remote: Counting objects: 51% (1248/2447) 2023-11-14T04:45:15.9257605Z remote: Counting objects: 52% (1273/2447) 2023-11-14T04:45:15.9258287Z remote: Counting objects: 53% (1297/2447) 2023-11-14T04:45:15.9259045Z remote: Counting objects: 54% (1322/2447) 2023-11-14T04:45:15.9259614Z remote: Counting objects: 55% (1346/2447) 2023-11-14T04:45:15.9260287Z remote: Counting objects: 56% (1371/2447) 2023-11-14T04:45:15.9260931Z remote: Counting objects: 57% (1395/2447) 2023-11-14T04:45:15.9261596Z remote: Counting objects: 58% (1420/2447) 2023-11-14T04:45:15.9262240Z remote: Counting objects: 59% (1444/2447) 2023-11-14T04:45:15.9262787Z remote: Counting objects: 60% (1469/2447) 2023-11-14T04:45:15.9263431Z remote: Counting objects: 61% (1493/2447) 2023-11-14T04:45:15.9264080Z remote: Counting objects: 62% (1518/2447) 2023-11-14T04:45:15.9264753Z remote: Counting objects: 63% (1542/2447) 2023-11-14T04:45:15.9265414Z remote: Counting objects: 64% (1567/2447) 2023-11-14T04:45:15.9266070Z remote: Counting objects: 65% (1591/2447) 2023-11-14T04:45:15.9266736Z remote: Counting objects: 66% (1616/2447) 2023-11-14T04:45:15.9267369Z remote: Counting objects: 67% (1640/2447) 2023-11-14T04:45:15.9268024Z remote: Counting objects: 68% (1664/2447) 2023-11-14T04:45:15.9268683Z remote: Counting objects: 69% (1689/2447) 2023-11-14T04:45:15.9269328Z remote: Counting objects: 70% (1713/2447) 2023-11-14T04:45:15.9269953Z remote: Counting objects: 71% (1738/2447) 2023-11-14T04:45:15.9270611Z remote: Counting objects: 72% (1762/2447) 2023-11-14T04:45:15.9271261Z remote: Counting objects: 73% (1787/2447) 2023-11-14T04:45:15.9272022Z remote: Counting objects: 74% (1811/2447) 2023-11-14T04:45:15.9272675Z remote: Counting objects: 75% (1836/2447) 2023-11-14T04:45:15.9273333Z remote: Counting objects: 76% (1860/2447) 2023-11-14T04:45:15.9273968Z remote: Counting objects: 77% (1885/2447) 2023-11-14T04:45:15.9274814Z remote: Counting objects: 78% (1909/2447) 2023-11-14T04:45:15.9275620Z remote: Counting objects: 79% (1934/2447) 2023-11-14T04:45:15.9276273Z remote: Counting objects: 80% (1958/2447) 2023-11-14T04:45:15.9276926Z remote: Counting objects: 81% (1983/2447) 2023-11-14T04:45:15.9277577Z remote: Counting objects: 82% (2007/2447) 2023-11-14T04:45:15.9278205Z remote: Counting objects: 83% (2032/2447) 2023-11-14T04:45:15.9278857Z remote: Counting objects: 84% (2056/2447) 2023-11-14T04:45:15.9279508Z remote: Counting objects: 85% (2080/2447) 2023-11-14T04:45:15.9280140Z remote: Counting objects: 86% (2105/2447) 2023-11-14T04:45:15.9280811Z remote: Counting objects: 87% (2129/2447) 2023-11-14T04:45:15.9281461Z remote: Counting objects: 88% (2154/2447) 2023-11-14T04:45:15.9282119Z remote: Counting objects: 89% (2178/2447) 2023-11-14T04:45:15.9282749Z remote: Counting objects: 90% (2203/2447) 2023-11-14T04:45:15.9283405Z remote: Counting objects: 91% (2227/2447) 2023-11-14T04:45:15.9284070Z remote: Counting objects: 92% (2252/2447) 2023-11-14T04:45:15.9284704Z remote: Counting objects: 93% (2276/2447) 2023-11-14T04:45:15.9285387Z remote: Counting objects: 94% (2301/2447) 2023-11-14T04:45:15.9286008Z remote: Counting objects: 95% (2325/2447) 2023-11-14T04:45:15.9286635Z remote: Counting objects: 96% (2350/2447) 2023-11-14T04:45:15.9287289Z remote: Counting objects: 97% (2374/2447) 2023-11-14T04:45:15.9287941Z remote: Counting objects: 98% (2399/2447) 2023-11-14T04:45:15.9288569Z remote: Counting objects: 99% (2423/2447) 2023-11-14T04:45:15.9289235Z remote: Counting objects: 100% (2447/2447) 2023-11-14T04:45:15.9289937Z remote: Counting objects: 100% (2447/2447), done. 2023-11-14T04:45:15.9358827Z remote: Compressing objects: 0% (1/1266) 2023-11-14T04:45:15.9759516Z remote: Compressing objects: 1% (13/1266) 2023-11-14T04:45:16.0183557Z remote: Compressing objects: 2% (26/1266) 2023-11-14T04:45:16.1969136Z remote: Compressing objects: 3% (38/1266) 2023-11-14T04:45:16.3161141Z remote: Compressing objects: 4% (51/1266) 2023-11-14T04:45:16.4271010Z remote: Compressing objects: 5% (64/1266) 2023-11-14T04:45:16.5118860Z remote: Compressing objects: 6% (76/1266) 2023-11-14T04:45:16.5874732Z remote: Compressing objects: 7% (89/1266) 2023-11-14T04:45:16.6478217Z remote: Compressing objects: 8% (102/1266) 2023-11-14T04:45:16.7129474Z remote: Compressing objects: 9% (114/1266) 2023-11-14T04:45:16.7540191Z remote: Compressing objects: 10% (127/1266) 2023-11-14T04:45:16.7877029Z remote: Compressing objects: 11% (140/1266) 2023-11-14T04:45:16.8233420Z remote: Compressing objects: 12% (152/1266) 2023-11-14T04:45:16.8421573Z remote: Compressing objects: 13% (165/1266) 2023-11-14T04:45:16.8598462Z remote: Compressing objects: 14% (178/1266) 2023-11-14T04:45:16.8801138Z remote: Compressing objects: 15% (190/1266) 2023-11-14T04:45:16.8977376Z remote: Compressing objects: 16% (203/1266) 2023-11-14T04:45:16.9097751Z remote: Compressing objects: 17% (216/1266) 2023-11-14T04:45:16.9240620Z remote: Compressing objects: 18% (228/1266) 2023-11-14T04:45:16.9269571Z remote: Compressing objects: 19% (241/1266) 2023-11-14T04:45:16.9344477Z remote: Compressing objects: 19% (245/1266) 2023-11-14T04:45:16.9443831Z remote: Compressing objects: 20% (254/1266) 2023-11-14T04:45:16.9513333Z remote: Compressing objects: 21% (266/1266) 2023-11-14T04:45:16.9560048Z remote: Compressing objects: 22% (279/1266) 2023-11-14T04:45:16.9608402Z remote: Compressing objects: 23% (292/1266) 2023-11-14T04:45:16.9667681Z remote: Compressing objects: 24% (304/1266) 2023-11-14T04:45:16.9690701Z remote: Compressing objects: 25% (317/1266) 2023-11-14T04:45:16.9708125Z remote: Compressing objects: 26% (330/1266) 2023-11-14T04:45:16.9718025Z remote: Compressing objects: 27% (342/1266) 2023-11-14T04:45:16.9721321Z remote: Compressing objects: 28% (355/1266) 2023-11-14T04:45:16.9736661Z remote: Compressing objects: 29% (368/1266) 2023-11-14T04:45:16.9746442Z remote: Compressing objects: 30% (380/1266) 2023-11-14T04:45:16.9759290Z remote: Compressing objects: 31% (393/1266) 2023-11-14T04:45:16.9763067Z remote: Compressing objects: 32% (406/1266) 2023-11-14T04:45:16.9774324Z remote: Compressing objects: 33% (418/1266) 2023-11-14T04:45:16.9788137Z remote: Compressing objects: 34% (431/1266) 2023-11-14T04:45:16.9816295Z remote: Compressing objects: 35% (444/1266) 2023-11-14T04:45:16.9830500Z remote: Compressing objects: 36% (456/1266) 2023-11-14T04:45:16.9844181Z remote: Compressing objects: 37% (469/1266) 2023-11-14T04:45:16.9863385Z remote: Compressing objects: 38% (482/1266) 2023-11-14T04:45:16.9870389Z remote: Compressing objects: 39% (494/1266) 2023-11-14T04:45:16.9875609Z remote: Compressing objects: 40% (507/1266) 2023-11-14T04:45:16.9889876Z remote: Compressing objects: 41% (520/1266) 2023-11-14T04:45:16.9903186Z remote: Compressing objects: 42% (532/1266) 2023-11-14T04:45:16.9917508Z remote: Compressing objects: 43% (545/1266) 2023-11-14T04:45:16.9927918Z remote: Compressing objects: 44% (558/1266) 2023-11-14T04:45:16.9934915Z remote: Compressing objects: 45% (570/1266) 2023-11-14T04:45:16.9943371Z remote: Compressing objects: 46% (583/1266) 2023-11-14T04:45:16.9950912Z remote: Compressing objects: 47% (596/1266) 2023-11-14T04:45:16.9962455Z remote: Compressing objects: 48% (608/1266) 2023-11-14T04:45:16.9967831Z remote: Compressing objects: 49% (621/1266) 2023-11-14T04:45:16.9971316Z remote: Compressing objects: 50% (633/1266) 2023-11-14T04:45:16.9978611Z remote: Compressing objects: 51% (646/1266) 2023-11-14T04:45:16.9980087Z remote: Compressing objects: 52% (659/1266) 2023-11-14T04:45:16.9984615Z remote: Compressing objects: 53% (671/1266) 2023-11-14T04:45:16.9987198Z remote: Compressing objects: 54% (684/1266) 2023-11-14T04:45:16.9990899Z remote: Compressing objects: 55% (697/1266) 2023-11-14T04:45:16.9995002Z remote: Compressing objects: 56% (709/1266) 2023-11-14T04:45:16.9998445Z remote: Compressing objects: 57% (722/1266) 2023-11-14T04:45:17.0000793Z remote: Compressing objects: 58% (735/1266) 2023-11-14T04:45:17.0003101Z remote: Compressing objects: 59% (747/1266) 2023-11-14T04:45:17.0006436Z remote: Compressing objects: 60% (760/1266) 2023-11-14T04:45:17.0007193Z remote: Compressing objects: 61% (773/1266) 2023-11-14T04:45:17.0007898Z remote: Compressing objects: 62% (785/1266) 2023-11-14T04:45:17.0008557Z remote: Compressing objects: 63% (798/1266) 2023-11-14T04:45:17.0009202Z remote: Compressing objects: 64% (811/1266) 2023-11-14T04:45:17.0010133Z remote: Compressing objects: 65% (823/1266) 2023-11-14T04:45:17.0010886Z remote: Compressing objects: 66% (836/1266) 2023-11-14T04:45:17.0012055Z remote: Compressing objects: 67% (849/1266) 2023-11-14T04:45:17.0027193Z remote: Compressing objects: 68% (861/1266) 2023-11-14T04:45:17.0040092Z remote: Compressing objects: 69% (874/1266) 2023-11-14T04:45:17.0049775Z remote: Compressing objects: 70% (887/1266) 2023-11-14T04:45:17.0055377Z remote: Compressing objects: 71% (899/1266) 2023-11-14T04:45:17.0061101Z remote: Compressing objects: 72% (912/1266) 2023-11-14T04:45:17.0066096Z remote: Compressing objects: 73% (925/1266) 2023-11-14T04:45:17.0069284Z remote: Compressing objects: 74% (937/1266) 2023-11-14T04:45:17.0077742Z remote: Compressing objects: 75% (950/1266) 2023-11-14T04:45:17.0084777Z remote: Compressing objects: 76% (963/1266) 2023-11-14T04:45:17.0091215Z remote: Compressing objects: 77% (975/1266) 2023-11-14T04:45:17.0094138Z remote: Compressing objects: 78% (988/1266) 2023-11-14T04:45:17.0095580Z remote: Compressing objects: 79% (1001/1266) 2023-11-14T04:45:17.0101857Z remote: Compressing objects: 80% (1013/1266) 2023-11-14T04:45:17.0106944Z remote: Compressing objects: 81% (1026/1266) 2023-11-14T04:45:17.0112137Z remote: Compressing objects: 82% (1039/1266) 2023-11-14T04:45:17.0114015Z remote: Compressing objects: 83% (1051/1266) 2023-11-14T04:45:17.0118290Z remote: Compressing objects: 84% (1064/1266) 2023-11-14T04:45:17.0122900Z remote: Compressing objects: 85% (1077/1266) 2023-11-14T04:45:17.0124874Z remote: Compressing objects: 86% (1089/1266) 2023-11-14T04:45:17.0127114Z remote: Compressing objects: 87% (1102/1266) 2023-11-14T04:45:17.0129867Z remote: Compressing objects: 88% (1115/1266) 2023-11-14T04:45:17.0141424Z remote: Compressing objects: 89% (1127/1266) 2023-11-14T04:45:17.0142456Z remote: Compressing objects: 90% (1140/1266) 2023-11-14T04:45:17.0143093Z remote: Compressing objects: 91% (1153/1266) 2023-11-14T04:45:17.0143789Z remote: Compressing objects: 92% (1165/1266) 2023-11-14T04:45:17.0144482Z remote: Compressing objects: 93% (1178/1266) 2023-11-14T04:45:17.0145444Z remote: Compressing objects: 94% (1191/1266) 2023-11-14T04:45:17.0148956Z remote: Compressing objects: 95% (1203/1266) 2023-11-14T04:45:17.0150067Z remote: Compressing objects: 96% (1216/1266) 2023-11-14T04:45:17.0152598Z remote: Compressing objects: 97% (1229/1266) 2023-11-14T04:45:17.0154655Z remote: Compressing objects: 98% (1241/1266) 2023-11-14T04:45:17.0156642Z remote: Compressing objects: 99% (1254/1266) 2023-11-14T04:45:17.0157342Z remote: Compressing objects: 100% (1266/1266) 2023-11-14T04:45:17.0158031Z remote: Compressing objects: 100% (1266/1266), done. 2023-11-14T04:45:48.1959934Z remote: Total 1200035 (delta 1537), reused 1955 (delta 1171), pack-reused 1197588 2023-11-14T04:46:16.9774602Z [command]/usr/bin/git rev-parse --verify --quiet fe705c0d795479e12d728ea79f708a42b978ffae^{object} 2023-11-14T04:46:16.9808289Z fe705c0d795479e12d728ea79f708a42b978ffae 2023-11-14T04:46:16.9815040Z ##[endgroup] 2023-11-14T04:46:16.9815743Z ##[group]Determining the checkout info 2023-11-14T04:46:16.9818023Z ##[endgroup] 2023-11-14T04:46:16.9818674Z ##[group]Checking out the ref 2023-11-14T04:46:16.9823191Z [command]/usr/bin/git checkout --quiet --force fe705c0d795479e12d728ea79f708a42b978ffae 2023-11-14T04:46:18.9345948Z ##[endgroup] 2023-11-14T04:46:18.9346765Z ##[group]Setting up auth for fetching submodules 2023-11-14T04:46:18.9354183Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2023-11-14T04:46:18.9419517Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2023-11-14T04:46:18.9455241Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2023-11-14T04:46:18.9492556Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2023-11-14T04:46:18.9524399Z ##[endgroup] 2023-11-14T04:46:18.9525035Z ##[group]Fetching submodules 2023-11-14T04:46:18.9530393Z [command]/usr/bin/git submodule sync --recursive 2023-11-14T04:46:18.9875463Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2023-11-14T04:46:19.0193010Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2023-11-14T04:46:19.0195649Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2023-11-14T04:46:19.0199561Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2023-11-14T04:46:19.0209291Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2023-11-14T04:46:19.0210889Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2023-11-14T04:46:19.0212998Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2023-11-14T04:46:19.0215090Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2023-11-14T04:46:19.0216817Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2023-11-14T04:46:19.0220691Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2023-11-14T04:46:19.0224656Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2023-11-14T04:46:19.0229068Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2023-11-14T04:46:19.0233353Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2023-11-14T04:46:19.0238425Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2023-11-14T04:46:19.0242911Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2023-11-14T04:46:19.0247939Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2023-11-14T04:46:19.0252673Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2023-11-14T04:46:19.0257779Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2023-11-14T04:46:19.0263105Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2023-11-14T04:46:19.0268199Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2023-11-14T04:46:19.0273638Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2023-11-14T04:46:19.0279648Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2023-11-14T04:46:19.0285489Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2023-11-14T04:46:19.0291263Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2023-11-14T04:46:19.0297128Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2023-11-14T04:46:19.0303250Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2023-11-14T04:46:19.0309466Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2023-11-14T04:46:19.0316061Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2023-11-14T04:46:19.0323730Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2023-11-14T04:46:19.0330255Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2023-11-14T04:46:19.0336979Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2023-11-14T04:46:19.0343976Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2023-11-14T04:46:19.0350719Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2023-11-14T04:46:19.0358332Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2023-11-14T04:46:19.0365766Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2023-11-14T04:46:19.0372903Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2023-11-14T04:46:19.0380431Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2023-11-14T04:46:19.0387842Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2023-11-14T04:46:19.0395488Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2023-11-14T04:46:19.0404162Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2023-11-14T04:46:19.0412163Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2023-11-14T04:46:19.0441925Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2023-11-14T04:46:19.3760983Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2023-11-14T04:46:19.5796328Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2023-11-14T04:46:19.7807188Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2023-11-14T04:46:20.0578829Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2023-11-14T04:46:20.3549100Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2023-11-14T04:46:22.3377415Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2023-11-14T04:46:33.2005684Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2023-11-14T04:46:33.7228265Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2023-11-14T04:46:34.3428515Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2023-11-14T04:46:35.9582781Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2023-11-14T04:46:37.4210572Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2023-11-14T04:46:39.7259403Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2023-11-14T04:46:46.9085319Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2023-11-14T04:46:48.3313813Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2023-11-14T04:46:50.1971526Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2023-11-14T04:46:51.4524351Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2023-11-14T04:46:51.6563605Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2023-11-14T04:46:52.1638359Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2023-11-14T04:46:52.5334331Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2023-11-14T04:46:53.7337763Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2023-11-14T04:46:54.1302035Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2023-11-14T04:46:54.3367895Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2023-11-14T04:46:54.6144963Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2023-11-14T04:46:56.3969796Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2023-11-14T04:46:57.0849212Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2023-11-14T04:46:57.7003592Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2023-11-14T04:46:58.0282570Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2023-11-14T04:47:04.9728263Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2023-11-14T04:47:07.4053440Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2023-11-14T04:47:07.9339280Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2023-11-14T04:47:08.1751875Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2023-11-14T04:47:16.8057340Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2023-11-14T04:47:17.0056350Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2023-11-14T04:47:17.2473945Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2023-11-14T04:47:18.2079958Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2023-11-14T04:47:18.5405944Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2023-11-14T04:47:19.0604344Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2023-11-14T04:47:21.9792744Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2023-11-14T04:47:22.4528105Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2023-11-14T04:47:25.3015106Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2023-11-14T04:47:25.3147999Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2023-11-14T04:47:25.3251785Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2023-11-14T04:47:25.3542445Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2023-11-14T04:47:25.3823523Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2023-11-14T04:47:25.4324970Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2023-11-14T04:47:26.4413648Z Submodule path 'third_party/XNNPACK': checked out '51a987591a6fc9f0fc0707077f53d763ac132cbf' 2023-11-14T04:47:26.4693194Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2023-11-14T04:47:26.5960554Z Submodule path 'third_party/cpuinfo': checked out '6481e8bef08f606ddd627e4d3be89f64d62e1b8a' 2023-11-14T04:47:26.6378328Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2023-11-14T04:47:27.0153648Z Submodule path 'third_party/cudnn_frontend': checked out '12f35fa2be5994c1106367cac2fba21457b064f4' 2023-11-14T04:47:27.6647139Z Submodule path 'third_party/cutlass': checked out '44c704eae85da352d277d6f092f41412772f70e4' 2023-11-14T04:47:27.9771601Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2023-11-14T04:47:28.0509417Z Submodule path 'third_party/fbgemm': checked out 'd4eec11d72a2280cdf22b77561ba9b6f594cdd7e' 2023-11-14T04:47:28.0529437Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2023-11-14T04:47:28.0532380Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2023-11-14T04:47:28.0536338Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2023-11-14T04:47:28.0539673Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2023-11-14T04:47:28.0543685Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2023-11-14T04:47:28.0571939Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2023-11-14T04:47:28.9744456Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2023-11-14T04:47:29.5890195Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2023-11-14T04:47:32.0844691Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2023-11-14T04:47:33.1700813Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2023-11-14T04:47:33.5469789Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2023-11-14T04:47:33.6759423Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2023-11-14T04:47:34.2427411Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2023-11-14T04:47:34.3183774Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2023-11-14T04:47:34.3312149Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2023-11-14T04:47:34.4711030Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2023-11-14T04:47:34.5139543Z Submodule path 'third_party/fmt': checked out 'f5e54359df4c26b6230fc61d38aa294581393084' 2023-11-14T04:47:34.5247612Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2023-11-14T04:47:34.5735306Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2023-11-14T04:47:34.6027616Z Submodule path 'third_party/gloo': checked out 'cf1e1abc95d0b961222ee82b6935f76250fbcf16' 2023-11-14T04:47:34.6622982Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2023-11-14T04:47:34.6768547Z Submodule path 'third_party/ideep': checked out '13dd1fdc6217c9eccfa7166798dd81a0476e63dd' 2023-11-14T04:47:34.6786934Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2023-11-14T04:47:34.6815593Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2023-11-14T04:47:45.3545135Z Submodule path 'third_party/ideep/mkl-dnn': checked out '64f6bcbcbab628e96f33a62c3e975f8535a7bde4' 2023-11-14T04:47:45.3669672Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2023-11-14T04:47:45.3894014Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2023-11-14T04:47:45.5130564Z Submodule path 'third_party/kineto': checked out 'a30ca3f9509c2cfd28561abbca51328f0bdf9014' 2023-11-14T04:47:45.5150912Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2023-11-14T04:47:45.5156458Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2023-11-14T04:47:45.5158669Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2023-11-14T04:47:45.5187204Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2023-11-14T04:47:46.0193980Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2023-11-14T04:47:47.2457482Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2023-11-14T04:47:48.4319329Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2023-11-14T04:47:48.4337802Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-11-14T04:47:48.4340657Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-11-14T04:47:48.4344500Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-11-14T04:47:48.4382897Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-11-14T04:47:48.4386867Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-11-14T04:47:48.4389201Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-11-14T04:47:48.4391578Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-11-14T04:47:48.4393737Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-11-14T04:47:48.4396209Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2023-11-14T04:47:49.2560427Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2023-11-14T04:47:49.6519594Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2023-11-14T04:47:50.9183859Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2023-11-14T04:47:51.2526222Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2023-11-14T04:47:51.9761303Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2023-11-14T04:47:53.0672028Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2023-11-14T04:47:59.9216826Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2023-11-14T04:48:00.4287875Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2023-11-14T04:48:00.4504660Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2023-11-14T04:48:00.4983634Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2023-11-14T04:48:00.5134511Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2023-11-14T04:48:00.5153133Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-11-14T04:48:00.5182439Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2023-11-14T04:48:02.3094815Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2023-11-14T04:48:02.3319002Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2023-11-14T04:48:02.3822755Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2023-11-14T04:48:02.5149893Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2023-11-14T04:48:02.5343979Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2023-11-14T04:48:02.5778197Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2023-11-14T04:48:02.6494528Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2023-11-14T04:48:02.6944113Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2023-11-14T04:48:02.7219554Z Submodule path 'third_party/nccl/nccl': checked out '8c6c5951854a57ba90c4424fa040497f6defac46' 2023-11-14T04:48:02.7390629Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2023-11-14T04:48:02.8780117Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2023-11-14T04:48:03.3557928Z Submodule path 'third_party/onnx': checked out 'ccde5da81388ffa770ca98b64e07f803ad089414' 2023-11-14T04:48:03.3597865Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2023-11-14T04:48:03.3600958Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2023-11-14T04:48:03.3631349Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2023-11-14T04:48:03.8922256Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2023-11-14T04:48:05.2036416Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2023-11-14T04:48:05.2451703Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2023-11-14T04:48:05.2657860Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2023-11-14T04:48:05.2675906Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2023-11-14T04:48:05.2704622Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2023-11-14T04:48:07.9020469Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2023-11-14T04:48:07.9042987Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-11-14T04:48:07.9046135Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-11-14T04:48:07.9077358Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2023-11-14T04:48:08.7791722Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2023-11-14T04:48:09.7214688Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2023-11-14T04:48:09.8062344Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2023-11-14T04:48:09.8079695Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-11-14T04:48:09.8108701Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2023-11-14T04:48:10.1007001Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2023-11-14T04:48:10.1116538Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a' 2023-11-14T04:48:10.4474465Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2023-11-14T04:48:10.4499735Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2023-11-14T04:48:10.4502332Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2023-11-14T04:48:10.4533834Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2023-11-14T04:48:10.9890593Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2023-11-14T04:48:12.1156753Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2023-11-14T04:48:12.2023650Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2023-11-14T04:48:12.2122699Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2023-11-14T04:48:12.2257531Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413' 2023-11-14T04:48:12.2680478Z Submodule path 'third_party/pybind11': checked out '8a099e44b3d5f85b20f05828d919d2332a8de841' 2023-11-14T04:48:12.3017618Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2023-11-14T04:48:12.3554194Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff' 2023-11-14T04:48:12.4963767Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2023-11-14T04:48:12.5278128Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2023-11-14T04:48:12.5297202Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2023-11-14T04:48:12.5299876Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2023-11-14T04:48:12.5303392Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2023-11-14T04:48:12.5306757Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2023-11-14T04:48:12.5336354Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2023-11-14T04:48:13.6216805Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2023-11-14T04:48:13.8948848Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2023-11-14T04:48:15.4387286Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2023-11-14T04:48:16.4060223Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2023-11-14T04:48:16.4233957Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2023-11-14T04:48:16.4993965Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2023-11-14T04:48:16.5330229Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2023-11-14T04:48:16.5347729Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-11-14T04:48:16.5376394Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2023-11-14T04:48:16.7872847Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2023-11-14T04:48:16.9646808Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2023-11-14T04:48:16.9683348Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2023-11-14T04:48:17.0029319Z Entering 'android/libs/fbjni' 2023-11-14T04:48:17.0075191Z Entering 'third_party/FP16' 2023-11-14T04:48:17.0119324Z Entering 'third_party/FXdiv' 2023-11-14T04:48:17.0165397Z Entering 'third_party/NNPACK' 2023-11-14T04:48:17.0210027Z Entering 'third_party/QNNPACK' 2023-11-14T04:48:17.0255927Z Entering 'third_party/VulkanMemoryAllocator' 2023-11-14T04:48:17.0301125Z Entering 'third_party/XNNPACK' 2023-11-14T04:48:17.0363629Z Entering 'third_party/benchmark' 2023-11-14T04:48:17.0407703Z Entering 'third_party/cpuinfo' 2023-11-14T04:48:17.0454300Z Entering 'third_party/cub' 2023-11-14T04:48:17.0498036Z Entering 'third_party/cudnn_frontend' 2023-11-14T04:48:17.0548859Z Entering 'third_party/cutlass' 2023-11-14T04:48:17.0602980Z Entering 'third_party/eigen' 2023-11-14T04:48:17.0651628Z Entering 'third_party/fbgemm' 2023-11-14T04:48:17.0696968Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-11-14T04:48:17.0739908Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-11-14T04:48:17.0783552Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-11-14T04:48:17.0834827Z Entering 'third_party/fbgemm/third_party/googletest' 2023-11-14T04:48:17.0878677Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-11-14T04:48:17.0924234Z Entering 'third_party/flatbuffers' 2023-11-14T04:48:17.0972691Z Entering 'third_party/fmt' 2023-11-14T04:48:17.1019585Z Entering 'third_party/foxi' 2023-11-14T04:48:17.1064121Z Entering 'third_party/gemmlowp/gemmlowp' 2023-11-14T04:48:17.1108421Z Entering 'third_party/gloo' 2023-11-14T04:48:17.1153075Z Entering 'third_party/googletest' 2023-11-14T04:48:17.1198383Z Entering 'third_party/ideep' 2023-11-14T04:48:17.1240867Z Entering 'third_party/ideep/mkl-dnn' 2023-11-14T04:48:17.1291388Z Entering 'third_party/ios-cmake' 2023-11-14T04:48:17.1335379Z Entering 'third_party/ittapi' 2023-11-14T04:48:17.1379802Z Entering 'third_party/kineto' 2023-11-14T04:48:17.1424291Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-11-14T04:48:17.1468136Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-11-14T04:48:17.1513298Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-11-14T04:48:17.1557103Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-11-14T04:48:17.1602445Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-11-14T04:48:17.1646167Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-11-14T04:48:17.1692748Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-11-14T04:48:17.1736802Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-11-14T04:48:17.1780379Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-11-14T04:48:17.1824969Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-11-14T04:48:17.1869397Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-11-14T04:48:17.1914037Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-11-14T04:48:17.1958283Z Entering 'third_party/mimalloc' 2023-11-14T04:48:17.2005654Z Entering 'third_party/nccl/nccl' 2023-11-14T04:48:17.2050967Z Entering 'third_party/neon2sse' 2023-11-14T04:48:17.2094648Z Entering 'third_party/nlohmann' 2023-11-14T04:48:17.2139506Z Entering 'third_party/onnx' 2023-11-14T04:48:17.2202213Z Entering 'third_party/onnx/third_party/benchmark' 2023-11-14T04:48:17.2245575Z Entering 'third_party/onnx/third_party/pybind11' 2023-11-14T04:48:17.2291182Z Entering 'third_party/onnx-tensorrt' 2023-11-14T04:48:17.2334090Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-11-14T04:48:17.2382938Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-11-14T04:48:17.2426506Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-11-14T04:48:17.2470264Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-11-14T04:48:17.2519180Z Entering 'third_party/pocketfft' 2023-11-14T04:48:17.2561903Z Entering 'third_party/protobuf' 2023-11-14T04:48:17.2608958Z Entering 'third_party/protobuf/third_party/benchmark' 2023-11-14T04:48:17.2652341Z Entering 'third_party/protobuf/third_party/googletest' 2023-11-14T04:48:17.2698312Z Entering 'third_party/psimd' 2023-11-14T04:48:17.2742031Z Entering 'third_party/pthreadpool' 2023-11-14T04:48:17.2785926Z Entering 'third_party/pybind11' 2023-11-14T04:48:17.2830485Z Entering 'third_party/python-peachpy' 2023-11-14T04:48:17.2873349Z Entering 'third_party/sleef' 2023-11-14T04:48:17.2917363Z Entering 'third_party/tbb' 2023-11-14T04:48:17.2963506Z Entering 'third_party/tensorpipe' 2023-11-14T04:48:17.3007906Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-11-14T04:48:17.3051805Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-11-14T04:48:17.3094535Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-11-14T04:48:17.3138120Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-11-14T04:48:17.3180713Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-11-14T04:48:17.3226732Z Entering 'third_party/zstd' 2023-11-14T04:48:17.3286830Z ##[endgroup] 2023-11-14T04:48:17.3287722Z ##[group]Persisting credentials for submodules 2023-11-14T04:48:17.3293431Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2023-11-14T04:48:17.3616884Z Entering 'android/libs/fbjni' 2023-11-14T04:48:17.3671231Z Entering 'third_party/FP16' 2023-11-14T04:48:17.3728661Z Entering 'third_party/FXdiv' 2023-11-14T04:48:17.3783725Z Entering 'third_party/NNPACK' 2023-11-14T04:48:17.3840349Z Entering 'third_party/QNNPACK' 2023-11-14T04:48:17.3894867Z Entering 'third_party/VulkanMemoryAllocator' 2023-11-14T04:48:17.3951019Z Entering 'third_party/XNNPACK' 2023-11-14T04:48:17.4022107Z Entering 'third_party/benchmark' 2023-11-14T04:48:17.4077159Z Entering 'third_party/cpuinfo' 2023-11-14T04:48:17.4132849Z Entering 'third_party/cub' 2023-11-14T04:48:17.4188786Z Entering 'third_party/cudnn_frontend' 2023-11-14T04:48:17.4250400Z Entering 'third_party/cutlass' 2023-11-14T04:48:17.4312732Z Entering 'third_party/eigen' 2023-11-14T04:48:17.4371111Z Entering 'third_party/fbgemm' 2023-11-14T04:48:17.4426421Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-11-14T04:48:17.4481375Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-11-14T04:48:17.4536747Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-11-14T04:48:17.4599662Z Entering 'third_party/fbgemm/third_party/googletest' 2023-11-14T04:48:17.4655042Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-11-14T04:48:17.4709980Z Entering 'third_party/flatbuffers' 2023-11-14T04:48:17.4768341Z Entering 'third_party/fmt' 2023-11-14T04:48:17.4825045Z Entering 'third_party/foxi' 2023-11-14T04:48:17.4880759Z Entering 'third_party/gemmlowp/gemmlowp' 2023-11-14T04:48:17.4936063Z Entering 'third_party/gloo' 2023-11-14T04:48:17.4990612Z Entering 'third_party/googletest' 2023-11-14T04:48:17.5046510Z Entering 'third_party/ideep' 2023-11-14T04:48:17.5102325Z Entering 'third_party/ideep/mkl-dnn' 2023-11-14T04:48:17.5164131Z Entering 'third_party/ios-cmake' 2023-11-14T04:48:17.5218447Z Entering 'third_party/ittapi' 2023-11-14T04:48:17.5275312Z Entering 'third_party/kineto' 2023-11-14T04:48:17.5331242Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-11-14T04:48:17.5386916Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-11-14T04:48:17.5444673Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-11-14T04:48:17.5499788Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-11-14T04:48:17.5554980Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-11-14T04:48:17.5610303Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-11-14T04:48:17.5667658Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-11-14T04:48:17.5724350Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-11-14T04:48:17.5780304Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-11-14T04:48:17.5838411Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-11-14T04:48:17.5893949Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-11-14T04:48:17.5948988Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-11-14T04:48:17.6005168Z Entering 'third_party/mimalloc' 2023-11-14T04:48:17.6060404Z Entering 'third_party/nccl/nccl' 2023-11-14T04:48:17.6115785Z Entering 'third_party/neon2sse' 2023-11-14T04:48:17.6172117Z Entering 'third_party/nlohmann' 2023-11-14T04:48:17.6228175Z Entering 'third_party/onnx' 2023-11-14T04:48:17.6301700Z Entering 'third_party/onnx/third_party/benchmark' 2023-11-14T04:48:17.6358820Z Entering 'third_party/onnx/third_party/pybind11' 2023-11-14T04:48:17.6417279Z Entering 'third_party/onnx-tensorrt' 2023-11-14T04:48:17.6473361Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-11-14T04:48:17.6534826Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-11-14T04:48:17.6591854Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-11-14T04:48:17.6647588Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-11-14T04:48:17.6707871Z Entering 'third_party/pocketfft' 2023-11-14T04:48:17.6765256Z Entering 'third_party/protobuf' 2023-11-14T04:48:17.6824624Z Entering 'third_party/protobuf/third_party/benchmark' 2023-11-14T04:48:17.6880491Z Entering 'third_party/protobuf/third_party/googletest' 2023-11-14T04:48:17.6936798Z Entering 'third_party/psimd' 2023-11-14T04:48:17.6992539Z Entering 'third_party/pthreadpool' 2023-11-14T04:48:17.7048226Z Entering 'third_party/pybind11' 2023-11-14T04:48:17.7103523Z Entering 'third_party/python-peachpy' 2023-11-14T04:48:17.7159115Z Entering 'third_party/sleef' 2023-11-14T04:48:17.7213500Z Entering 'third_party/tbb' 2023-11-14T04:48:17.7270672Z Entering 'third_party/tensorpipe' 2023-11-14T04:48:17.7327440Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-11-14T04:48:17.7381234Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-11-14T04:48:17.7437376Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-11-14T04:48:17.7491350Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-11-14T04:48:17.7546137Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-11-14T04:48:17.7604779Z Entering 'third_party/zstd' 2023-11-14T04:48:17.7677138Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2023-11-14T04:48:17.7998680Z Entering 'android/libs/fbjni' 2023-11-14T04:48:17.8051867Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2023-11-14T04:48:17.8071261Z Entering 'third_party/FP16' 2023-11-14T04:48:17.8124540Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2023-11-14T04:48:17.8143501Z Entering 'third_party/FXdiv' 2023-11-14T04:48:17.8195840Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2023-11-14T04:48:17.8216715Z Entering 'third_party/NNPACK' 2023-11-14T04:48:17.8268286Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2023-11-14T04:48:17.8287639Z Entering 'third_party/QNNPACK' 2023-11-14T04:48:17.8339412Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2023-11-14T04:48:17.8358874Z Entering 'third_party/VulkanMemoryAllocator' 2023-11-14T04:48:17.8410965Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2023-11-14T04:48:17.8430189Z Entering 'third_party/XNNPACK' 2023-11-14T04:48:17.8484134Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2023-11-14T04:48:17.8518055Z Entering 'third_party/benchmark' 2023-11-14T04:48:17.8569979Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2023-11-14T04:48:17.8589559Z Entering 'third_party/cpuinfo' 2023-11-14T04:48:17.8641514Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2023-11-14T04:48:17.8660890Z Entering 'third_party/cub' 2023-11-14T04:48:17.8713956Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2023-11-14T04:48:17.8733587Z Entering 'third_party/cudnn_frontend' 2023-11-14T04:48:17.8785657Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2023-11-14T04:48:17.8811209Z Entering 'third_party/cutlass' 2023-11-14T04:48:17.8862987Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2023-11-14T04:48:17.8891300Z Entering 'third_party/eigen' 2023-11-14T04:48:17.8943192Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2023-11-14T04:48:17.8965483Z Entering 'third_party/fbgemm' 2023-11-14T04:48:17.9017904Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2023-11-14T04:48:17.9038473Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-11-14T04:48:17.9090343Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2023-11-14T04:48:17.9109015Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-11-14T04:48:17.9161263Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2023-11-14T04:48:17.9180799Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-11-14T04:48:17.9233562Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2023-11-14T04:48:17.9259978Z Entering 'third_party/fbgemm/third_party/googletest' 2023-11-14T04:48:17.9311264Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2023-11-14T04:48:17.9330078Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-11-14T04:48:17.9381190Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2023-11-14T04:48:17.9401522Z Entering 'third_party/flatbuffers' 2023-11-14T04:48:17.9454737Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2023-11-14T04:48:17.9477005Z Entering 'third_party/fmt' 2023-11-14T04:48:17.9529674Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2023-11-14T04:48:17.9548369Z Entering 'third_party/foxi' 2023-11-14T04:48:17.9600774Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2023-11-14T04:48:17.9619656Z Entering 'third_party/gemmlowp/gemmlowp' 2023-11-14T04:48:17.9672364Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2023-11-14T04:48:17.9691353Z Entering 'third_party/gloo' 2023-11-14T04:48:17.9743543Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2023-11-14T04:48:17.9763543Z Entering 'third_party/googletest' 2023-11-14T04:48:17.9815915Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2023-11-14T04:48:17.9835098Z Entering 'third_party/ideep' 2023-11-14T04:48:17.9887177Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2023-11-14T04:48:17.9905356Z Entering 'third_party/ideep/mkl-dnn' 2023-11-14T04:48:17.9957074Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2023-11-14T04:48:17.9982428Z Entering 'third_party/ios-cmake' 2023-11-14T04:48:18.0035997Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2023-11-14T04:48:18.0055239Z Entering 'third_party/ittapi' 2023-11-14T04:48:18.0107148Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2023-11-14T04:48:18.0126525Z Entering 'third_party/kineto' 2023-11-14T04:48:18.0179199Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2023-11-14T04:48:18.0198731Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-11-14T04:48:18.0251318Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2023-11-14T04:48:18.0270191Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-11-14T04:48:18.0322737Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2023-11-14T04:48:18.0343153Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-11-14T04:48:18.0397246Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2023-11-14T04:48:18.0415979Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-11-14T04:48:18.0468212Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2023-11-14T04:48:18.0487536Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-11-14T04:48:18.0540839Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2023-11-14T04:48:18.0558763Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-11-14T04:48:18.0610921Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2023-11-14T04:48:18.0631164Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-11-14T04:48:18.0685324Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2023-11-14T04:48:18.0703731Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-11-14T04:48:18.0756760Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2023-11-14T04:48:18.0776292Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-11-14T04:48:18.0828766Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2023-11-14T04:48:18.0850376Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-11-14T04:48:18.0902532Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2023-11-14T04:48:18.0923049Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-11-14T04:48:18.0975091Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2023-11-14T04:48:18.0993779Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-11-14T04:48:18.1045621Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2023-11-14T04:48:18.1065780Z Entering 'third_party/mimalloc' 2023-11-14T04:48:18.1118067Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2023-11-14T04:48:18.1138025Z Entering 'third_party/nccl/nccl' 2023-11-14T04:48:18.1190892Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2023-11-14T04:48:18.1210417Z Entering 'third_party/neon2sse' 2023-11-14T04:48:18.1262027Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2023-11-14T04:48:18.1281054Z Entering 'third_party/nlohmann' 2023-11-14T04:48:18.1333752Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2023-11-14T04:48:18.1354226Z Entering 'third_party/onnx' 2023-11-14T04:48:18.1406525Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2023-11-14T04:48:18.1444386Z Entering 'third_party/onnx/third_party/benchmark' 2023-11-14T04:48:18.1498600Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2023-11-14T04:48:18.1518566Z Entering 'third_party/onnx/third_party/pybind11' 2023-11-14T04:48:18.1571441Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2023-11-14T04:48:18.1591799Z Entering 'third_party/onnx-tensorrt' 2023-11-14T04:48:18.1644294Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2023-11-14T04:48:18.1662641Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-11-14T04:48:18.1716417Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2023-11-14T04:48:18.1740132Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-11-14T04:48:18.1793184Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2023-11-14T04:48:18.1812719Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-11-14T04:48:18.1866259Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2023-11-14T04:48:18.1885059Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-11-14T04:48:18.1938262Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2023-11-14T04:48:18.1962251Z Entering 'third_party/pocketfft' 2023-11-14T04:48:18.2014766Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2023-11-14T04:48:18.2033740Z Entering 'third_party/protobuf' 2023-11-14T04:48:18.2087218Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2023-11-14T04:48:18.2110519Z Entering 'third_party/protobuf/third_party/benchmark' 2023-11-14T04:48:18.2163890Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2023-11-14T04:48:18.2182787Z Entering 'third_party/protobuf/third_party/googletest' 2023-11-14T04:48:18.2236084Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2023-11-14T04:48:18.2255362Z Entering 'third_party/psimd' 2023-11-14T04:48:18.2307303Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2023-11-14T04:48:18.2326794Z Entering 'third_party/pthreadpool' 2023-11-14T04:48:18.2378838Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2023-11-14T04:48:18.2398381Z Entering 'third_party/pybind11' 2023-11-14T04:48:18.2450005Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2023-11-14T04:48:18.2469423Z Entering 'third_party/python-peachpy' 2023-11-14T04:48:18.2521615Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2023-11-14T04:48:18.2540709Z Entering 'third_party/sleef' 2023-11-14T04:48:18.2592602Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2023-11-14T04:48:18.2612077Z Entering 'third_party/tbb' 2023-11-14T04:48:18.2663566Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2023-11-14T04:48:18.2685583Z Entering 'third_party/tensorpipe' 2023-11-14T04:48:18.2738715Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2023-11-14T04:48:18.2758523Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-11-14T04:48:18.2810212Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2023-11-14T04:48:18.2828991Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-11-14T04:48:18.2881040Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2023-11-14T04:48:18.2899323Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-11-14T04:48:18.2950407Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2023-11-14T04:48:18.2969754Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-11-14T04:48:18.3021202Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2023-11-14T04:48:18.3039434Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-11-14T04:48:18.3090643Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2023-11-14T04:48:18.3112260Z Entering 'third_party/zstd' 2023-11-14T04:48:18.3164301Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2023-11-14T04:48:18.4162096Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2023-11-14T04:48:18.4497887Z Entering 'android/libs/fbjni' 2023-11-14T04:48:18.4542829Z Entering 'third_party/FP16' 2023-11-14T04:48:18.4588782Z Entering 'third_party/FXdiv' 2023-11-14T04:48:18.4633170Z Entering 'third_party/NNPACK' 2023-11-14T04:48:18.4678815Z Entering 'third_party/QNNPACK' 2023-11-14T04:48:18.4725525Z Entering 'third_party/VulkanMemoryAllocator' 2023-11-14T04:48:18.4769819Z Entering 'third_party/XNNPACK' 2023-11-14T04:48:18.4829907Z Entering 'third_party/benchmark' 2023-11-14T04:48:18.4874319Z Entering 'third_party/cpuinfo' 2023-11-14T04:48:18.4919514Z Entering 'third_party/cub' 2023-11-14T04:48:18.4963594Z Entering 'third_party/cudnn_frontend' 2023-11-14T04:48:18.5014630Z Entering 'third_party/cutlass' 2023-11-14T04:48:18.5068411Z Entering 'third_party/eigen' 2023-11-14T04:48:18.5117019Z Entering 'third_party/fbgemm' 2023-11-14T04:48:18.5162181Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-11-14T04:48:18.5207198Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-11-14T04:48:18.5251337Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-11-14T04:48:18.5301073Z Entering 'third_party/fbgemm/third_party/googletest' 2023-11-14T04:48:18.5345541Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-11-14T04:48:18.5389212Z Entering 'third_party/flatbuffers' 2023-11-14T04:48:18.5436742Z Entering 'third_party/fmt' 2023-11-14T04:48:18.5480956Z Entering 'third_party/foxi' 2023-11-14T04:48:18.5524675Z Entering 'third_party/gemmlowp/gemmlowp' 2023-11-14T04:48:18.5568677Z Entering 'third_party/gloo' 2023-11-14T04:48:18.5613130Z Entering 'third_party/googletest' 2023-11-14T04:48:18.5658691Z Entering 'third_party/ideep' 2023-11-14T04:48:18.5702206Z Entering 'third_party/ideep/mkl-dnn' 2023-11-14T04:48:18.5752650Z Entering 'third_party/ios-cmake' 2023-11-14T04:48:18.5797173Z Entering 'third_party/ittapi' 2023-11-14T04:48:18.5840785Z Entering 'third_party/kineto' 2023-11-14T04:48:18.5884894Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-11-14T04:48:18.5928625Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-11-14T04:48:18.5977890Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-11-14T04:48:18.6021599Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-11-14T04:48:18.6065858Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-11-14T04:48:18.6110084Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-11-14T04:48:18.6157436Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-11-14T04:48:18.6202617Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-11-14T04:48:18.6246018Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-11-14T04:48:18.6291651Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-11-14T04:48:18.6337373Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-11-14T04:48:18.6380898Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-11-14T04:48:18.6426406Z Entering 'third_party/mimalloc' 2023-11-14T04:48:18.6471458Z Entering 'third_party/nccl/nccl' 2023-11-14T04:48:18.6515587Z Entering 'third_party/neon2sse' 2023-11-14T04:48:18.6559007Z Entering 'third_party/nlohmann' 2023-11-14T04:48:18.6604645Z Entering 'third_party/onnx' 2023-11-14T04:48:18.6667688Z Entering 'third_party/onnx/third_party/benchmark' 2023-11-14T04:48:18.6711896Z Entering 'third_party/onnx/third_party/pybind11' 2023-11-14T04:48:18.6760086Z Entering 'third_party/onnx-tensorrt' 2023-11-14T04:48:18.6803666Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-11-14T04:48:18.6852443Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-11-14T04:48:18.6897253Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-11-14T04:48:18.6940937Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-11-14T04:48:18.6990835Z Entering 'third_party/pocketfft' 2023-11-14T04:48:18.7035997Z Entering 'third_party/protobuf' 2023-11-14T04:48:18.7087114Z Entering 'third_party/protobuf/third_party/benchmark' 2023-11-14T04:48:18.7132520Z Entering 'third_party/protobuf/third_party/googletest' 2023-11-14T04:48:18.7180339Z Entering 'third_party/psimd' 2023-11-14T04:48:18.7225070Z Entering 'third_party/pthreadpool' 2023-11-14T04:48:18.7269192Z Entering 'third_party/pybind11' 2023-11-14T04:48:18.7314292Z Entering 'third_party/python-peachpy' 2023-11-14T04:48:18.7358416Z Entering 'third_party/sleef' 2023-11-14T04:48:18.7402797Z Entering 'third_party/tbb' 2023-11-14T04:48:18.7448857Z Entering 'third_party/tensorpipe' 2023-11-14T04:48:18.7493240Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-11-14T04:48:18.7536761Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-11-14T04:48:18.7580955Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-11-14T04:48:18.7624392Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-11-14T04:48:18.7667389Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-11-14T04:48:18.7713475Z Entering 'third_party/zstd' 2023-11-14T04:48:18.7774441Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2023-11-14T04:48:18.8097748Z Entering 'android/libs/fbjni' 2023-11-14T04:48:18.8142854Z Entering 'third_party/FP16' 2023-11-14T04:48:18.8189399Z Entering 'third_party/FXdiv' 2023-11-14T04:48:18.8234764Z Entering 'third_party/NNPACK' 2023-11-14T04:48:18.8280632Z Entering 'third_party/QNNPACK' 2023-11-14T04:48:18.8327046Z Entering 'third_party/VulkanMemoryAllocator' 2023-11-14T04:48:18.8371589Z Entering 'third_party/XNNPACK' 2023-11-14T04:48:18.8433034Z Entering 'third_party/benchmark' 2023-11-14T04:48:18.8478338Z Entering 'third_party/cpuinfo' 2023-11-14T04:48:18.8524066Z Entering 'third_party/cub' 2023-11-14T04:48:18.8568116Z Entering 'third_party/cudnn_frontend' 2023-11-14T04:48:18.8620122Z Entering 'third_party/cutlass' 2023-11-14T04:48:18.8673002Z Entering 'third_party/eigen' 2023-11-14T04:48:18.8723431Z Entering 'third_party/fbgemm' 2023-11-14T04:48:18.8769477Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-11-14T04:48:18.8815052Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-11-14T04:48:18.8861509Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-11-14T04:48:18.8915691Z Entering 'third_party/fbgemm/third_party/googletest' 2023-11-14T04:48:18.8961173Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-11-14T04:48:18.9007762Z Entering 'third_party/flatbuffers' 2023-11-14T04:48:18.9058547Z Entering 'third_party/fmt' 2023-11-14T04:48:18.9105641Z Entering 'third_party/foxi' 2023-11-14T04:48:18.9151163Z Entering 'third_party/gemmlowp/gemmlowp' 2023-11-14T04:48:18.9198445Z Entering 'third_party/gloo' 2023-11-14T04:48:18.9244815Z Entering 'third_party/googletest' 2023-11-14T04:48:18.9291263Z Entering 'third_party/ideep' 2023-11-14T04:48:18.9336916Z Entering 'third_party/ideep/mkl-dnn' 2023-11-14T04:48:18.9391890Z Entering 'third_party/ios-cmake' 2023-11-14T04:48:18.9439437Z Entering 'third_party/ittapi' 2023-11-14T04:48:18.9485216Z Entering 'third_party/kineto' 2023-11-14T04:48:18.9532394Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-11-14T04:48:18.9579518Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-11-14T04:48:18.9628247Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-11-14T04:48:18.9674062Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-11-14T04:48:18.9720642Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-11-14T04:48:18.9766044Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-11-14T04:48:18.9871718Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-11-14T04:48:18.9919114Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-11-14T04:48:18.9965231Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-11-14T04:48:19.0011212Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-11-14T04:48:19.0058970Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-11-14T04:48:19.0105149Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-11-14T04:48:19.0151742Z Entering 'third_party/mimalloc' 2023-11-14T04:48:19.0199678Z Entering 'third_party/nccl/nccl' 2023-11-14T04:48:19.0245025Z Entering 'third_party/neon2sse' 2023-11-14T04:48:19.0289568Z Entering 'third_party/nlohmann' 2023-11-14T04:48:19.0340073Z Entering 'third_party/onnx' 2023-11-14T04:48:19.0409912Z Entering 'third_party/onnx/third_party/benchmark' 2023-11-14T04:48:19.0456029Z Entering 'third_party/onnx/third_party/pybind11' 2023-11-14T04:48:19.0507210Z Entering 'third_party/onnx-tensorrt' 2023-11-14T04:48:19.0553549Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-11-14T04:48:19.0605172Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-11-14T04:48:19.0652763Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-11-14T04:48:19.0698648Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-11-14T04:48:19.0749367Z Entering 'third_party/pocketfft' 2023-11-14T04:48:19.0796031Z Entering 'third_party/protobuf' 2023-11-14T04:48:19.0846282Z Entering 'third_party/protobuf/third_party/benchmark' 2023-11-14T04:48:19.0891399Z Entering 'third_party/protobuf/third_party/googletest' 2023-11-14T04:48:19.0937400Z Entering 'third_party/psimd' 2023-11-14T04:48:19.0983403Z Entering 'third_party/pthreadpool' 2023-11-14T04:48:19.1029271Z Entering 'third_party/pybind11' 2023-11-14T04:48:19.1074611Z Entering 'third_party/python-peachpy' 2023-11-14T04:48:19.1120838Z Entering 'third_party/sleef' 2023-11-14T04:48:19.1166095Z Entering 'third_party/tbb' 2023-11-14T04:48:19.1214518Z Entering 'third_party/tensorpipe' 2023-11-14T04:48:19.1261221Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-11-14T04:48:19.1305837Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-11-14T04:48:19.1350414Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-11-14T04:48:19.1396852Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-11-14T04:48:19.1439773Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-11-14T04:48:19.1489517Z Entering 'third_party/zstd' 2023-11-14T04:48:19.1550129Z ##[endgroup] 2023-11-14T04:48:19.1598010Z [command]/usr/bin/git log -1 --format='%H' 2023-11-14T04:48:19.1627700Z 'fe705c0d795479e12d728ea79f708a42b978ffae' 2023-11-14T04:48:19.1848223Z Prepare all required actions 2023-11-14T04:48:19.1848922Z Getting action download info 2023-11-14T04:48:19.3241358Z ##[group]Run ./.github/actions/setup-linux 2023-11-14T04:48:19.3241902Z env: 2023-11-14T04:48:19.3242267Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:19.3242691Z ##[endgroup] 2023-11-14T04:48:19.3330049Z ##[group]Run set -euo pipefail 2023-11-14T04:48:19.3330596Z set -euo pipefail 2023-11-14T04:48:19.3331103Z function get_ec2_metadata() { 2023-11-14T04:48:19.3331789Z  # Pulled from instance metadata endpoint for EC2 2023-11-14T04:48:19.3332898Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2023-11-14T04:48:19.3333892Z  category=$1 2023-11-14T04:48:19.3334568Z  # If it is GCP runner (runner name contains gcp), do not run this 2023-11-14T04:48:19.3335373Z  runner_name_str=i-0bfddc9f582173679 2023-11-14T04:48:19.3336115Z  if [[ $runner_name_str != *"gcp"* ]]; then 2023-11-14T04:48:19.3336931Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2023-11-14T04:48:19.3337668Z  else 2023-11-14T04:48:19.3338324Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2023-11-14T04:48:19.3339090Z  fi 2023-11-14T04:48:19.3339428Z } 2023-11-14T04:48:19.3339883Z echo "ami-id: $(get_ec2_metadata ami-id)" 2023-11-14T04:48:19.3340626Z echo "instance-id: $(get_ec2_metadata instance-id)" 2023-11-14T04:48:19.3341430Z echo "instance-type: $(get_ec2_metadata instance-type)" 2023-11-14T04:48:19.3342150Z echo "system info $(uname -a)" 2023-11-14T04:48:19.3351283Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:48:19.3351933Z env: 2023-11-14T04:48:19.3352264Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:19.3352699Z ##[endgroup] 2023-11-14T04:48:19.3461809Z ami-id: ami-096198a0bccc6bad4 2023-11-14T04:48:19.3530086Z instance-id: i-0bfddc9f582173679 2023-11-14T04:48:19.3596558Z instance-type: g3.8xlarge 2023-11-14T04:48:19.3606312Z system info Linux ip-10-0-49-138.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux 2023-11-14T04:48:19.3633922Z ##[group]Run if systemctl is-active --quiet docker; then 2023-11-14T04:48:19.3634997Z if systemctl is-active --quiet docker; then 2023-11-14T04:48:19.3635703Z  echo "Docker daemon is running..."; 2023-11-14T04:48:19.3636251Z else 2023-11-14T04:48:19.3636886Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2023-11-14T04:48:19.3637620Z fi 2023-11-14T04:48:19.3645370Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:48:19.3645997Z env: 2023-11-14T04:48:19.3646352Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:19.3646799Z ##[endgroup] 2023-11-14T04:48:19.3697612Z Docker daemon is running... 2023-11-14T04:48:19.3760764Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2023-11-14T04:48:19.3761679Z with: 2023-11-14T04:48:19.3762167Z shell: bash 2023-11-14T04:48:19.3762683Z timeout_minutes: 5 2023-11-14T04:48:19.3763412Z max_attempts: 3 2023-11-14T04:48:19.3764016Z retry_wait_seconds: 30 2023-11-14T04:48:19.3766013Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2023-11-14T04:48:19.3768072Z polling_interval_seconds: 1 2023-11-14T04:48:19.3768684Z warning_on_retry: true 2023-11-14T04:48:19.3769368Z continue_on_error: false 2023-11-14T04:48:19.3769942Z env: 2023-11-14T04:48:19.3770481Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:19.3771432Z AWS_RETRY_MODE: standard 2023-11-14T04:48:19.3772008Z AWS_MAX_ATTEMPTS: 5 2023-11-14T04:48:19.3772573Z AWS_DEFAULT_REGION: us-east-1 2023-11-14T04:48:19.3773356Z ##[endgroup] 2023-11-14T04:48:20.4206036Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2023-11-14T04:48:20.4207242Z Configure a credential helper to remove this warning. See 2023-11-14T04:48:20.4208444Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2023-11-14T04:48:20.4209200Z 2023-11-14T04:48:20.4209398Z Login Succeeded 2023-11-14T04:48:20.4534171Z Command completed after 1 attempt(s). 2023-11-14T04:48:20.4608123Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2023-11-14T04:48:20.4609132Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2023-11-14T04:48:20.4610038Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2023-11-14T04:48:20.4620240Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:48:20.4620987Z env: 2023-11-14T04:48:20.4621394Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:20.4621919Z ##[endgroup] 2023-11-14T04:48:20.4711167Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2023-11-14T04:48:20.4712219Z # ignore expansion of "docker ps -q" since it could be empty 2023-11-14T04:48:20.4713039Z # shellcheck disable=SC2046 2023-11-14T04:48:20.4713695Z docker stop $(docker ps -q) || true 2023-11-14T04:48:20.4714353Z # Prune all of the docker images 2023-11-14T04:48:20.4715328Z docker system prune -af 2023-11-14T04:48:20.4724077Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:48:20.4724791Z env: 2023-11-14T04:48:20.4725191Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:20.4725707Z ##[endgroup] 2023-11-14T04:48:20.5201520Z "docker stop" requires at least 1 argument. 2023-11-14T04:48:20.5202349Z See 'docker stop --help'. 2023-11-14T04:48:20.5202761Z 2023-11-14T04:48:20.5203084Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2023-11-14T04:48:20.5203615Z 2023-11-14T04:48:20.5203839Z Stop one or more running containers 2023-11-14T04:48:20.5437084Z Total reclaimed space: 0B 2023-11-14T04:48:20.5477172Z ##[group]Run set +e 2023-11-14T04:48:20.5477621Z set +e 2023-11-14T04:48:20.5478007Z set -x 2023-11-14T04:48:20.5478378Z  2023-11-14T04:48:20.5478767Z PT_DOMAIN=download.pytorch.org 2023-11-14T04:48:20.5479779Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2023-11-14T04:48:20.5481141Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2023-11-14T04:48:20.5482091Z # one is returned at random 2023-11-14T04:48:20.5482758Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2023-11-14T04:48:20.5483409Z  2023-11-14T04:48:20.5483809Z if [ -z "${RESOLVED_IP}" ]; then 2023-11-14T04:48:20.5484589Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2023-11-14T04:48:20.5485561Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2023-11-14T04:48:20.5486270Z  2023-11-14T04:48:20.5486657Z  if [ -z "${RESOLVED_IP}" ]; then 2023-11-14T04:48:20.5487358Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2023-11-14T04:48:20.5488010Z  exit 1 2023-11-14T04:48:20.5488391Z  fi 2023-11-14T04:48:20.5488749Z fi 2023-11-14T04:48:20.5489100Z  2023-11-14T04:48:20.5489530Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2023-11-14T04:48:20.5490187Z  # Clean up any old records first 2023-11-14T04:48:20.5490832Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2023-11-14T04:48:20.5491387Z fi 2023-11-14T04:48:20.5491746Z  2023-11-14T04:48:20.5492283Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2023-11-14T04:48:20.5493186Z cat /etc/hosts 2023-11-14T04:48:20.5501642Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:48:20.5502480Z env: 2023-11-14T04:48:20.5502862Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:20.5503290Z ##[endgroup] 2023-11-14T04:48:20.5531703Z + PT_DOMAIN=download.pytorch.org 2023-11-14T04:48:20.5537448Z ++ dig -4 +short download.pytorch.org 2023-11-14T04:48:20.5538428Z ++ tail -n1 2023-11-14T04:48:20.5668816Z + RESOLVED_IP=18.160.10.22 2023-11-14T04:48:20.5669354Z + '[' -z 18.160.10.22 ']' 2023-11-14T04:48:20.5669903Z + grep -r download.pytorch.org /etc/hosts 2023-11-14T04:48:20.5679044Z 18.160.10.28 download.pytorch.org 2023-11-14T04:48:20.5680460Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2023-11-14T04:48:20.5842373Z + echo '18.160.10.22 download.pytorch.org' 2023-11-14T04:48:20.5843394Z + sudo tee -a /etc/hosts 2023-11-14T04:48:20.6256705Z 18.160.10.22 download.pytorch.org 2023-11-14T04:48:20.6273657Z + cat /etc/hosts 2023-11-14T04:48:20.6281137Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2023-11-14T04:48:20.6297772Z ::1 localhost6 localhost6.localdomain6 2023-11-14T04:48:20.6298436Z 18.160.10.22 download.pytorch.org 2023-11-14T04:48:20.6473807Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2023-11-14T04:48:20.6474898Z with: 2023-11-14T04:48:20.6476253Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:20.6477668Z docker-build-dir: .ci/docker 2023-11-14T04:48:20.6478156Z working-directory: . 2023-11-14T04:48:20.6478724Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2023-11-14T04:48:20.6479386Z env: 2023-11-14T04:48:20.6479736Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:20.6480153Z ##[endgroup] 2023-11-14T04:48:20.6508543Z ##[group]Run set -ex 2023-11-14T04:48:20.6509024Z set -ex 2023-11-14T04:48:20.6509390Z  2023-11-14T04:48:20.6510102Z # If the docker build directory or the build script doesn't exist, the action will 2023-11-14T04:48:20.6511370Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2023-11-14T04:48:20.6512376Z # job could then download the pre-built image as usual 2023-11-14T04:48:20.6513319Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2023-11-14T04:48:20.6514176Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2023-11-14T04:48:20.6515532Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2023-11-14T04:48:20.6516263Z  2023-11-14T04:48:20.6516914Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2023-11-14T04:48:20.6517714Z  exit 0 2023-11-14T04:48:20.6518084Z else 2023-11-14T04:48:20.6518558Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2023-11-14T04:48:20.6519140Z fi 2023-11-14T04:48:20.6519473Z  2023-11-14T04:48:20.6520064Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2023-11-14T04:48:20.6521147Z  # The docker image name already includes the ECR prefix and tag, so we can just 2023-11-14T04:48:20.6522116Z  # use it as it is, but first let's extract the tag 2023-11-14T04:48:20.6523009Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2023-11-14T04:48:20.6523929Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2023-11-14T04:48:20.6524789Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2023-11-14T04:48:20.6525497Z else 2023-11-14T04:48:20.6526053Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2023-11-14T04:48:20.6526894Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2023-11-14T04:48:20.6527999Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2023-11-14T04:48:20.6529180Z fi 2023-11-14T04:48:20.6537567Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:48:20.6538214Z env: 2023-11-14T04:48:20.6538545Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:20.6539001Z REPO_NAME: pytorch 2023-11-14T04:48:20.6540283Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:20.6541634Z DOCKER_BUILD_DIR: .ci/docker 2023-11-14T04:48:20.6542273Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2023-11-14T04:48:20.6542935Z ##[endgroup] 2023-11-14T04:48:20.6571161Z + [[ ! -d .ci/docker ]] 2023-11-14T04:48:20.6571685Z + [[ ! -f .ci/docker/build.sh ]] 2023-11-14T04:48:20.6572182Z + echo skip=false 2023-11-14T04:48:20.6574478Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2023-11-14T04:48:20.6579657Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:20.6581075Z ++ awk -F '[:,]' '{print $2}' 2023-11-14T04:48:20.6590492Z + DOCKER_TAG=3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:20.6591324Z + echo docker-tag=3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:20.6593345Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:20.6659188Z ##[group]Run set +e 2023-11-14T04:48:20.6659670Z set +e 2023-11-14T04:48:20.6660065Z set -x 2023-11-14T04:48:20.6660465Z  2023-11-14T04:48:20.6661050Z # Check if image already exists, if it does then skip building it 2023-11-14T04:48:20.6661955Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2023-11-14T04:48:20.6662600Z  exit 0 2023-11-14T04:48:20.6662962Z fi 2023-11-14T04:48:20.6663328Z  2023-11-14T04:48:20.6663967Z # NB: This part requires a full checkout. Otherwise, the merge base will 2023-11-14T04:48:20.6665016Z # be empty. The default action would be to continue rebuild the image 2023-11-14T04:48:20.6665925Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2023-11-14T04:48:20.6666780Z  # if we're on the base branch then use the parent commit 2023-11-14T04:48:20.6667526Z  MERGE_BASE=$(git rev-parse HEAD~) 2023-11-14T04:48:20.6668050Z else 2023-11-14T04:48:20.6668641Z  # otherwise we're on a PR, so use the most recent base commit 2023-11-14T04:48:20.6669515Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2023-11-14T04:48:20.6670144Z fi 2023-11-14T04:48:20.6670500Z  2023-11-14T04:48:20.6670914Z if [[ -z "${MERGE_BASE}" ]]; then 2023-11-14T04:48:20.6671543Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2023-11-14T04:48:20.6672126Z  2023-11-14T04:48:20.6672959Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2023-11-14T04:48:20.6673939Z  exit 0 2023-11-14T04:48:20.6674300Z fi 2023-11-14T04:48:20.6675059Z  2023-11-14T04:48:20.6675628Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2023-11-14T04:48:20.6676869Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2023-11-14T04:48:20.6677891Z  exit 1 2023-11-14T04:48:20.6678278Z fi 2023-11-14T04:48:20.6678615Z  2023-11-14T04:48:20.6679236Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2023-11-14T04:48:20.6680615Z # If no image exists but the hash is the same as the previous hash then we should error out here 2023-11-14T04:48:20.6681687Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2023-11-14T04:48:20.6682927Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2023-11-14T04:48:20.6684317Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2023-11-14T04:48:20.6685138Z fi 2023-11-14T04:48:20.6685476Z  2023-11-14T04:48:20.6685931Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2023-11-14T04:48:20.6693918Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:48:20.6694555Z env: 2023-11-14T04:48:20.6694889Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:20.6695361Z DOCKER_BUILD_DIR: .ci/docker 2023-11-14T04:48:20.6695947Z BASE_REVISION: e1c872e009e8aa07281f38723353dc43baf2fc12 2023-11-14T04:48:20.6697388Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:20.6698986Z DOCKER_TAG: 3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:20.6699621Z ##[endgroup] 2023-11-14T04:48:20.6727900Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:20.9117487Z { 2023-11-14T04:48:20.9117901Z "schemaVersion": 2, 2023-11-14T04:48:20.9118542Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2023-11-14T04:48:20.9119290Z "config": { 2023-11-14T04:48:20.9119847Z "mediaType": "application/vnd.docker.container.image.v1+json", 2023-11-14T04:48:20.9120575Z "size": 44107, 2023-11-14T04:48:20.9121269Z "digest": "sha256:2e9cd9f64120afd8fb911e301ea6adc44cad71e8bfde2785095ab22e8b7f1f32" 2023-11-14T04:48:20.9122074Z }, 2023-11-14T04:48:20.9122437Z "layers": [ 2023-11-14T04:48:20.9122777Z { 2023-11-14T04:48:20.9123364Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9124059Z "size": 28580519, 2023-11-14T04:48:20.9124847Z "digest": "sha256:f0412dfb1aaea4892c13dab6f771bcd79641a4bdcb521ce881f5dcfc0df9a7a5" 2023-11-14T04:48:20.9125603Z }, 2023-11-14T04:48:20.9125953Z { 2023-11-14T04:48:20.9126493Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9127164Z "size": 7943208, 2023-11-14T04:48:20.9127890Z "digest": "sha256:20d547ab5eb53128ec992abaae1d1f5a15bfe39d7a18ac0e5e265b383ebf71bc" 2023-11-14T04:48:20.9128685Z }, 2023-11-14T04:48:20.9128993Z { 2023-11-14T04:48:20.9129518Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9130290Z "size": 56226342, 2023-11-14T04:48:20.9130947Z "digest": "sha256:ece84004a3cd8e7689a2ff0b1a1856a941e6cc31f7ec1eedc39073035a7adddd" 2023-11-14T04:48:20.9131747Z }, 2023-11-14T04:48:20.9132088Z { 2023-11-14T04:48:20.9132662Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9133386Z "size": 186, 2023-11-14T04:48:20.9134057Z "digest": "sha256:b08eef4b90c81de27f73f4aa3522ac14ee1e1146db910e97e17d9d25a969a292" 2023-11-14T04:48:20.9134853Z }, 2023-11-14T04:48:20.9135130Z { 2023-11-14T04:48:20.9135989Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9136718Z "size": 6879, 2023-11-14T04:48:20.9137385Z "digest": "sha256:aa315b7808f0b5e5badf02398e1f7ce1780952eda559defa9fb204e4f19c54c4" 2023-11-14T04:48:20.9138160Z }, 2023-11-14T04:48:20.9138500Z { 2023-11-14T04:48:20.9139028Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9139780Z "size": 1383661429, 2023-11-14T04:48:20.9140445Z "digest": "sha256:3288913d5f128020c71260b4e042a0fdda1ccf5966b3d1c9ef205a5099796a74" 2023-11-14T04:48:20.9141170Z }, 2023-11-14T04:48:20.9141529Z { 2023-11-14T04:48:20.9142366Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9143095Z "size": 62266, 2023-11-14T04:48:20.9143752Z "digest": "sha256:afae2e58c21592a24791ecc7c032f4eb2a1f279ebc922dd3f5d061f62ccb27ce" 2023-11-14T04:48:20.9144570Z }, 2023-11-14T04:48:20.9144841Z { 2023-11-14T04:48:20.9145417Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9146162Z "size": 1679, 2023-11-14T04:48:20.9146869Z "digest": "sha256:52a5515aef88a30087855f1aea16480c9b3f88f5b596e2575d95f805fa827a66" 2023-11-14T04:48:20.9147594Z }, 2023-11-14T04:48:20.9147888Z { 2023-11-14T04:48:20.9148420Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9149126Z "size": 1522, 2023-11-14T04:48:20.9149814Z "digest": "sha256:fc5398be2cb5046540483506f64caad54db6a0c992ddd1ac93b179c04d2566e5" 2023-11-14T04:48:20.9150577Z }, 2023-11-14T04:48:20.9150887Z { 2023-11-14T04:48:20.9151432Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9152127Z "size": 2445828736, 2023-11-14T04:48:20.9152836Z "digest": "sha256:c868322d41012c4c8c7a47807ec9da5a73e6fba9d43508f958818fedecfdd7d0" 2023-11-14T04:48:20.9153614Z }, 2023-11-14T04:48:20.9154058Z { 2023-11-14T04:48:20.9155004Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9155721Z "size": 85863, 2023-11-14T04:48:20.9156392Z "digest": "sha256:77c5d5a121d582b9bed76ba030514196fc1f7fa0a97d644dd9f48be27fc44030" 2023-11-14T04:48:20.9157321Z + exit 0 2023-11-14T04:48:20.9157658Z }, 2023-11-14T04:48:20.9157946Z { 2023-11-14T04:48:20.9158508Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9159212Z "size": 1802, 2023-11-14T04:48:20.9159874Z "digest": "sha256:742ea107d5593f5832e73f71fcd75de0d497ebd6aa3b52731ae9ff645f7c8dfa" 2023-11-14T04:48:20.9160657Z }, 2023-11-14T04:48:20.9160963Z { 2023-11-14T04:48:20.9161489Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9162204Z "size": 703101545, 2023-11-14T04:48:20.9162921Z "digest": "sha256:ea54d9137d4eaa49d45e879985d46d37927eb6888ed76ac84c44bbea18aa5214" 2023-11-14T04:48:20.9163684Z }, 2023-11-14T04:48:20.9164007Z { 2023-11-14T04:48:20.9164554Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9165236Z "size": 544, 2023-11-14T04:48:20.9165915Z "digest": "sha256:34d6327ee799ef8cfce6ce61681fdc8857945693f582a0f802ef484078d1d4fe" 2023-11-14T04:48:20.9166692Z }, 2023-11-14T04:48:20.9166983Z { 2023-11-14T04:48:20.9167528Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9168235Z "size": 1275, 2023-11-14T04:48:20.9168887Z "digest": "sha256:2f5ab0288987c559a6be7efb54eb348273e5fd656728862f147635163bd90195" 2023-11-14T04:48:20.9169659Z }, 2023-11-14T04:48:20.9169972Z { 2023-11-14T04:48:20.9170499Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9171221Z "size": 483, 2023-11-14T04:48:20.9171903Z "digest": "sha256:5e730d5fd91ad45c18d388d8678881deb27a599a37d8f5bafe8f8e9260c52074" 2023-11-14T04:48:20.9172665Z }, 2023-11-14T04:48:20.9172975Z { 2023-11-14T04:48:20.9173533Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9174223Z "size": 91702437, 2023-11-14T04:48:20.9174918Z "digest": "sha256:2020052b174f67659c6ad7b917a5c66ac34e08371e50f056bf55efb5d7070b49" 2023-11-14T04:48:20.9175688Z }, 2023-11-14T04:48:20.9175976Z { 2023-11-14T04:48:20.9176529Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9177235Z "size": 2999, 2023-11-14T04:48:20.9177892Z "digest": "sha256:80f0c3756826aaf127c22986788cf834fa34bde763927f4fffae89e27d26d918" 2023-11-14T04:48:20.9178672Z }, 2023-11-14T04:48:20.9178985Z { 2023-11-14T04:48:20.9179508Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9180421Z "size": 1812, 2023-11-14T04:48:20.9181107Z "digest": "sha256:5cb4c5d69233695279a8806cf1493a8cd477a9dc5e5990a1ecba05e1eb72c4f1" 2023-11-14T04:48:20.9181863Z }, 2023-11-14T04:48:20.9182177Z { 2023-11-14T04:48:20.9182732Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9183421Z "size": 701, 2023-11-14T04:48:20.9184100Z "digest": "sha256:192c8af52128c5185bdfc54e509898f55fcdd6b549c0776386a4339c89c8bdd2" 2023-11-14T04:48:20.9184877Z }, 2023-11-14T04:48:20.9185225Z { 2023-11-14T04:48:20.9185725Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9186510Z "size": 2447950741, 2023-11-14T04:48:20.9187265Z "digest": "sha256:fac5f79d8f40abb579dd6eb7b36f53158f64357b1d6c62014051d84db330407f" 2023-11-14T04:48:20.9188064Z }, 2023-11-14T04:48:20.9188357Z { 2023-11-14T04:48:20.9188910Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9189612Z "size": 380, 2023-11-14T04:48:20.9190277Z "digest": "sha256:ac2c08198bc746ba4dd1830557c7357375c3243e14783b9477f80d10debdbe68" 2023-11-14T04:48:20.9191050Z }, 2023-11-14T04:48:20.9191363Z { 2023-11-14T04:48:20.9192001Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9192739Z "size": 6320, 2023-11-14T04:48:20.9193435Z "digest": "sha256:54c34a5c940ffcc48f1786028cd3b73e2b5861a3ef9eef9dc8fd7a6f2949e2b5" 2023-11-14T04:48:20.9194206Z }, 2023-11-14T04:48:20.9194516Z { 2023-11-14T04:48:20.9195324Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9196011Z "size": 802, 2023-11-14T04:48:20.9196698Z "digest": "sha256:c597c95fbef4a958bd6184477f9fb135ea98103919dbe55a4e6ab6b406fd3fbf" 2023-11-14T04:48:20.9197487Z }, 2023-11-14T04:48:20.9197776Z { 2023-11-14T04:48:20.9198323Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9199025Z "size": 106, 2023-11-14T04:48:20.9199679Z "digest": "sha256:3808c65d8c0f541cbeb2bb89123651d4fea349565b734464c125ef27df379d84" 2023-11-14T04:48:20.9200477Z }, 2023-11-14T04:48:20.9200786Z { 2023-11-14T04:48:20.9201310Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9202024Z "size": 887, 2023-11-14T04:48:20.9202720Z "digest": "sha256:c715fe586a58db9e4bbf1fd8c7d3a8e7618a20681fb03a15391f8e6e9d0874d1" 2023-11-14T04:48:20.9203492Z }, 2023-11-14T04:48:20.9203804Z { 2023-11-14T04:48:20.9204354Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9205037Z "size": 121477383, 2023-11-14T04:48:20.9205761Z "digest": "sha256:1f385a3986c3b6edfc7c0a892c30cfda8ffd49d10bdae2d86142103c78e346e5" 2023-11-14T04:48:20.9206551Z }, 2023-11-14T04:48:20.9206840Z { 2023-11-14T04:48:20.9207392Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9208153Z "size": 109, 2023-11-14T04:48:20.9208811Z "digest": "sha256:d8ad661dea2125376bf054839466f30a001f2e348e54fe2951b0bcf5875a5582" 2023-11-14T04:48:20.9209607Z }, 2023-11-14T04:48:20.9209922Z { 2023-11-14T04:48:20.9210448Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9211156Z "size": 559, 2023-11-14T04:48:20.9211826Z "digest": "sha256:7195438c44c18d6054779424c46121854fdf1ca160e56cf544f828e4a06b1daf" 2023-11-14T04:48:20.9212569Z }, 2023-11-14T04:48:20.9212888Z { 2023-11-14T04:48:20.9213438Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9214119Z "size": 1176144, 2023-11-14T04:48:20.9214833Z "digest": "sha256:3c5a16afcc5fb51b7d68a3adff72d95957eebfe9a7d1250f676b833192fae795" 2023-11-14T04:48:20.9215642Z }, 2023-11-14T04:48:20.9215937Z { 2023-11-14T04:48:20.9216486Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9217191Z "size": 103, 2023-11-14T04:48:20.9217843Z "digest": "sha256:23ecc86c3805726132ad4de0755e2e1b4b67b288a4c275ee768335e02532353b" 2023-11-14T04:48:20.9218775Z }, 2023-11-14T04:48:20.9219098Z { 2023-11-14T04:48:20.9219625Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9220324Z "size": 1489, 2023-11-14T04:48:20.9221019Z "digest": "sha256:54f5b07b5ce6d502f30fb982b02be02b8a98e060cd57201966010e0e6b5045a7" 2023-11-14T04:48:20.9221783Z }, 2023-11-14T04:48:20.9222096Z { 2023-11-14T04:48:20.9222648Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9223337Z "size": 437057585, 2023-11-14T04:48:20.9224053Z "digest": "sha256:e8400b0ad3ec1329a7bd5a0e8f517c90acc59ea28ed92e1935a5065dd4b35df4" 2023-11-14T04:48:20.9224845Z }, 2023-11-14T04:48:20.9225136Z { 2023-11-14T04:48:20.9225943Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9226653Z "size": 160, 2023-11-14T04:48:20.9227306Z "digest": "sha256:7ef1ab3712ec09d84d8276812432df58573eb5a4f5091c247e9b4e25b6af87c6" 2023-11-14T04:48:20.9228085Z }, 2023-11-14T04:48:20.9228399Z { 2023-11-14T04:48:20.9228939Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9229643Z "size": 452, 2023-11-14T04:48:20.9230320Z "digest": "sha256:51783a9a2a4aa2c18cbe4d45228485f49089300d9d197432d086ecaee74af1ad" 2023-11-14T04:48:20.9231207Z }, 2023-11-14T04:48:20.9231542Z { 2023-11-14T04:48:20.9232098Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9232783Z "size": 56590734, 2023-11-14T04:48:20.9233480Z "digest": "sha256:0778288efbc7d96e3abf251814c144df31f43041202d85b61f621d86575bbd6b" 2023-11-14T04:48:20.9234248Z }, 2023-11-14T04:48:20.9234814Z { 2023-11-14T04:48:20.9235352Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9236051Z "size": 104, 2023-11-14T04:48:20.9236728Z "digest": "sha256:8b7bba61649c14d9c31c57ee9f8fa23bf80a3d67f18e41227207fa66991b6ac8" 2023-11-14T04:48:20.9237496Z }, 2023-11-14T04:48:20.9237810Z { 2023-11-14T04:48:20.9238341Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9239054Z "size": 402, 2023-11-14T04:48:20.9239748Z "digest": "sha256:d913ee57fbe5fa4af8feec010598662835bc5df4d9e5af62be2433f2d9ad6aed" 2023-11-14T04:48:20.9240523Z }, 2023-11-14T04:48:20.9240846Z { 2023-11-14T04:48:20.9241393Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9242077Z "size": 20262063, 2023-11-14T04:48:20.9242802Z "digest": "sha256:ad4cadf638a3dafe145549809d6ff107dd1e75fc67391c130eb3a2d3fecd0505" 2023-11-14T04:48:20.9243600Z }, 2023-11-14T04:48:20.9243889Z { 2023-11-14T04:48:20.9244435Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9245140Z "size": 440, 2023-11-14T04:48:20.9245804Z "digest": "sha256:88ffa4df8050fbf76da69f38a234218b38e293d0e0fc07c4c7c3cadbb5181c71" 2023-11-14T04:48:20.9246599Z }, 2023-11-14T04:48:20.9246913Z { 2023-11-14T04:48:20.9247441Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9248154Z "size": 701, 2023-11-14T04:48:20.9248831Z "digest": "sha256:192c8af52128c5185bdfc54e509898f55fcdd6b549c0776386a4339c89c8bdd2" 2023-11-14T04:48:20.9249580Z }, 2023-11-14T04:48:20.9249890Z { 2023-11-14T04:48:20.9250440Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9251121Z "size": 140, 2023-11-14T04:48:20.9251801Z "digest": "sha256:7b66c65e46fc5f6f1b5820e15fbe4297e3c649284760f9d27ee4a43cf7927757" 2023-11-14T04:48:20.9252582Z }, 2023-11-14T04:48:20.9252871Z { 2023-11-14T04:48:20.9253421Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9254134Z "size": 137, 2023-11-14T04:48:20.9254791Z "digest": "sha256:ca20e2be29876a691955ecb5b6730649f660a1f08ac903aebd73332a8442b002" 2023-11-14T04:48:20.9255572Z }, 2023-11-14T04:48:20.9255889Z { 2023-11-14T04:48:20.9256416Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9257269Z "size": 189, 2023-11-14T04:48:20.9257952Z "digest": "sha256:cb0f483351b235bd9f8e8148798ae79f5e9c3a9875fcee4c94836530dea0db07" 2023-11-14T04:48:20.9258709Z }, 2023-11-14T04:48:20.9259032Z { 2023-11-14T04:48:20.9259590Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9260270Z "size": 563, 2023-11-14T04:48:20.9260963Z "digest": "sha256:034bfa5626bffed7d19af0b036fe4a772abb0c2fb625f9a2b7208f90083da374" 2023-11-14T04:48:20.9261756Z }, 2023-11-14T04:48:20.9262043Z { 2023-11-14T04:48:20.9262594Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9263296Z "size": 43166336, 2023-11-14T04:48:20.9263972Z "digest": "sha256:c1310534828e57dc483e3b3dbf61df39e6bf2706423a53f15f6d165bd6da4ba8" 2023-11-14T04:48:20.9264745Z }, 2023-11-14T04:48:20.9265055Z { 2023-11-14T04:48:20.9265581Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9266284Z "size": 106, 2023-11-14T04:48:20.9266970Z "digest": "sha256:2aa9bf51637bd210c796735975c6f8b5a890a3d19f0595a9b082d11f34e7f713" 2023-11-14T04:48:20.9267718Z }, 2023-11-14T04:48:20.9268036Z { 2023-11-14T04:48:20.9268587Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9269418Z "size": 1154, 2023-11-14T04:48:20.9270146Z "digest": "sha256:f9bcb4185c0a07a32c8ac0f2c81c3998c78fc6e80d28eb4f17d1624dde28f8fe" 2023-11-14T04:48:20.9270938Z }, 2023-11-14T04:48:20.9271232Z { 2023-11-14T04:48:20.9271788Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9272494Z "size": 701, 2023-11-14T04:48:20.9273148Z "digest": "sha256:192c8af52128c5185bdfc54e509898f55fcdd6b549c0776386a4339c89c8bdd2" 2023-11-14T04:48:20.9273924Z }, 2023-11-14T04:48:20.9274238Z { 2023-11-14T04:48:20.9275039Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9275794Z "size": 138, 2023-11-14T04:48:20.9276475Z "digest": "sha256:c70de30968b07031424eeb3b61dd9d598a58272e2831f2230ec1bcb25ab20f6e" 2023-11-14T04:48:20.9277242Z }, 2023-11-14T04:48:20.9277558Z { 2023-11-14T04:48:20.9278106Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9278787Z "size": 118, 2023-11-14T04:48:20.9279472Z "digest": "sha256:1ad45855bb002887803eb27e45e0e99c3ce1ddb2d99dcbf2c8a9476c4482b406" 2023-11-14T04:48:20.9280261Z }, 2023-11-14T04:48:20.9280557Z { 2023-11-14T04:48:20.9281108Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9281822Z "size": 1046484338, 2023-11-14T04:48:20.9282529Z "digest": "sha256:9882884fae03caa7dcebd2e2f82e96a4f7fcf0091edfe59c469e73b7e191ece1" 2023-11-14T04:48:20.9283328Z }, 2023-11-14T04:48:20.9283643Z { 2023-11-14T04:48:20.9284173Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9284881Z "size": 172, 2023-11-14T04:48:20.9285569Z "digest": "sha256:ef5e32cf3412ce34dfa983a36d1dc538b162222f661f9d304117a3e93065d9a9" 2023-11-14T04:48:20.9286343Z }, 2023-11-14T04:48:20.9286661Z { 2023-11-14T04:48:20.9287206Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9287888Z "size": 1840, 2023-11-14T04:48:20.9288607Z "digest": "sha256:cdf3b334dcb11e7aab89d40de4889eb4e7ea9998cedd6ef890ca38a48c45b8d1" 2023-11-14T04:48:20.9289414Z }, 2023-11-14T04:48:20.9289703Z { 2023-11-14T04:48:20.9290258Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9290963Z "size": 7529791, 2023-11-14T04:48:20.9291660Z "digest": "sha256:ba019cce1eadffd12abe55f725b19e0c87b222ce899d0b66d30531601b152a98" 2023-11-14T04:48:20.9292466Z }, 2023-11-14T04:48:20.9292778Z { 2023-11-14T04:48:20.9293300Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9294000Z "size": 163, 2023-11-14T04:48:20.9294682Z "digest": "sha256:b59f1e2807d4563379c56ed6f4f0354b92b58465efcaa63deb40ee9737ef7e49" 2023-11-14T04:48:20.9295441Z }, 2023-11-14T04:48:20.9295914Z { 2023-11-14T04:48:20.9296474Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9297154Z "size": 7940, 2023-11-14T04:48:20.9297869Z "digest": "sha256:4dce0aae7c66ff5572e6f7ac9b1c66af607aade58e0e3ff0215edb344f52389f" 2023-11-14T04:48:20.9298678Z }, 2023-11-14T04:48:20.9298970Z { 2023-11-14T04:48:20.9299516Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9300215Z "size": 8071, 2023-11-14T04:48:20.9300883Z "digest": "sha256:93c6911b02879dc3f1578acef87dc6faea15c8d288cdb6ad21e5789a8f36af0c" 2023-11-14T04:48:20.9301672Z }, 2023-11-14T04:48:20.9301981Z { 2023-11-14T04:48:20.9302504Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9303206Z "size": 304, 2023-11-14T04:48:20.9303901Z "digest": "sha256:fcdaac0fbb69c12f1e7245a397b7ec568e12226a94726853e85fbd19d3d78fe7" 2023-11-14T04:48:20.9304673Z }, 2023-11-14T04:48:20.9304984Z { 2023-11-14T04:48:20.9305545Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9306227Z "size": 7614964, 2023-11-14T04:48:20.9306927Z "digest": "sha256:d80b8d0f01dcea7652b8d14f9705e86811d00a8fe09dac606c4b3e815a865fcb" 2023-11-14T04:48:20.9307880Z }, 2023-11-14T04:48:20.9308195Z { 2023-11-14T04:48:20.9308754Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9309454Z "size": 108, 2023-11-14T04:48:20.9310097Z "digest": "sha256:762352f3c28b2180f18b6245538bc16e3e52af45829072df88c6e4ab202eeddc" 2023-11-14T04:48:20.9310863Z }, 2023-11-14T04:48:20.9311176Z { 2023-11-14T04:48:20.9311700Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9312408Z "size": 54145478, 2023-11-14T04:48:20.9313126Z "digest": "sha256:f39a9cff48d63e07c1065b34be0d3815db7edd7292a1ea820a3a48ad62781f26" 2023-11-14T04:48:20.9313919Z }, 2023-11-14T04:48:20.9314211Z { 2023-11-14T04:48:20.9314939Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9315668Z "size": 513, 2023-11-14T04:48:20.9316330Z "digest": "sha256:0c4fbc80353d21fa656812cf7eb51b983b9f8513d020d52c8da372a08897ee39" 2023-11-14T04:48:20.9317109Z }, 2023-11-14T04:48:20.9317440Z { 2023-11-14T04:48:20.9317973Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9318680Z "size": 2997737455, 2023-11-14T04:48:20.9319406Z "digest": "sha256:bb1924fd13d8efdb4bc768ada19376b6b89d094b36dd6e624d81e54a8ed37c55" 2023-11-14T04:48:20.9320179Z }, 2023-11-14T04:48:20.9320488Z { 2023-11-14T04:48:20.9321037Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2023-11-14T04:48:20.9321713Z "size": 106, 2023-11-14T04:48:20.9322393Z "digest": "sha256:ea619973e9afb03f2d49a961986c66cf734bd9243e567cd5acd9818fd418ef49" 2023-11-14T04:48:20.9323172Z } 2023-11-14T04:48:20.9323460Z ] 2023-11-14T04:48:20.9323771Z } 2023-11-14T04:48:20.9576762Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2023-11-14T04:48:20.9577530Z with: 2023-11-14T04:48:20.9578709Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:20.9579999Z env: 2023-11-14T04:48:20.9580327Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:20.9580738Z ##[endgroup] 2023-11-14T04:48:20.9604557Z ##[group]Run retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2023-11-14T04:48:20.9605414Z retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2023-11-14T04:48:20.9606253Z # ignore output since only exit code is used for conditional 2023-11-14T04:48:20.9607119Z # only pull docker image if it's not available locally 2023-11-14T04:48:20.9608121Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2023-11-14T04:48:20.9609069Z  retry docker pull "${DOCKER_IMAGE}" 2023-11-14T04:48:20.9609608Z fi 2023-11-14T04:48:20.9618664Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:48:20.9619283Z env: 2023-11-14T04:48:20.9619623Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:48:20.9620874Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:48:20.9622178Z ##[endgroup] 2023-11-14T04:48:21.2150157Z 3cbebab0ba3af5b92c112f499347ddfdd0f631c8: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9 2023-11-14T04:48:21.2151526Z f0412dfb1aae: Pulling fs layer 2023-11-14T04:48:21.2152185Z 20d547ab5eb5: Pulling fs layer 2023-11-14T04:48:21.2152834Z ece84004a3cd: Pulling fs layer 2023-11-14T04:48:21.2153361Z b08eef4b90c8: Pulling fs layer 2023-11-14T04:48:21.2154078Z aa315b7808f0: Pulling fs layer 2023-11-14T04:48:21.2154877Z 3288913d5f12: Pulling fs layer 2023-11-14T04:48:21.2155574Z afae2e58c215: Pulling fs layer 2023-11-14T04:48:21.2156472Z 52a5515aef88: Pulling fs layer 2023-11-14T04:48:21.2157049Z fc5398be2cb5: Pulling fs layer 2023-11-14T04:48:21.2157720Z c868322d4101: Pulling fs layer 2023-11-14T04:48:21.2158314Z 77c5d5a121d5: Pulling fs layer 2023-11-14T04:48:21.2158782Z aa315b7808f0: Waiting 2023-11-14T04:48:21.2159442Z 742ea107d559: Pulling fs layer 2023-11-14T04:48:21.2160201Z ea54d9137d4e: Pulling fs layer 2023-11-14T04:48:21.2161117Z afae2e58c215: Waiting 2023-11-14T04:48:21.2161528Z 34d6327ee799: Pulling fs layer 2023-11-14T04:48:21.2162092Z b08eef4b90c8: Waiting 2023-11-14T04:48:21.2162846Z 2f5ab0288987: Pulling fs layer 2023-11-14T04:48:21.2164041Z 52a5515aef88: Waiting 2023-11-14T04:48:21.2164923Z 5e730d5fd91a: Pulling fs layer 2023-11-14T04:48:21.2165842Z fc5398be2cb5: Waiting 2023-11-14T04:48:21.2166554Z 77c5d5a121d5: Waiting 2023-11-14T04:48:21.2167151Z 2020052b174f: Pulling fs layer 2023-11-14T04:48:21.2167591Z 742ea107d559: Waiting 2023-11-14T04:48:21.2168054Z ea54d9137d4e: Waiting 2023-11-14T04:48:21.2168640Z 34d6327ee799: Waiting 2023-11-14T04:48:21.2169020Z c868322d4101: Waiting 2023-11-14T04:48:21.2169733Z 5e730d5fd91a: Waiting 2023-11-14T04:48:21.2170315Z 2f5ab0288987: Waiting 2023-11-14T04:48:21.2171063Z 80f0c3756826: Pulling fs layer 2023-11-14T04:48:21.2171952Z 5cb4c5d69233: Pulling fs layer 2023-11-14T04:48:21.2173235Z 2020052b174f: Waiting 2023-11-14T04:48:21.2173887Z 192c8af52128: Pulling fs layer 2023-11-14T04:48:21.2175026Z 80f0c3756826: Waiting 2023-11-14T04:48:21.2175702Z fac5f79d8f40: Pulling fs layer 2023-11-14T04:48:21.2176461Z ac2c08198bc7: Pulling fs layer 2023-11-14T04:48:21.2177343Z 5cb4c5d69233: Waiting 2023-11-14T04:48:21.2177926Z 54c34a5c940f: Pulling fs layer 2023-11-14T04:48:21.2178714Z fac5f79d8f40: Waiting 2023-11-14T04:48:21.2179119Z c597c95fbef4: Pulling fs layer 2023-11-14T04:48:21.2179813Z 3808c65d8c0f: Pulling fs layer 2023-11-14T04:48:21.2180669Z c715fe586a58: Pulling fs layer 2023-11-14T04:48:21.2181288Z 54c34a5c940f: Waiting 2023-11-14T04:48:21.2181960Z 1f385a3986c3: Pulling fs layer 2023-11-14T04:48:21.2182459Z ac2c08198bc7: Waiting 2023-11-14T04:48:21.2183137Z 3808c65d8c0f: Waiting 2023-11-14T04:48:21.2183599Z d8ad661dea21: Pulling fs layer 2023-11-14T04:48:21.2184508Z c715fe586a58: Waiting 2023-11-14T04:48:21.2184949Z 7195438c44c1: Pulling fs layer 2023-11-14T04:48:21.2185378Z 3c5a16afcc5f: Pulling fs layer 2023-11-14T04:48:21.2185862Z 23ecc86c3805: Pulling fs layer 2023-11-14T04:48:21.2186723Z 1f385a3986c3: Waiting 2023-11-14T04:48:21.2187149Z 54f5b07b5ce6: Pulling fs layer 2023-11-14T04:48:21.2187619Z 3c5a16afcc5f: Waiting 2023-11-14T04:48:21.2188007Z d8ad661dea21: Waiting 2023-11-14T04:48:21.2188431Z e8400b0ad3ec: Pulling fs layer 2023-11-14T04:48:21.2188918Z 7ef1ab3712ec: Pulling fs layer 2023-11-14T04:48:21.2189358Z e8400b0ad3ec: Waiting 2023-11-14T04:48:21.2189779Z 51783a9a2a4a: Pulling fs layer 2023-11-14T04:48:21.2190267Z 0778288efbc7: Pulling fs layer 2023-11-14T04:48:21.2190935Z 8b7bba61649c: Pulling fs layer 2023-11-14T04:48:21.2191598Z 51783a9a2a4a: Waiting 2023-11-14T04:48:21.2192005Z 0778288efbc7: Waiting 2023-11-14T04:48:21.2192651Z d913ee57fbe5: Pulling fs layer 2023-11-14T04:48:21.2193134Z ad4cadf638a3: Pulling fs layer 2023-11-14T04:48:21.2193813Z d913ee57fbe5: Waiting 2023-11-14T04:48:21.2194235Z 88ffa4df8050: Pulling fs layer 2023-11-14T04:48:21.2195970Z 7b66c65e46fc: Pulling fs layer 2023-11-14T04:48:21.2197185Z ca20e2be2987: Pulling fs layer 2023-11-14T04:48:21.2197634Z ad4cadf638a3: Waiting 2023-11-14T04:48:21.2198067Z 8b7bba61649c: Waiting 2023-11-14T04:48:21.2198467Z cb0f483351b2: Pulling fs layer 2023-11-14T04:48:21.2198924Z 034bfa5626bf: Pulling fs layer 2023-11-14T04:48:21.2199376Z 88ffa4df8050: Waiting 2023-11-14T04:48:21.2199790Z c1310534828e: Pulling fs layer 2023-11-14T04:48:21.2200243Z 2aa9bf51637b: Pulling fs layer 2023-11-14T04:48:21.2200717Z f9bcb4185c0a: Pulling fs layer 2023-11-14T04:48:21.2201196Z c70de30968b0: Pulling fs layer 2023-11-14T04:48:21.2201628Z 2aa9bf51637b: Waiting 2023-11-14T04:48:21.2202381Z 1ad45855bb00: Pulling fs layer 2023-11-14T04:48:21.2202870Z c1310534828e: Waiting 2023-11-14T04:48:21.2203260Z 034bfa5626bf: Waiting 2023-11-14T04:48:21.2203652Z c70de30968b0: Waiting 2023-11-14T04:48:21.2204260Z 9882884fae03: Pulling fs layer 2023-11-14T04:48:21.2204879Z ef5e32cf3412: Pulling fs layer 2023-11-14T04:48:21.2205496Z 1ad45855bb00: Waiting 2023-11-14T04:48:21.2206297Z f9bcb4185c0a: Waiting 2023-11-14T04:48:21.2206849Z cdf3b334dcb1: Pulling fs layer 2023-11-14T04:48:21.2207343Z ba019cce1ead: Pulling fs layer 2023-11-14T04:48:21.2207860Z b59f1e2807d4: Pulling fs layer 2023-11-14T04:48:21.2208298Z 7b66c65e46fc: Waiting 2023-11-14T04:48:21.2208719Z 4dce0aae7c66: Pulling fs layer 2023-11-14T04:48:21.2209175Z ef5e32cf3412: Waiting 2023-11-14T04:48:21.2209556Z cdf3b334dcb1: Waiting 2023-11-14T04:48:21.2209970Z 93c6911b0287: Pulling fs layer 2023-11-14T04:48:21.2210425Z 4dce0aae7c66: Waiting 2023-11-14T04:48:21.2210829Z fcdaac0fbb69: Pulling fs layer 2023-11-14T04:48:21.2211308Z d80b8d0f01dc: Pulling fs layer 2023-11-14T04:48:21.2211791Z 762352f3c28b: Pulling fs layer 2023-11-14T04:48:21.2212256Z f39a9cff48d6: Pulling fs layer 2023-11-14T04:48:21.2212731Z 0c4fbc80353d: Pulling fs layer 2023-11-14T04:48:21.2213212Z d80b8d0f01dc: Waiting 2023-11-14T04:48:21.2213593Z 93c6911b0287: Waiting 2023-11-14T04:48:21.2214011Z bb1924fd13d8: Pulling fs layer 2023-11-14T04:48:21.2214463Z 0c4fbc80353d: Waiting 2023-11-14T04:48:21.2214864Z ea619973e9af: Pulling fs layer 2023-11-14T04:48:21.2215761Z bb1924fd13d8: Waiting 2023-11-14T04:48:21.2216163Z ea619973e9af: Waiting 2023-11-14T04:48:21.4156255Z 20d547ab5eb5: Verifying Checksum 2023-11-14T04:48:21.4156894Z 20d547ab5eb5: Download complete 2023-11-14T04:48:21.5334937Z b08eef4b90c8: Verifying Checksum 2023-11-14T04:48:21.5335492Z b08eef4b90c8: Download complete 2023-11-14T04:48:21.5533929Z f0412dfb1aae: Verifying Checksum 2023-11-14T04:48:21.5534952Z f0412dfb1aae: Download complete 2023-11-14T04:48:21.6082808Z aa315b7808f0: Download complete 2023-11-14T04:48:21.6999037Z afae2e58c215: Download complete 2023-11-14T04:48:21.8090591Z 52a5515aef88: Verifying Checksum 2023-11-14T04:48:21.8091251Z 52a5515aef88: Download complete 2023-11-14T04:48:21.8914309Z fc5398be2cb5: Verifying Checksum 2023-11-14T04:48:21.8915583Z fc5398be2cb5: Download complete 2023-11-14T04:48:22.0265531Z ece84004a3cd: Verifying Checksum 2023-11-14T04:48:22.0266173Z ece84004a3cd: Download complete 2023-11-14T04:48:22.3746193Z 77c5d5a121d5: Verifying Checksum 2023-11-14T04:48:22.3746954Z 77c5d5a121d5: Download complete 2023-11-14T04:48:22.3883120Z f0412dfb1aae: Pull complete 2023-11-14T04:48:22.4509484Z 742ea107d559: Download complete 2023-11-14T04:48:22.6831996Z 20d547ab5eb5: Pull complete 2023-11-14T04:48:23.7323050Z ece84004a3cd: Pull complete 2023-11-14T04:48:23.8395161Z b08eef4b90c8: Pull complete 2023-11-14T04:48:23.9552004Z aa315b7808f0: Pull complete 2023-11-14T04:48:34.5843350Z ea54d9137d4e: Verifying Checksum 2023-11-14T04:48:34.5844156Z ea54d9137d4e: Download complete 2023-11-14T04:48:34.6614053Z 34d6327ee799: Verifying Checksum 2023-11-14T04:48:34.6614702Z 34d6327ee799: Download complete 2023-11-14T04:48:34.7412867Z 2f5ab0288987: Verifying Checksum 2023-11-14T04:48:34.7413519Z 2f5ab0288987: Download complete 2023-11-14T04:48:34.8277680Z 5e730d5fd91a: Verifying Checksum 2023-11-14T04:48:34.8278314Z 5e730d5fd91a: Download complete 2023-11-14T04:48:37.0644894Z 2020052b174f: Verifying Checksum 2023-11-14T04:48:37.0645576Z 2020052b174f: Download complete 2023-11-14T04:48:37.1516399Z 80f0c3756826: Verifying Checksum 2023-11-14T04:48:37.1516942Z 80f0c3756826: Download complete 2023-11-14T04:48:37.2317130Z 5cb4c5d69233: Download complete 2023-11-14T04:48:37.3219167Z 192c8af52128: Verifying Checksum 2023-11-14T04:48:37.3219677Z 192c8af52128: Download complete 2023-11-14T04:49:10.4898682Z 3288913d5f12: Download complete 2023-11-14T04:49:10.6208211Z ac2c08198bc7: Verifying Checksum 2023-11-14T04:49:10.6208915Z ac2c08198bc7: Download complete 2023-11-14T04:49:10.7136222Z 54c34a5c940f: Download complete 2023-11-14T04:49:10.7987275Z c597c95fbef4: Verifying Checksum 2023-11-14T04:49:10.7987967Z c597c95fbef4: Download complete 2023-11-14T04:49:10.8885953Z 3808c65d8c0f: Verifying Checksum 2023-11-14T04:49:10.8886812Z 3808c65d8c0f: Download complete 2023-11-14T04:49:10.9752605Z c715fe586a58: Verifying Checksum 2023-11-14T04:49:10.9753704Z c715fe586a58: Download complete 2023-11-14T04:49:14.1526839Z 1f385a3986c3: Verifying Checksum 2023-11-14T04:49:14.1527517Z 1f385a3986c3: Download complete 2023-11-14T04:49:14.2526145Z d8ad661dea21: Download complete 2023-11-14T04:49:14.3311509Z 7195438c44c1: Download complete 2023-11-14T04:49:14.4628547Z 3c5a16afcc5f: Verifying Checksum 2023-11-14T04:49:14.4629791Z 3c5a16afcc5f: Download complete 2023-11-14T04:49:14.5407191Z 23ecc86c3805: Download complete 2023-11-14T04:49:14.6353034Z 54f5b07b5ce6: Verifying Checksum 2023-11-14T04:49:14.6354004Z 54f5b07b5ce6: Download complete 2023-11-14T04:49:23.6591833Z e8400b0ad3ec: Verifying Checksum 2023-11-14T04:49:23.6592525Z e8400b0ad3ec: Download complete 2023-11-14T04:49:23.7435620Z 7ef1ab3712ec: Verifying Checksum 2023-11-14T04:49:23.7436379Z 7ef1ab3712ec: Download complete 2023-11-14T04:49:23.8245940Z 51783a9a2a4a: Verifying Checksum 2023-11-14T04:49:23.8246637Z 51783a9a2a4a: Download complete 2023-11-14T04:49:25.1502505Z 0778288efbc7: Verifying Checksum 2023-11-14T04:49:25.1503676Z 0778288efbc7: Download complete 2023-11-14T04:49:25.2403710Z 8b7bba61649c: Verifying Checksum 2023-11-14T04:49:25.2404813Z 8b7bba61649c: Download complete 2023-11-14T04:49:25.3307878Z d913ee57fbe5: Verifying Checksum 2023-11-14T04:49:25.3308842Z d913ee57fbe5: Download complete 2023-11-14T04:49:25.5951997Z ad4cadf638a3: Verifying Checksum 2023-11-14T04:49:25.5952722Z ad4cadf638a3: Download complete 2023-11-14T04:49:25.6765323Z 88ffa4df8050: Verifying Checksum 2023-11-14T04:49:25.6766019Z 88ffa4df8050: Download complete 2023-11-14T04:49:25.7693165Z 7b66c65e46fc: Verifying Checksum 2023-11-14T04:49:25.7693808Z 7b66c65e46fc: Download complete 2023-11-14T04:49:25.8627575Z ca20e2be2987: Verifying Checksum 2023-11-14T04:49:25.8628220Z ca20e2be2987: Download complete 2023-11-14T04:49:25.9579401Z cb0f483351b2: Verifying Checksum 2023-11-14T04:49:25.9580102Z cb0f483351b2: Download complete 2023-11-14T04:49:26.0412409Z 034bfa5626bf: Verifying Checksum 2023-11-14T04:49:26.0413318Z 034bfa5626bf: Download complete 2023-11-14T04:49:27.1550071Z c1310534828e: Verifying Checksum 2023-11-14T04:49:27.1550837Z c1310534828e: Download complete 2023-11-14T04:49:27.2324461Z 2aa9bf51637b: Verifying Checksum 2023-11-14T04:49:27.2325915Z 2aa9bf51637b: Download complete 2023-11-14T04:49:27.3381368Z f9bcb4185c0a: Download complete 2023-11-14T04:49:27.4361019Z c70de30968b0: Verifying Checksum 2023-11-14T04:49:27.4361640Z c70de30968b0: Download complete 2023-11-14T04:49:27.5249319Z 1ad45855bb00: Verifying Checksum 2023-11-14T04:49:27.5250532Z 1ad45855bb00: Download complete 2023-11-14T04:49:28.5580235Z fac5f79d8f40: Verifying Checksum 2023-11-14T04:49:28.5580912Z fac5f79d8f40: Download complete 2023-11-14T04:49:28.6428689Z ef5e32cf3412: Download complete 2023-11-14T04:49:28.7196958Z cdf3b334dcb1: Verifying Checksum 2023-11-14T04:49:28.7197888Z cdf3b334dcb1: Download complete 2023-11-14T04:49:29.0061874Z ba019cce1ead: Verifying Checksum 2023-11-14T04:49:29.0062437Z ba019cce1ead: Download complete 2023-11-14T04:49:29.1538356Z b59f1e2807d4: Verifying Checksum 2023-11-14T04:49:29.1538983Z b59f1e2807d4: Download complete 2023-11-14T04:49:29.2645705Z 4dce0aae7c66: Verifying Checksum 2023-11-14T04:49:29.2646591Z 4dce0aae7c66: Download complete 2023-11-14T04:49:29.3636052Z 93c6911b0287: Verifying Checksum 2023-11-14T04:49:29.3636810Z 93c6911b0287: Download complete 2023-11-14T04:49:29.4563826Z fcdaac0fbb69: Verifying Checksum 2023-11-14T04:49:29.4564446Z fcdaac0fbb69: Download complete 2023-11-14T04:49:29.6195436Z d80b8d0f01dc: Verifying Checksum 2023-11-14T04:49:29.6196290Z d80b8d0f01dc: Download complete 2023-11-14T04:49:29.6241814Z 3288913d5f12: Pull complete 2023-11-14T04:49:29.7086704Z 762352f3c28b: Verifying Checksum 2023-11-14T04:49:29.7087315Z 762352f3c28b: Download complete 2023-11-14T04:49:30.7886912Z afae2e58c215: Pull complete 2023-11-14T04:49:30.9030899Z 52a5515aef88: Pull complete 2023-11-14T04:49:31.0276745Z f39a9cff48d6: Verifying Checksum 2023-11-14T04:49:31.0277404Z f39a9cff48d6: Download complete 2023-11-14T04:49:31.0473526Z fc5398be2cb5: Pull complete 2023-11-14T04:49:31.1353701Z 0c4fbc80353d: Verifying Checksum 2023-11-14T04:49:31.1354253Z 0c4fbc80353d: Download complete 2023-11-14T04:49:49.8148098Z 9882884fae03: Verifying Checksum 2023-11-14T04:49:49.8148742Z 9882884fae03: Download complete 2023-11-14T04:49:49.9116283Z ea619973e9af: Download complete 2023-11-14T04:49:58.3659630Z c868322d4101: Download complete 2023-11-14T04:50:32.6659841Z bb1924fd13d8: Verifying Checksum 2023-11-14T04:50:32.6660420Z bb1924fd13d8: Download complete 2023-11-14T04:50:32.9772976Z c868322d4101: Pull complete 2023-11-14T04:50:34.8149030Z 77c5d5a121d5: Pull complete 2023-11-14T04:50:36.8697429Z 742ea107d559: Pull complete 2023-11-14T04:50:51.0091706Z ea54d9137d4e: Pull complete 2023-11-14T04:50:52.8880194Z 34d6327ee799: Pull complete 2023-11-14T04:50:54.7960546Z 2f5ab0288987: Pull complete 2023-11-14T04:50:56.6444562Z 5e730d5fd91a: Pull complete 2023-11-14T04:51:01.1931323Z 2020052b174f: Pull complete 2023-11-14T04:51:03.0696389Z 80f0c3756826: Pull complete 2023-11-14T04:51:04.9489796Z 5cb4c5d69233: Pull complete 2023-11-14T04:51:06.7935776Z 192c8af52128: Pull complete 2023-11-14T04:51:48.1830816Z fac5f79d8f40: Pull complete 2023-11-14T04:51:50.5244004Z ac2c08198bc7: Pull complete 2023-11-14T04:51:52.5350523Z 54c34a5c940f: Pull complete 2023-11-14T04:51:54.4098070Z c597c95fbef4: Pull complete 2023-11-14T04:51:56.2549742Z 3808c65d8c0f: Pull complete 2023-11-14T04:51:58.1000883Z c715fe586a58: Pull complete 2023-11-14T04:52:04.0317979Z 1f385a3986c3: Pull complete 2023-11-14T04:52:07.1085353Z d8ad661dea21: Pull complete 2023-11-14T04:52:10.0207722Z 7195438c44c1: Pull complete 2023-11-14T04:52:13.6655971Z 3c5a16afcc5f: Pull complete 2023-11-14T04:52:15.5132623Z 23ecc86c3805: Pull complete 2023-11-14T04:52:17.6223676Z 54f5b07b5ce6: Pull complete 2023-11-14T04:52:25.3054957Z e8400b0ad3ec: Pull complete 2023-11-14T04:52:25.4200890Z 7ef1ab3712ec: Pull complete 2023-11-14T04:52:25.5235866Z 51783a9a2a4a: Pull complete 2023-11-14T04:52:26.9785620Z 0778288efbc7: Pull complete 2023-11-14T04:52:27.1118862Z 8b7bba61649c: Pull complete 2023-11-14T04:52:27.2034009Z d913ee57fbe5: Pull complete 2023-11-14T04:52:27.5654189Z ad4cadf638a3: Pull complete 2023-11-14T04:52:27.6710931Z 88ffa4df8050: Pull complete 2023-11-14T04:52:27.8640890Z 7b66c65e46fc: Pull complete 2023-11-14T04:52:27.9803329Z ca20e2be2987: Pull complete 2023-11-14T04:52:28.0721968Z cb0f483351b2: Pull complete 2023-11-14T04:52:28.1703330Z 034bfa5626bf: Pull complete 2023-11-14T04:52:29.6023607Z c1310534828e: Pull complete 2023-11-14T04:52:29.7110383Z 2aa9bf51637b: Pull complete 2023-11-14T04:52:29.8231468Z f9bcb4185c0a: Pull complete 2023-11-14T04:52:30.0396535Z c70de30968b0: Pull complete 2023-11-14T04:52:30.1192826Z 1ad45855bb00: Pull complete 2023-11-14T04:52:51.1040770Z 9882884fae03: Pull complete 2023-11-14T04:52:51.2027152Z ef5e32cf3412: Pull complete 2023-11-14T04:52:51.3146774Z cdf3b334dcb1: Pull complete 2023-11-14T04:52:51.5703031Z ba019cce1ead: Pull complete 2023-11-14T04:52:51.6846350Z b59f1e2807d4: Pull complete 2023-11-14T04:52:52.4385946Z 4dce0aae7c66: Pull complete 2023-11-14T04:52:54.5339993Z 93c6911b0287: Pull complete 2023-11-14T04:52:57.2792236Z fcdaac0fbb69: Pull complete 2023-11-14T04:53:00.6129748Z d80b8d0f01dc: Pull complete 2023-11-14T04:53:02.9252034Z 762352f3c28b: Pull complete 2023-11-14T04:53:06.7440955Z f39a9cff48d6: Pull complete 2023-11-14T04:53:08.5878547Z 0c4fbc80353d: Pull complete 2023-11-14T04:53:48.0174470Z bb1924fd13d8: Pull complete 2023-11-14T04:53:49.8650816Z ea619973e9af: Pull complete 2023-11-14T04:53:51.1503370Z Digest: sha256:155cc70f81c46492d7369e444f8774cfbc3ff3ec6c7d0217deb03d9c42253389 2023-11-14T04:53:51.6517486Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:53:51.9337310Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:53:51.9544404Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2023-11-14T04:53:51.9545207Z with: 2023-11-14T04:53:51.9545651Z driver-version: 535.54.03 2023-11-14T04:53:51.9546143Z env: 2023-11-14T04:53:51.9546570Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:53:51.9547089Z ##[endgroup] 2023-11-14T04:53:52.0918513Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2023-11-14T04:53:52.0919356Z with: 2023-11-14T04:53:52.0919781Z timeout_minutes: 10 2023-11-14T04:53:52.0920251Z max_attempts: 3 2023-11-14T04:53:52.0959884Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2023-11-14T04:53:52.1000265Z retry_wait_seconds: 10 2023-11-14T04:53:52.1000812Z polling_interval_seconds: 1 2023-11-14T04:53:52.1001340Z warning_on_retry: true 2023-11-14T04:53:52.1001859Z continue_on_error: false 2023-11-14T04:53:52.1002368Z env: 2023-11-14T04:53:52.1002770Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:53:52.1003317Z DRIVER_VERSION: 535.54.03 2023-11-14T04:53:52.1003880Z ##[endgroup] 2023-11-14T04:53:52.1797917Z == Installing nvidia driver NVIDIA-Linux-x86_64-535.54.03.run == 2023-11-14T04:53:52.1799657Z + pre_install_nvidia_driver_amzn2 2023-11-14T04:53:52.1800430Z + sudo yum remove -y nvidia-driver-latest-dkms 2023-11-14T04:53:52.7249386Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2023-11-14T04:53:52.7875563Z No Match for argument: nvidia-driver-latest-dkms 2023-11-14T04:53:52.8288631Z No Packages marked for removal 2023-11-14T04:53:52.8477723Z + install_nvidia_driver_common 2023-11-14T04:53:52.8482220Z + echo 'Before installing NVIDIA driver' 2023-11-14T04:53:52.8483019Z + lspci 2023-11-14T04:53:52.8485152Z Before installing NVIDIA driver 2023-11-14T04:53:52.8695986Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2023-11-14T04:53:52.8697076Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2023-11-14T04:53:52.8698557Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2023-11-14T04:53:52.8699564Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2023-11-14T04:53:52.8700506Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2023-11-14T04:53:52.8701475Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2023-11-14T04:53:52.8702557Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2023-11-14T04:53:52.8703742Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2023-11-14T04:53:52.8704818Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2023-11-14T04:53:52.8705643Z + lsmod 2023-11-14T04:53:52.8721340Z Module Size Used by 2023-11-14T04:53:52.8721954Z ib_core 266240 0 2023-11-14T04:53:52.8722517Z nvidia_modeset 1306624 0 2023-11-14T04:53:52.8723071Z veth 16384 0 2023-11-14T04:53:52.8723598Z nvidia_uvm 1466368 0 2023-11-14T04:53:52.8724243Z nvidia 56422400 2 nvidia_uvm,nvidia_modeset 2023-11-14T04:53:52.8724942Z drm 425984 1 nvidia 2023-11-14T04:53:52.8725843Z i2c_core 77824 2 nvidia,drm 2023-11-14T04:53:52.8727087Z backlight 16384 1 nvidia_modeset 2023-11-14T04:53:52.8728476Z xt_conntrack 16384 1 2023-11-14T04:53:52.8729657Z ipt_MASQUERADE 16384 1 2023-11-14T04:53:52.8730484Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2023-11-14T04:53:52.8731169Z nf_conntrack_netlink 49152 0 2023-11-14T04:53:52.8731769Z nfnetlink 16384 2 nf_conntrack_netlink 2023-11-14T04:53:52.8732591Z xfrm_user 45056 1 2023-11-14T04:53:52.8733501Z xfrm_algo 16384 1 xfrm_user 2023-11-14T04:53:52.8734836Z iptable_nat 16384 1 2023-11-14T04:53:52.8735731Z nf_conntrack_ipv4 16384 3 2023-11-14T04:53:52.8736368Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2023-11-14T04:53:52.8737071Z nf_nat_ipv4 16384 1 iptable_nat 2023-11-14T04:53:52.8737797Z nf_nat 36864 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2023-11-14T04:53:52.8739417Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2023-11-14T04:53:52.8741910Z xt_addrtype 16384 2 2023-11-14T04:53:52.8742487Z iptable_filter 16384 1 2023-11-14T04:53:52.8743277Z br_netfilter 24576 0 2023-11-14T04:53:52.8744590Z bridge 172032 1 br_netfilter 2023-11-14T04:53:52.8745204Z stp 16384 1 bridge 2023-11-14T04:53:52.8745784Z llc 16384 2 bridge,stp 2023-11-14T04:53:52.8746391Z overlay 86016 0 2023-11-14T04:53:52.8746935Z sunrpc 393216 1 2023-11-14T04:53:52.8747456Z dm_mirror 28672 0 2023-11-14T04:53:52.8748038Z dm_region_hash 20480 1 dm_mirror 2023-11-14T04:53:52.8748731Z dm_log 20480 2 dm_region_hash,dm_mirror 2023-11-14T04:53:52.8749446Z dm_mod 143360 2 dm_log,dm_mirror 2023-11-14T04:53:52.8750087Z dax 69632 1 dm_mod 2023-11-14T04:53:52.8750671Z sb_edac 24576 0 2023-11-14T04:53:52.8751199Z crc32_pclmul 16384 0 2023-11-14T04:53:52.8751760Z ghash_clmulni_intel 16384 0 2023-11-14T04:53:52.8752317Z pcbc 16384 0 2023-11-14T04:53:52.8752838Z ata_piix 36864 0 2023-11-14T04:53:52.8753406Z libata 266240 1 ata_piix 2023-11-14T04:53:52.8754001Z aesni_intel 188416 0 2023-11-14T04:53:52.8754936Z aes_x86_64 20480 1 aesni_intel 2023-11-14T04:53:52.8755892Z pcc_cpufreq 16384 0 2023-11-14T04:53:52.8756494Z crypto_simd 16384 1 aesni_intel 2023-11-14T04:53:52.8757115Z mousedev 24576 0 2023-11-14T04:53:52.8757794Z glue_helper 16384 1 aesni_intel 2023-11-14T04:53:52.8758426Z scsi_mod 245760 1 libata 2023-11-14T04:53:52.8759657Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2023-11-14T04:53:52.8760378Z evdev 20480 3 2023-11-14T04:53:52.8760920Z psmouse 32768 0 2023-11-14T04:53:52.8761478Z button 16384 0 2023-11-14T04:53:52.8761995Z ena 114688 0 2023-11-14T04:53:52.8762551Z xen_blkfront 49152 2 2023-11-14T04:53:52.8763100Z crc32c_intel 24576 0 2023-11-14T04:53:52.8763621Z autofs4 49152 2 2023-11-14T04:53:52.8764160Z + modinfo nvidia 2023-11-14T04:53:52.8765077Z filename: /lib/modules/4.14.252-195.483.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2023-11-14T04:53:52.8766008Z firmware: nvidia/535.54.03/gsp_tu10x.bin 2023-11-14T04:53:52.8766700Z firmware: nvidia/535.54.03/gsp_ga10x.bin 2023-11-14T04:53:52.8767409Z alias: char-major-195-* 2023-11-14T04:53:52.8767947Z version: 535.54.03 2023-11-14T04:53:52.8768464Z supported: external 2023-11-14T04:53:52.8768984Z license: NVIDIA 2023-11-14T04:53:52.8769489Z srcversion: EA9C7EF32617E104C8240C4 2023-11-14T04:53:52.8770164Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2023-11-14T04:53:52.8771047Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2023-11-14T04:53:52.8771755Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2023-11-14T04:53:52.8772467Z depends: i2c-core,drm 2023-11-14T04:53:52.8773001Z retpoline: Y 2023-11-14T04:53:52.8799168Z name: nvidia 2023-11-14T04:53:52.8800116Z vermagic: 4.14.252-195.483.amzn2.x86_64 SMP mod_unload modversions 2023-11-14T04:53:52.8801055Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2023-11-14T04:53:52.8801939Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2023-11-14T04:53:52.8802786Z parm: NVreg_ResmanDebugLevel:int 2023-11-14T04:53:52.8803431Z parm: NVreg_RmLogonRC:int 2023-11-14T04:53:52.8804042Z parm: NVreg_ModifyDeviceFiles:int 2023-11-14T04:53:52.8804729Z parm: NVreg_DeviceFileUID:int 2023-11-14T04:53:52.8805374Z parm: NVreg_DeviceFileGID:int 2023-11-14T04:53:52.8805994Z parm: NVreg_DeviceFileMode:int 2023-11-14T04:53:52.8806797Z parm: NVreg_InitializeSystemMemoryAllocations:int 2023-11-14T04:53:52.8807572Z parm: NVreg_UsePageAttributeTable:int 2023-11-14T04:53:52.8808248Z parm: NVreg_EnablePCIeGen3:int 2023-11-14T04:53:52.8808885Z parm: NVreg_EnableMSI:int 2023-11-14T04:53:52.8809501Z parm: NVreg_TCEBypassMode:int 2023-11-14T04:53:52.8810144Z parm: NVreg_EnableStreamMemOPs:int 2023-11-14T04:53:52.8810906Z parm: NVreg_RestrictProfilingToAdminUsers:int 2023-11-14T04:53:52.8811737Z parm: NVreg_PreserveVideoMemoryAllocations:int 2023-11-14T04:53:52.8812513Z parm: NVreg_EnableS0ixPowerManagement:int 2023-11-14T04:53:52.8813379Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2023-11-14T04:53:52.8814241Z parm: NVreg_DynamicPowerManagement:int 2023-11-14T04:53:52.8815097Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2023-11-14T04:53:52.8815948Z parm: NVreg_EnableGpuFirmware:int 2023-11-14T04:53:52.8816673Z parm: NVreg_EnableGpuFirmwareLogs:int 2023-11-14T04:53:52.8817416Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2023-11-14T04:53:52.8818199Z parm: NVreg_EnableUserNUMAManagement:int 2023-11-14T04:53:52.8818917Z parm: NVreg_MemoryPoolSize:int 2023-11-14T04:53:52.8819646Z parm: NVreg_KMallocHeapMaxSize:int 2023-11-14T04:53:52.8820328Z parm: NVreg_VMallocHeapMaxSize:int 2023-11-14T04:53:52.8821020Z parm: NVreg_IgnoreMMIOCheck:int 2023-11-14T04:53:52.8821679Z parm: NVreg_NvLinkDisable:int 2023-11-14T04:53:52.8822383Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2023-11-14T04:53:52.8823140Z parm: NVreg_RegisterPCIDriver:int 2023-11-14T04:53:52.8823843Z parm: NVreg_EnableResizableBar:int 2023-11-14T04:53:52.8824771Z parm: NVreg_EnableDbgBreakpoint:int 2023-11-14T04:53:52.8825468Z parm: NVreg_RegistryDwords:charp 2023-11-14T04:53:52.8826199Z parm: NVreg_RegistryDwordsPerDevice:charp 2023-11-14T04:53:52.8826880Z parm: NVreg_RmMsg:charp 2023-11-14T04:53:52.8827495Z parm: NVreg_GpuBlacklist:charp 2023-11-14T04:53:52.8828178Z parm: NVreg_TemporaryFilePath:charp 2023-11-14T04:53:52.8828843Z parm: NVreg_ExcludedGpus:charp 2023-11-14T04:53:52.8829505Z parm: NVreg_DmaRemapPeerMmio:int 2023-11-14T04:53:52.8830203Z parm: NVreg_RmNvlinkBandwidth:charp 2023-11-14T04:53:52.8830855Z parm: rm_firmware_active:charp 2023-11-14T04:53:52.8831470Z + HAS_NVIDIA_DRIVER=0 2023-11-14T04:53:52.8832053Z ++ command -v nvidia-smi 2023-11-14T04:53:52.8832628Z + '[' -x /usr/bin/nvidia-smi ']' 2023-11-14T04:53:52.8833187Z + set +e 2023-11-14T04:53:52.8833919Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2023-11-14T04:53:57.3023074Z + INSTALLED_DRIVER_VERSION=535.54.03 2023-11-14T04:53:57.3023769Z + NVIDIA_SMI_STATUS=0 2023-11-14T04:53:57.3024450Z + '[' 0 -ne 0 ']' 2023-11-14T04:53:57.3025268Z + '[' 535.54.03 '!=' 535.54.03 ']' 2023-11-14T04:53:57.3025812Z + HAS_NVIDIA_DRIVER=1 2023-11-14T04:53:57.3026782Z + echo 'NVIDIA driver (535.54.03) has already been installed. Skipping NVIDIA driver installation' 2023-11-14T04:53:57.3027670Z + set -e 2023-11-14T04:53:57.3028074Z + '[' 1 -eq 0 ']' 2023-11-14T04:53:57.3028822Z NVIDIA driver (535.54.03) has already been installed. Skipping NVIDIA driver installation 2023-11-14T04:53:57.3029688Z + post_install_nvidia_driver_common 2023-11-14T04:53:57.3030209Z + sudo modprobe nvidia 2023-11-14T04:53:57.3186816Z + echo 'After installing NVIDIA driver' 2023-11-14T04:53:57.3187335Z + lspci 2023-11-14T04:53:57.3187720Z After installing NVIDIA driver 2023-11-14T04:53:57.3384889Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2023-11-14T04:53:57.3386265Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2023-11-14T04:53:57.3387289Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2023-11-14T04:53:57.3388270Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2023-11-14T04:53:57.3389104Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2023-11-14T04:53:57.3389958Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2023-11-14T04:53:57.3391015Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2023-11-14T04:53:57.3392097Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2023-11-14T04:53:57.3393122Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2023-11-14T04:53:57.3393869Z + lsmod 2023-11-14T04:53:57.3408901Z Module Size Used by 2023-11-14T04:53:57.3409631Z ib_core 266240 0 2023-11-14T04:53:57.3410465Z nvidia_modeset 1306624 0 2023-11-14T04:53:57.3411231Z veth 16384 0 2023-11-14T04:53:57.3411948Z nvidia_uvm 1466368 0 2023-11-14T04:53:57.3413066Z nvidia 56422400 2 nvidia_uvm,nvidia_modeset 2023-11-14T04:53:57.3414394Z drm 425984 1 nvidia 2023-11-14T04:53:57.3415192Z i2c_core 77824 2 nvidia,drm 2023-11-14T04:53:57.3415784Z backlight 16384 1 nvidia_modeset 2023-11-14T04:53:57.3416361Z xt_conntrack 16384 1 2023-11-14T04:53:57.3416852Z ipt_MASQUERADE 16384 1 2023-11-14T04:53:57.3417530Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2023-11-14T04:53:57.3418246Z nf_conntrack_netlink 49152 0 2023-11-14T04:53:57.3418861Z nfnetlink 16384 2 nf_conntrack_netlink 2023-11-14T04:53:57.3419461Z xfrm_user 45056 1 2023-11-14T04:53:57.3419969Z xfrm_algo 16384 1 xfrm_user 2023-11-14T04:53:57.3420764Z iptable_nat 16384 1 2023-11-14T04:53:57.3422110Z nf_conntrack_ipv4 16384 3 2023-11-14T04:53:57.3423174Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2023-11-14T04:53:57.3423810Z nf_nat_ipv4 16384 1 iptable_nat 2023-11-14T04:53:57.3424498Z nf_nat 36864 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2023-11-14T04:53:57.3425724Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2023-11-14T04:53:57.3426778Z xt_addrtype 16384 2 2023-11-14T04:53:57.3427251Z iptable_filter 16384 1 2023-11-14T04:53:57.3427751Z br_netfilter 24576 0 2023-11-14T04:53:57.3428286Z bridge 172032 1 br_netfilter 2023-11-14T04:53:57.3428828Z stp 16384 1 bridge 2023-11-14T04:53:57.3429368Z llc 16384 2 bridge,stp 2023-11-14T04:53:57.3429906Z overlay 86016 0 2023-11-14T04:53:57.3430360Z sunrpc 393216 1 2023-11-14T04:53:57.3430840Z dm_mirror 28672 0 2023-11-14T04:53:57.3431370Z dm_region_hash 20480 1 dm_mirror 2023-11-14T04:53:57.3432001Z dm_log 20480 2 dm_region_hash,dm_mirror 2023-11-14T04:53:57.3432792Z dm_mod 143360 2 dm_log,dm_mirror 2023-11-14T04:53:57.3433397Z dax 69632 1 dm_mod 2023-11-14T04:53:57.3433910Z sb_edac 24576 0 2023-11-14T04:53:57.3434364Z crc32_pclmul 16384 0 2023-11-14T04:53:57.3435093Z ghash_clmulni_intel 16384 0 2023-11-14T04:53:57.3435766Z pcbc 16384 0 2023-11-14T04:53:57.3436231Z ata_piix 36864 0 2023-11-14T04:53:57.3436734Z libata 266240 1 ata_piix 2023-11-14T04:53:57.3437389Z aesni_intel 188416 0 2023-11-14T04:53:57.3437891Z aes_x86_64 20480 1 aesni_intel 2023-11-14T04:53:57.3438443Z pcc_cpufreq 16384 0 2023-11-14T04:53:57.3438966Z crypto_simd 16384 1 aesni_intel 2023-11-14T04:53:57.3439506Z mousedev 24576 0 2023-11-14T04:53:57.3440042Z glue_helper 16384 1 aesni_intel 2023-11-14T04:53:57.3440600Z scsi_mod 245760 1 libata 2023-11-14T04:53:57.3441276Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2023-11-14T04:53:57.3441969Z evdev 20480 3 2023-11-14T04:53:57.3442456Z psmouse 32768 0 2023-11-14T04:53:57.3442916Z button 16384 0 2023-11-14T04:53:57.3443385Z ena 114688 0 2023-11-14T04:53:57.3443897Z xen_blkfront 49152 2 2023-11-14T04:53:57.3444357Z crc32c_intel 24576 0 2023-11-14T04:53:57.3444845Z autofs4 49152 2 2023-11-14T04:53:57.3445307Z + modinfo nvidia 2023-11-14T04:53:57.3446136Z filename: /lib/modules/4.14.252-195.483.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2023-11-14T04:53:57.3447028Z firmware: nvidia/535.54.03/gsp_tu10x.bin 2023-11-14T04:53:57.3447648Z firmware: nvidia/535.54.03/gsp_ga10x.bin 2023-11-14T04:53:57.3448268Z alias: char-major-195-* 2023-11-14T04:53:57.3448779Z version: 535.54.03 2023-11-14T04:53:57.3449230Z supported: external 2023-11-14T04:53:57.3449652Z license: NVIDIA 2023-11-14T04:53:57.3450120Z srcversion: EA9C7EF32617E104C8240C4 2023-11-14T04:53:57.3450727Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2023-11-14T04:53:57.3451345Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2023-11-14T04:53:57.3451973Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2023-11-14T04:53:57.3452616Z depends: i2c-core,drm 2023-11-14T04:53:57.3453063Z retpoline: Y 2023-11-14T04:53:57.3453465Z name: nvidia 2023-11-14T04:53:57.3454183Z vermagic: 4.14.252-195.483.amzn2.x86_64 SMP mod_unload modversions 2023-11-14T04:53:57.3455006Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2023-11-14T04:53:57.3455836Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2023-11-14T04:53:57.3456600Z parm: NVreg_ResmanDebugLevel:int 2023-11-14T04:53:57.3457368Z parm: NVreg_RmLogonRC:int 2023-11-14T04:53:57.3457906Z parm: NVreg_ModifyDeviceFiles:int 2023-11-14T04:53:57.3458499Z parm: NVreg_DeviceFileUID:int 2023-11-14T04:53:57.3459061Z parm: NVreg_DeviceFileGID:int 2023-11-14T04:53:57.3459609Z parm: NVreg_DeviceFileMode:int 2023-11-14T04:53:57.3460286Z parm: NVreg_InitializeSystemMemoryAllocations:int 2023-11-14T04:53:57.3461013Z parm: NVreg_UsePageAttributeTable:int 2023-11-14T04:53:57.3461614Z parm: NVreg_EnablePCIeGen3:int 2023-11-14T04:53:57.3462165Z parm: NVreg_EnableMSI:int 2023-11-14T04:53:57.3462704Z parm: NVreg_TCEBypassMode:int 2023-11-14T04:53:57.3463271Z parm: NVreg_EnableStreamMemOPs:int 2023-11-14T04:53:57.3463948Z parm: NVreg_RestrictProfilingToAdminUsers:int 2023-11-14T04:53:57.3464699Z parm: NVreg_PreserveVideoMemoryAllocations:int 2023-11-14T04:53:57.3465396Z parm: NVreg_EnableS0ixPowerManagement:int 2023-11-14T04:53:57.3466191Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2023-11-14T04:53:57.3466974Z parm: NVreg_DynamicPowerManagement:int 2023-11-14T04:53:57.3467869Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2023-11-14T04:53:57.3468666Z parm: NVreg_EnableGpuFirmware:int 2023-11-14T04:53:57.3469318Z parm: NVreg_EnableGpuFirmwareLogs:int 2023-11-14T04:53:57.3469988Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2023-11-14T04:53:57.3470706Z parm: NVreg_EnableUserNUMAManagement:int 2023-11-14T04:53:57.3471344Z parm: NVreg_MemoryPoolSize:int 2023-11-14T04:53:57.3471919Z parm: NVreg_KMallocHeapMaxSize:int 2023-11-14T04:53:57.3472537Z parm: NVreg_VMallocHeapMaxSize:int 2023-11-14T04:53:57.3473152Z parm: NVreg_IgnoreMMIOCheck:int 2023-11-14T04:53:57.3473708Z parm: NVreg_NvLinkDisable:int 2023-11-14T04:53:57.3474354Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2023-11-14T04:53:57.3475307Z parm: NVreg_RegisterPCIDriver:int 2023-11-14T04:53:57.3475912Z parm: NVreg_EnableResizableBar:int 2023-11-14T04:53:57.3476545Z parm: NVreg_EnableDbgBreakpoint:int 2023-11-14T04:53:57.3477177Z parm: NVreg_RegistryDwords:charp 2023-11-14T04:53:57.3477806Z parm: NVreg_RegistryDwordsPerDevice:charp 2023-11-14T04:53:57.3478431Z parm: NVreg_RmMsg:charp 2023-11-14T04:53:57.3478961Z parm: NVreg_GpuBlacklist:charp 2023-11-14T04:53:57.3479561Z parm: NVreg_TemporaryFilePath:charp 2023-11-14T04:53:57.3480144Z parm: NVreg_ExcludedGpus:charp 2023-11-14T04:53:57.3480733Z parm: NVreg_DmaRemapPeerMmio:int 2023-11-14T04:53:57.3481341Z parm: NVreg_RmNvlinkBandwidth:charp 2023-11-14T04:53:57.3481917Z parm: rm_firmware_active:charp 2023-11-14T04:53:57.3482436Z + set +e 2023-11-14T04:53:57.3482845Z + nvidia-smi 2023-11-14T04:53:59.0499091Z Tue Nov 14 04:53:59 2023 2023-11-14T04:53:59.0500200Z +---------------------------------------------------------------------------------------+ 2023-11-14T04:53:59.0501296Z | NVIDIA-SMI 535.54.03 Driver Version: 535.54.03 CUDA Version: 12.2 | 2023-11-14T04:53:59.0502335Z |-----------------------------------------+----------------------+----------------------+ 2023-11-14T04:53:59.0503365Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2023-11-14T04:53:59.0504491Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2023-11-14T04:53:59.0505346Z | | | MIG M. | 2023-11-14T04:53:59.0506058Z |=========================================+======================+======================| 2023-11-14T04:53:59.0769373Z | 0 Tesla M60 Off | 00000000:00:1D.0 Off | 0 | 2023-11-14T04:53:59.0770297Z | N/A 32C P0 38W / 150W | 0MiB / 7680MiB | 71% Default | 2023-11-14T04:53:59.0771542Z | | | N/A | 2023-11-14T04:53:59.0772438Z +-----------------------------------------+----------------------+----------------------+ 2023-11-14T04:53:59.0773291Z | 1 Tesla M60 Off | 00000000:00:1E.0 Off | 0 | 2023-11-14T04:53:59.0774181Z | N/A 26C P0 38W / 150W | 0MiB / 7680MiB | 5% Default | 2023-11-14T04:53:59.0774963Z | | | N/A | 2023-11-14T04:53:59.0775817Z +-----------------------------------------+----------------------+----------------------+ 2023-11-14T04:53:59.0776569Z 2023-11-14T04:53:59.0777374Z +---------------------------------------------------------------------------------------+ 2023-11-14T04:53:59.0778190Z | Processes: | 2023-11-14T04:53:59.0779105Z | GPU GI CI PID Type Process name GPU Memory | 2023-11-14T04:53:59.0780158Z | ID ID Usage | 2023-11-14T04:53:59.0780918Z |=======================================================================================| 2023-11-14T04:53:59.0781750Z | No running processes found | 2023-11-14T04:53:59.0782739Z +---------------------------------------------------------------------------------------+ 2023-11-14T04:54:00.0052663Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2023-11-14T04:54:01.6251386Z Tesla M60 2023-11-14T04:54:03.1301610Z + NVIDIA_SMI_STATUS=0 2023-11-14T04:54:03.1302424Z + '[' 0 -eq 0 ']' 2023-11-14T04:54:03.1302903Z + echo 'INFO: Ignoring allowed status 0' 2023-11-14T04:54:03.1303474Z + set -e 2023-11-14T04:54:03.1303918Z INFO: Ignoring allowed status 0 2023-11-14T04:54:03.1308533Z == Installing nvidia container toolkit for amzn2 == 2023-11-14T04:54:03.1312916Z + sudo yum install -y yum-utils 2023-11-14T04:54:03.6733750Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2023-11-14T04:54:05.3839532Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2023-11-14T04:54:05.3840453Z Nothing to do 2023-11-14T04:54:05.5690747Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2023-11-14T04:54:06.1317754Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2023-11-14T04:54:06.1617302Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2023-11-14T04:54:06.1618840Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2023-11-14T04:54:06.1620097Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2023-11-14T04:54:06.1788276Z + sudo yum install -y nvidia-docker2 2023-11-14T04:54:06.7206422Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2023-11-14T04:54:08.4494306Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2023-11-14T04:54:08.4495205Z Nothing to do 2023-11-14T04:54:08.6284805Z + sudo systemctl restart docker 2023-11-14T04:54:19.2075122Z Command completed after 1 attempt(s). 2023-11-14T04:54:19.2164257Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2023-11-14T04:54:19.2165260Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2023-11-14T04:54:19.2166160Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2023-11-14T04:54:19.2167020Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2023-11-14T04:54:19.2176863Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:54:19.2177488Z env: 2023-11-14T04:54:19.2177863Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:19.2178729Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:19.2179337Z ##[endgroup] 2023-11-14T04:54:19.5177052Z Defaulting to user installation because normal site-packages is not writeable 2023-11-14T04:54:19.5401409Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2023-11-14T04:54:19.5527916Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2023-11-14T04:54:19.6910094Z Prepare all required actions 2023-11-14T04:54:19.6910875Z Getting action download info 2023-11-14T04:54:19.8272748Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2023-11-14T04:54:20.0414130Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2023-11-14T04:54:20.1820877Z ##[group]Run ./.github/actions/download-build-artifacts 2023-11-14T04:54:20.1821489Z with: 2023-11-14T04:54:20.1821877Z name: linux-focal-cuda11.8-py3.10-gcc9 2023-11-14T04:54:20.1822382Z env: 2023-11-14T04:54:20.1822721Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:20.1823262Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:20.1823856Z ##[endgroup] 2023-11-14T04:54:20.1870535Z ##[group]Run seemethere/download-artifact-s3@v4 2023-11-14T04:54:20.1871092Z with: 2023-11-14T04:54:20.1871487Z name: linux-focal-cuda11.8-py3.10-gcc9 2023-11-14T04:54:20.1872070Z s3-bucket: gha-artifacts 2023-11-14T04:54:20.1872492Z region: us-east-1 2023-11-14T04:54:20.1872860Z env: 2023-11-14T04:54:20.1873182Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:20.1873712Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:20.1874302Z ##[endgroup] 2023-11-14T04:54:20.8584094Z (node:48459) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2023-11-14T04:54:20.8585050Z 2023-11-14T04:54:20.8585383Z Please migrate your code to use AWS SDK for JavaScript (v3). 2023-11-14T04:54:20.8586385Z For more information, check the migration guide at https://a.co/7PzMCcy 2023-11-14T04:54:20.8587472Z (Use `node --trace-warnings ...` to show where the warning was created) 2023-11-14T04:54:20.9923095Z Found 1 objects with prefix pytorch/pytorch/6859165800/linux-focal-cuda11.8-py3.10-gcc9/ 2023-11-14T04:54:20.9924417Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2023-11-14T04:54:28.1406399Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2023-11-14T04:54:28.1416685Z Artifact download has finished successfully 2023-11-14T04:54:28.1692915Z ##[group]Run unzip -o artifacts.zip 2023-11-14T04:54:28.1693494Z unzip -o artifacts.zip 2023-11-14T04:54:28.1702793Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:54:28.1703446Z env: 2023-11-14T04:54:28.1703819Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:28.1704437Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:28.1705075Z ##[endgroup] 2023-11-14T04:54:28.1750475Z Archive: artifacts.zip 2023-11-14T04:54:28.1752889Z creating: dist/ 2023-11-14T04:54:30.5333748Z inflating: dist/torch-2.2.0a0+gitfe705c0-cp310-cp310-linux_x86_64.whl 2023-11-14T04:54:30.5334565Z creating: build/custom_test_artifacts/ 2023-11-14T04:54:30.5335328Z creating: build/custom_test_artifacts/custom-op-build/ 2023-11-14T04:54:30.5336293Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2023-11-14T04:54:30.5337352Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2023-11-14T04:54:30.5344178Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2023-11-14T04:54:30.5345357Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2023-11-14T04:54:30.5346559Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2023-11-14T04:54:30.5347856Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2023-11-14T04:54:30.5349613Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2023-11-14T04:54:30.5351134Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2023-11-14T04:54:30.5352639Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2023-11-14T04:54:30.5354208Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2023-11-14T04:54:30.5356896Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2023-11-14T04:54:30.5358979Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2023-11-14T04:54:30.5360436Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2023-11-14T04:54:30.5361925Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2023-11-14T04:54:30.5363387Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2023-11-14T04:54:30.5364882Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2023-11-14T04:54:30.5366398Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2023-11-14T04:54:30.5367746Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2023-11-14T04:54:30.5369076Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2023-11-14T04:54:30.5423600Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2023-11-14T04:54:30.5425513Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2023-11-14T04:54:30.5427415Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2023-11-14T04:54:30.5429300Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2023-11-14T04:54:30.5431200Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2023-11-14T04:54:30.5432996Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2023-11-14T04:54:30.5435407Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2023-11-14T04:54:30.5437616Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2023-11-14T04:54:30.5439439Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2023-11-14T04:54:30.5481776Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2023-11-14T04:54:30.5532382Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2023-11-14T04:54:30.5534211Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2023-11-14T04:54:30.5535892Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2023-11-14T04:54:30.5537483Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2023-11-14T04:54:30.5539115Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2023-11-14T04:54:30.5540670Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2023-11-14T04:54:30.5542452Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2023-11-14T04:54:30.5544000Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2023-11-14T04:54:30.5616362Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2023-11-14T04:54:30.5693405Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2023-11-14T04:54:30.5694916Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2023-11-14T04:54:30.5696469Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2023-11-14T04:54:30.5697636Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2023-11-14T04:54:30.5698779Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2023-11-14T04:54:30.5699964Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2023-11-14T04:54:30.5701300Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2023-11-14T04:54:30.5702826Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2023-11-14T04:54:30.5704299Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2023-11-14T04:54:30.5705689Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2023-11-14T04:54:30.5707094Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2023-11-14T04:54:30.5708505Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2023-11-14T04:54:30.5709879Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2023-11-14T04:54:30.5711289Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2023-11-14T04:54:30.5712672Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2023-11-14T04:54:30.5727886Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2023-11-14T04:54:30.5859261Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2023-11-14T04:54:30.5860578Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2023-11-14T04:54:30.5862007Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2023-11-14T04:54:30.5863560Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2023-11-14T04:54:30.5865072Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2023-11-14T04:54:30.5866509Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2023-11-14T04:54:30.5868011Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2023-11-14T04:54:30.5869454Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2023-11-14T04:54:30.5870933Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2023-11-14T04:54:30.5872412Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2023-11-14T04:54:30.5873849Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2023-11-14T04:54:30.5892306Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2023-11-14T04:54:30.5993951Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2023-11-14T04:54:30.5996262Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2023-11-14T04:54:30.5997668Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2023-11-14T04:54:30.5998910Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2023-11-14T04:54:30.6000239Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2023-11-14T04:54:30.6001443Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2023-11-14T04:54:30.6002603Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2023-11-14T04:54:30.6003759Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2023-11-14T04:54:30.6004697Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2023-11-14T04:54:30.6005761Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2023-11-14T04:54:30.6112549Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2023-11-14T04:54:30.6189017Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2023-11-14T04:54:30.6190124Z creating: build/custom_test_artifacts/jit-hook-build/ 2023-11-14T04:54:30.6191004Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2023-11-14T04:54:30.6192089Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2023-11-14T04:54:30.6199188Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2023-11-14T04:54:30.6200683Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2023-11-14T04:54:30.6201894Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2023-11-14T04:54:30.6203162Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2023-11-14T04:54:30.6204416Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2023-11-14T04:54:30.6205826Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2023-11-14T04:54:30.6207224Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2023-11-14T04:54:30.6208514Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2023-11-14T04:54:30.6209795Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2023-11-14T04:54:30.6211266Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2023-11-14T04:54:30.6212775Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2023-11-14T04:54:30.6214233Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2023-11-14T04:54:30.6215683Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2023-11-14T04:54:30.6217186Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2023-11-14T04:54:30.6218670Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2023-11-14T04:54:30.6219973Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2023-11-14T04:54:30.6221314Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2023-11-14T04:54:30.6277503Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2023-11-14T04:54:30.6279332Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2023-11-14T04:54:30.6281177Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2023-11-14T04:54:30.6283288Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2023-11-14T04:54:30.6285180Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2023-11-14T04:54:30.6287089Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2023-11-14T04:54:30.6288894Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2023-11-14T04:54:30.6290699Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2023-11-14T04:54:30.6292484Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2023-11-14T04:54:30.6334365Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2023-11-14T04:54:30.6385119Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2023-11-14T04:54:30.6386896Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2023-11-14T04:54:30.6388561Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2023-11-14T04:54:30.6390176Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2023-11-14T04:54:30.6391754Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2023-11-14T04:54:30.6393322Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2023-11-14T04:54:30.6395538Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2023-11-14T04:54:30.6397713Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2023-11-14T04:54:30.6469280Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2023-11-14T04:54:30.6545923Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2023-11-14T04:54:30.6547523Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2023-11-14T04:54:30.6548737Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2023-11-14T04:54:30.6550065Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2023-11-14T04:54:30.6551310Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2023-11-14T04:54:30.6552474Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2023-11-14T04:54:30.6553937Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2023-11-14T04:54:30.6556335Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2023-11-14T04:54:30.6557843Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2023-11-14T04:54:30.6559235Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2023-11-14T04:54:30.6560686Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2023-11-14T04:54:30.6562121Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2023-11-14T04:54:30.6563549Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2023-11-14T04:54:30.6565186Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2023-11-14T04:54:30.6566615Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2023-11-14T04:54:30.6581088Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2023-11-14T04:54:30.6660590Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2023-11-14T04:54:30.6662154Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2023-11-14T04:54:30.6663544Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2023-11-14T04:54:30.6664801Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2023-11-14T04:54:30.6665985Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2023-11-14T04:54:30.6667184Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2023-11-14T04:54:30.6668552Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2023-11-14T04:54:30.6669690Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2023-11-14T04:54:30.6670696Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2023-11-14T04:54:30.6671779Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2023-11-14T04:54:30.6730904Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2023-11-14T04:54:30.6731801Z creating: build/custom_test_artifacts/custom-backend-build/ 2023-11-14T04:54:30.6732778Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2023-11-14T04:54:30.6733924Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2023-11-14T04:54:30.6741345Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2023-11-14T04:54:30.6742672Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2023-11-14T04:54:30.6743914Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2023-11-14T04:54:30.6745377Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2023-11-14T04:54:30.6746733Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2023-11-14T04:54:30.6748285Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2023-11-14T04:54:30.6749814Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2023-11-14T04:54:30.6751187Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2023-11-14T04:54:30.6752522Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2023-11-14T04:54:30.6754096Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2023-11-14T04:54:30.6756647Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2023-11-14T04:54:30.6758209Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2023-11-14T04:54:30.6759738Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2023-11-14T04:54:30.6761330Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2023-11-14T04:54:30.6762918Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2023-11-14T04:54:30.6764342Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2023-11-14T04:54:30.6765688Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2023-11-14T04:54:30.6819688Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2023-11-14T04:54:30.6821886Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2023-11-14T04:54:30.6824183Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2023-11-14T04:54:30.6826191Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2023-11-14T04:54:30.6828175Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2023-11-14T04:54:30.6830051Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2023-11-14T04:54:30.6831951Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2023-11-14T04:54:30.6833821Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2023-11-14T04:54:30.6836197Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2023-11-14T04:54:30.6877684Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2023-11-14T04:54:30.6928335Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2023-11-14T04:54:30.6930209Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2023-11-14T04:54:30.6931963Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2023-11-14T04:54:30.6933672Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2023-11-14T04:54:30.6935327Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2023-11-14T04:54:30.6936986Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2023-11-14T04:54:30.6938614Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2023-11-14T04:54:30.6940276Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2023-11-14T04:54:30.7012496Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2023-11-14T04:54:30.7089353Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2023-11-14T04:54:30.7091054Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2023-11-14T04:54:30.7092432Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2023-11-14T04:54:30.7093548Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2023-11-14T04:54:30.7120489Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2023-11-14T04:54:30.7121900Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2023-11-14T04:54:30.7123382Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2023-11-14T04:54:30.7125018Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2023-11-14T04:54:30.7126572Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2023-11-14T04:54:30.7128316Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2023-11-14T04:54:30.7129884Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2023-11-14T04:54:30.7131439Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2023-11-14T04:54:30.7133160Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2023-11-14T04:54:30.7134742Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2023-11-14T04:54:30.7136242Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2023-11-14T04:54:30.7137867Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2023-11-14T04:54:30.7258627Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2023-11-14T04:54:30.7260155Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2023-11-14T04:54:30.7261701Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2023-11-14T04:54:30.7263404Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2023-11-14T04:54:30.7264993Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2023-11-14T04:54:30.7266524Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2023-11-14T04:54:30.7268135Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2023-11-14T04:54:30.7269739Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2023-11-14T04:54:30.7271361Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2023-11-14T04:54:30.7272953Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2023-11-14T04:54:30.7274998Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2023-11-14T04:54:30.7291431Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2023-11-14T04:54:30.7358448Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2023-11-14T04:54:30.7360097Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2023-11-14T04:54:30.7361637Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2023-11-14T04:54:30.7362982Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2023-11-14T04:54:30.7364211Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2023-11-14T04:54:30.7365515Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2023-11-14T04:54:30.7366777Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2023-11-14T04:54:30.7367938Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2023-11-14T04:54:30.7368968Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2023-11-14T04:54:30.7370092Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2023-11-14T04:54:30.7493947Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2023-11-14T04:54:30.7546746Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2023-11-14T04:54:30.7547804Z creating: build/lib/ 2023-11-14T04:54:30.7548291Z inflating: build/lib/libclog.a 2023-11-14T04:54:30.7558821Z inflating: build/lib/libpthreadpool.a 2023-11-14T04:54:30.7640757Z inflating: build/lib/libgtest.a 2023-11-14T04:54:30.7752622Z inflating: build/lib/libprotobuf-lite.a 2023-11-14T04:54:30.7846921Z inflating: build/lib/libbenchmark.a 2023-11-14T04:54:30.7856644Z inflating: build/lib/libittnotify.a 2023-11-14T04:54:30.7891570Z inflating: build/lib/libtensorpipe_uv.a 2023-11-14T04:54:30.8466682Z inflating: build/lib/libprotobuf.a 2023-11-14T04:54:30.8547014Z inflating: build/lib/libasmjit.a 2023-11-14T04:54:30.8707465Z inflating: build/lib/libgloo.a 2023-11-14T04:54:30.8734348Z inflating: build/lib/libfmt.a 2023-11-14T04:54:30.8735233Z inflating: build/lib/libfoxi_loader.a 2023-11-14T04:54:30.8737660Z inflating: build/lib/libcaffe2_nvrtc.so 2023-11-14T04:54:30.8739108Z inflating: build/lib/libtorch_global_deps.so 2023-11-14T04:54:30.8749202Z inflating: build/lib/libcpuinfo.a 2023-11-14T04:54:30.8759336Z inflating: build/lib/libcpuinfo_internals.a 2023-11-14T04:54:30.9393269Z inflating: build/lib/libprotoc.a 2023-11-14T04:54:30.9410601Z inflating: build/lib/libqnnpack.a 2023-11-14T04:54:30.9434920Z inflating: build/lib/libpytorch_qnnpack.a 2023-11-14T04:54:30.9438263Z inflating: build/lib/libnnpack_reference_layers.a 2023-11-14T04:54:30.9460587Z inflating: build/lib/libnnpack.a 2023-11-14T04:54:30.9483377Z inflating: build/lib/libgmock.a 2023-11-14T04:54:30.9484445Z inflating: build/lib/libgtest_main.a 2023-11-14T04:54:30.9485141Z inflating: build/lib/libbenchmark_main.a 2023-11-14T04:54:31.0209437Z inflating: build/lib/libtensorpipe.a 2023-11-14T04:54:32.1498704Z inflating: build/lib/libdnnl.a 2023-11-14T04:54:32.1607904Z inflating: build/lib/libc10.so 2023-11-14T04:54:32.1608797Z inflating: build/lib/libgmock_main.a 2023-11-14T04:54:32.3193494Z inflating: build/lib/libfbgemm.a 2023-11-14T04:54:32.3822820Z inflating: build/lib/libkineto.a 2023-11-14T04:54:32.4145494Z inflating: build/lib/libtensorpipe_cuda.a 2023-11-14T04:54:32.4194484Z inflating: build/lib/libcaffe2_protos.a 2023-11-14T04:54:32.4261337Z inflating: build/lib/libc10_cuda.so 2023-11-14T04:54:32.4313963Z inflating: build/lib/libonnx_proto.a 2023-11-14T04:54:32.4460168Z inflating: build/lib/libXNNPACK.a 2023-11-14T04:54:32.4952783Z inflating: build/lib/libgloo_cuda.a 2023-11-14T04:54:32.5807393Z inflating: build/lib/libonnx.a 2023-11-14T04:54:35.4385526Z inflating: build/lib/libtorch_cpu.so 2023-11-14T04:54:35.4390902Z inflating: build/lib/libunbox_lib.a 2023-11-14T04:54:37.7156435Z inflating: build/lib/libtorch_cuda.so 2023-11-14T04:54:37.7157463Z inflating: build/lib/libtorch.so 2023-11-14T04:54:37.7160591Z inflating: build/lib/libc10d_cuda_test.so 2023-11-14T04:54:38.8665751Z inflating: build/lib/libtorch_cuda_linalg.so 2023-11-14T04:54:38.8690712Z inflating: build/lib/libjitbackend_test.so 2023-11-14T04:54:38.8756303Z inflating: build/lib/libtorchbind_test.so 2023-11-14T04:54:38.8787487Z inflating: build/lib/libbackend_with_compiler.so 2023-11-14T04:54:38.8792560Z inflating: build/lib/libshm.so 2023-11-14T04:54:39.1040113Z inflating: build/lib/libtorch_python.so 2023-11-14T04:54:39.1082938Z inflating: build/lib/libnnapi_backend.so 2023-11-14T04:54:39.1083549Z creating: build/bin/ 2023-11-14T04:54:39.1647425Z inflating: build/bin/protoc-3.13.0.0 2023-11-14T04:54:39.2210805Z inflating: build/bin/protoc 2023-11-14T04:54:39.2274290Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2023-11-14T04:54:39.2339346Z inflating: build/bin/c10_DeviceGuard_test 2023-11-14T04:54:39.2402920Z inflating: build/bin/c10_Device_test 2023-11-14T04:54:39.2477599Z inflating: build/bin/c10_DispatchKeySet_test 2023-11-14T04:54:39.2539237Z inflating: build/bin/c10_StreamGuard_test 2023-11-14T04:54:39.2607058Z inflating: build/bin/c10_SymInt_test 2023-11-14T04:54:39.2675829Z inflating: build/bin/c10_InlineDeviceGuard_test 2023-11-14T04:54:39.2745946Z inflating: build/bin/c10_InlineStreamGuard_test 2023-11-14T04:54:39.2817387Z inflating: build/bin/c10_SizesAndStrides_test 2023-11-14T04:54:39.2906452Z inflating: build/bin/c10_cow_test 2023-11-14T04:54:39.2973769Z inflating: build/bin/c10_Bitset_test 2023-11-14T04:54:39.3035399Z inflating: build/bin/c10_ConstexprCrc_test 2023-11-14T04:54:39.3098412Z inflating: build/bin/c10_DeadlockDetection_test 2023-11-14T04:54:39.3161409Z inflating: build/bin/c10_Half_test 2023-11-14T04:54:39.3232349Z inflating: build/bin/c10_LeftRight_test 2023-11-14T04:54:39.3301877Z inflating: build/bin/c10_Metaprogramming_test 2023-11-14T04:54:39.3365333Z inflating: build/bin/c10_Synchronized_test 2023-11-14T04:54:39.3435534Z inflating: build/bin/c10_ThreadLocal_test 2023-11-14T04:54:39.3500722Z inflating: build/bin/c10_TypeIndex_test 2023-11-14T04:54:39.3565504Z inflating: build/bin/c10_TypeList_test 2023-11-14T04:54:39.3627165Z inflating: build/bin/c10_TypeTraits_test 2023-11-14T04:54:39.3692783Z inflating: build/bin/c10_accumulate_test 2023-11-14T04:54:39.3762924Z inflating: build/bin/c10_bfloat16_test 2023-11-14T04:54:39.3826671Z inflating: build/bin/c10_bit_cast_test 2023-11-14T04:54:39.3897981Z inflating: build/bin/c10_complex_math_test 2023-11-14T04:54:39.3968373Z inflating: build/bin/c10_complex_test 2023-11-14T04:54:39.4108027Z inflating: build/bin/c10_either_test 2023-11-14T04:54:39.4174893Z inflating: build/bin/c10_exception_test 2023-11-14T04:54:39.4238618Z inflating: build/bin/c10_flags_test 2023-11-14T04:54:39.4302749Z inflating: build/bin/c10_irange_test 2023-11-14T04:54:39.4511886Z inflating: build/bin/c10_intrusive_ptr_test 2023-11-14T04:54:39.4582442Z inflating: build/bin/c10_logging_test 2023-11-14T04:54:39.4677151Z inflating: build/bin/c10_optional_test 2023-11-14T04:54:39.4745102Z inflating: build/bin/c10_registry_test 2023-11-14T04:54:39.4824087Z inflating: build/bin/c10_ordered_preserving_dict_test 2023-11-14T04:54:39.5015702Z inflating: build/bin/c10_small_vector_test 2023-11-14T04:54:39.5080971Z inflating: build/bin/c10_ssize_test 2023-11-14T04:54:39.5144449Z inflating: build/bin/c10_tempfile_test 2023-11-14T04:54:39.5218659Z inflating: build/bin/c10_string_view_test 2023-11-14T04:54:39.5289574Z inflating: build/bin/c10_typeid_test 2023-11-14T04:54:39.5350733Z inflating: build/bin/c10_intrusive_ptr_benchmark 2023-11-14T04:54:39.5417430Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2023-11-14T04:54:39.5484473Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2023-11-14T04:54:39.5551053Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2023-11-14T04:54:39.5617920Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2023-11-14T04:54:39.5680131Z inflating: build/bin/c10_cuda_CUDATest 2023-11-14T04:54:39.5746979Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2023-11-14T04:54:39.5813923Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2023-11-14T04:54:39.5880451Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2023-11-14T04:54:39.6259326Z inflating: build/bin/vec_test_all_types_DEFAULT 2023-11-14T04:54:39.6657701Z inflating: build/bin/vec_test_all_types_AVX512 2023-11-14T04:54:39.7075401Z inflating: build/bin/vec_test_all_types_AVX2 2023-11-14T04:54:39.7142849Z inflating: build/bin/HashStoreTest 2023-11-14T04:54:39.7212447Z inflating: build/bin/TCPStoreTest 2023-11-14T04:54:39.7279563Z inflating: build/bin/FileStoreTest 2023-11-14T04:54:39.7296713Z inflating: build/bin/ProcessGroupMPITest 2023-11-14T04:54:39.7365005Z inflating: build/bin/test_edge_op_registration 2023-11-14T04:54:39.7368484Z inflating: build/bin/example_allreduce 2023-11-14T04:54:39.7433952Z inflating: build/bin/Dimname_test 2023-11-14T04:54:39.7526924Z inflating: build/bin/Dict_test 2023-11-14T04:54:39.7608352Z inflating: build/bin/MaybeOwned_test 2023-11-14T04:54:39.7680953Z inflating: build/bin/NamedTensor_test 2023-11-14T04:54:39.7755919Z inflating: build/bin/apply_utils_test 2023-11-14T04:54:39.7829723Z inflating: build/bin/atest 2023-11-14T04:54:39.7899143Z inflating: build/bin/broadcast_test 2023-11-14T04:54:39.7978623Z inflating: build/bin/basic 2023-11-14T04:54:39.8045864Z inflating: build/bin/cpu_profiling_allocator_test 2023-11-14T04:54:39.8119451Z inflating: build/bin/cpu_generator_test 2023-11-14T04:54:39.8182828Z inflating: build/bin/dispatch_key_set_test 2023-11-14T04:54:39.8300201Z inflating: build/bin/cpu_rng_test 2023-11-14T04:54:39.8364060Z inflating: build/bin/dlconvertor_test 2023-11-14T04:54:39.8436960Z inflating: build/bin/extension_backend_test 2023-11-14T04:54:39.8506831Z inflating: build/bin/half_test 2023-11-14T04:54:39.8625809Z inflating: build/bin/ivalue_test 2023-11-14T04:54:39.8695036Z inflating: build/bin/math_kernel_test 2023-11-14T04:54:39.8758161Z inflating: build/bin/lazy_tensor_test 2023-11-14T04:54:39.8826064Z inflating: build/bin/memory_format_test 2023-11-14T04:54:39.8893172Z inflating: build/bin/memory_overlapping_test 2023-11-14T04:54:39.8957191Z inflating: build/bin/operator_name_test 2023-11-14T04:54:39.9024154Z inflating: build/bin/mobile_memory_cleanup 2023-11-14T04:54:39.9094866Z inflating: build/bin/native_test 2023-11-14T04:54:39.9160178Z inflating: build/bin/operators_test 2023-11-14T04:54:39.9225565Z inflating: build/bin/packedtensoraccessor_test 2023-11-14T04:54:39.9309229Z inflating: build/bin/pow_test 2023-11-14T04:54:39.9381042Z inflating: build/bin/quantized_test 2023-11-14T04:54:39.9445891Z inflating: build/bin/reportMemoryUsage_test 2023-11-14T04:54:39.9508616Z inflating: build/bin/reduce_ops_test 2023-11-14T04:54:39.9581862Z inflating: build/bin/scalar_test 2023-11-14T04:54:39.9653820Z inflating: build/bin/scalar_tensor_test 2023-11-14T04:54:39.9719352Z inflating: build/bin/StorageUtils_test 2023-11-14T04:54:39.9789673Z inflating: build/bin/type_ptr_test 2023-11-14T04:54:39.9856262Z inflating: build/bin/stride_properties_test 2023-11-14T04:54:39.9954814Z inflating: build/bin/tensor_iterator_test 2023-11-14T04:54:39.9958525Z inflating: build/bin/thread_init_test 2023-11-14T04:54:40.0026756Z inflating: build/bin/test_parallel 2023-11-14T04:54:40.0103813Z inflating: build/bin/type_test 2023-11-14T04:54:40.0105539Z inflating: build/bin/verify_api_visibility 2023-11-14T04:54:40.0171915Z inflating: build/bin/undefined_tensor_test 2023-11-14T04:54:40.0259668Z inflating: build/bin/legacy_vmap_test 2023-11-14T04:54:40.0324938Z inflating: build/bin/weakref_test 2023-11-14T04:54:40.0390039Z inflating: build/bin/wrapdim_test 2023-11-14T04:54:40.0525199Z inflating: build/bin/List_test 2023-11-14T04:54:40.0587685Z inflating: build/bin/xla_tensor_test 2023-11-14T04:54:40.0663552Z inflating: build/bin/IListRef_test 2023-11-14T04:54:40.0817317Z inflating: build/bin/kernel_function_legacy_test 2023-11-14T04:54:40.0901080Z inflating: build/bin/KernelFunction_test 2023-11-14T04:54:40.1023477Z inflating: build/bin/kernel_function_test 2023-11-14T04:54:40.1185250Z inflating: build/bin/kernel_lambda_legacy_test 2023-11-14T04:54:40.1263017Z inflating: build/bin/kernel_stackbased_test 2023-11-14T04:54:40.1393835Z inflating: build/bin/kernel_lambda_test 2023-11-14T04:54:40.1458663Z inflating: build/bin/CppSignature_test 2023-11-14T04:54:40.1580951Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2023-11-14T04:54:40.1642719Z inflating: build/bin/op_allowlist_test 2023-11-14T04:54:40.1713073Z inflating: build/bin/backend_fallback_test 2023-11-14T04:54:40.1793355Z inflating: build/bin/inline_container_test 2023-11-14T04:54:40.2176369Z inflating: build/bin/op_registration_test 2023-11-14T04:54:40.2243491Z inflating: build/bin/cuda_apply_test 2023-11-14T04:54:40.2318104Z inflating: build/bin/cuda_atomic_ops_test 2023-11-14T04:54:40.2386934Z inflating: build/bin/cuda_caching_host_allocator_test 2023-11-14T04:54:40.2449716Z inflating: build/bin/cuda_device_test 2023-11-14T04:54:40.2538395Z inflating: build/bin/cuda_complex_math_test 2023-11-14T04:54:40.2613097Z inflating: build/bin/cuda_complex_test 2023-11-14T04:54:40.2686494Z inflating: build/bin/cuda_cub_test 2023-11-14T04:54:40.2750633Z inflating: build/bin/cuda_dlconvertor_test 2023-11-14T04:54:40.2815184Z inflating: build/bin/cuda_integer_divider_test 2023-11-14T04:54:40.2897356Z inflating: build/bin/cuda_distributions_test 2023-11-14T04:54:40.2960229Z inflating: build/bin/cuda_half_test 2023-11-14T04:54:40.3032042Z inflating: build/bin/cuda_generator_test 2023-11-14T04:54:40.3094405Z inflating: build/bin/cuda_optional_test 2023-11-14T04:54:40.3158151Z inflating: build/bin/cuda_allocatorTraceTracker_test 2023-11-14T04:54:40.3224798Z inflating: build/bin/cuda_reportMemoryUsage_test 2023-11-14T04:54:40.3300995Z inflating: build/bin/cuda_stream_test 2023-11-14T04:54:40.3366547Z inflating: build/bin/cuda_packedtensoraccessor_test 2023-11-14T04:54:40.3428621Z inflating: build/bin/cuda_cudnn_test 2023-11-14T04:54:40.3494389Z inflating: build/bin/cuda_vectorized_test 2023-11-14T04:54:40.3513400Z inflating: build/bin/tutorial_tensorexpr 2023-11-14T04:54:40.3596429Z inflating: build/bin/ProcessGroupGlooTest 2023-11-14T04:54:40.3668694Z inflating: build/bin/ProcessGroupGlooAsyncTest 2023-11-14T04:54:40.3747934Z inflating: build/bin/ProcessGroupNCCLTest 2023-11-14T04:54:40.3822944Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2023-11-14T04:54:40.3889476Z inflating: build/bin/ProcessGroupUCCTest 2023-11-14T04:54:40.3958957Z inflating: build/bin/test_dist_autograd 2023-11-14T04:54:40.4047287Z inflating: build/bin/test_cpp_rpc 2023-11-14T04:54:40.4050264Z inflating: build/bin/parallel_benchmark 2023-11-14T04:54:40.4061983Z inflating: build/bin/aot_model_compiler_test 2023-11-14T04:54:40.4145998Z inflating: build/bin/test_mobile_nnc 2023-11-14T04:54:40.4584543Z inflating: build/bin/test_lazy 2023-11-14T04:54:40.5634520Z inflating: build/bin/test_tensorexpr 2023-11-14T04:54:40.5640110Z inflating: build/bin/torch_shm_manager 2023-11-14T04:54:40.7111342Z inflating: build/bin/test_api 2023-11-14T04:54:40.7820347Z inflating: build/bin/test_jit 2023-11-14T04:54:40.7820984Z creating: .additional_ci_files/ 2023-11-14T04:54:40.7855983Z inflating: .additional_ci_files/test-times.json 2023-11-14T04:54:40.7995620Z inflating: .additional_ci_files/test-class-times.json 2023-11-14T04:54:40.8892703Z inflating: .additional_ci_files/test-file-ratings.json 2023-11-14T04:54:41.0511283Z inflating: .additional_ci_files/test-class-ratings.json 2023-11-14T04:54:41.0716005Z inflating: .additional_ci_files/td_heuristic_historical_edited_files.json 2023-11-14T04:54:41.0809487Z inflating: .additional_ci_files/td_heuristic_profiling.json 2023-11-14T04:54:41.0853729Z ##[group]Run df -H 2023-11-14T04:54:41.0854162Z df -H 2023-11-14T04:54:41.0863340Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:54:41.0863967Z env: 2023-11-14T04:54:41.0864334Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:41.0864923Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:41.0865541Z ##[endgroup] 2023-11-14T04:54:41.0905794Z Filesystem Size Used Avail Use% Mounted on 2023-11-14T04:54:41.0906442Z devtmpfs 129G 0 129G 0% /dev 2023-11-14T04:54:41.0907019Z tmpfs 129G 24M 129G 1% /dev/shm 2023-11-14T04:54:41.0908150Z tmpfs 129G 553k 129G 1% /run 2023-11-14T04:54:41.0908739Z tmpfs 129G 0 129G 0% /sys/fs/cgroup 2023-11-14T04:54:41.0909346Z /dev/xvda1 162G 39G 123G 25% / 2023-11-14T04:54:41.0948868Z ##[group]Run .github/scripts/parse_ref.py 2023-11-14T04:54:41.0949530Z .github/scripts/parse_ref.py 2023-11-14T04:54:41.0957682Z shell: /usr/bin/bash -e {0} 2023-11-14T04:54:41.0958123Z env: 2023-11-14T04:54:41.0958480Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:41.0959074Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:41.0959672Z ##[endgroup] 2023-11-14T04:54:41.1280357Z Prepare all required actions 2023-11-14T04:54:41.1332089Z ##[group]Run ./.github/actions/get-workflow-job-id 2023-11-14T04:54:41.1332672Z with: 2023-11-14T04:54:41.1333430Z github-token: *** 2023-11-14T04:54:41.1333835Z env: 2023-11-14T04:54:41.1334160Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:41.1334740Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:41.1335375Z ##[endgroup] 2023-11-14T04:54:41.1360193Z ##[group]Run set -eux 2023-11-14T04:54:41.1360641Z set -eux 2023-11-14T04:54:41.1361400Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2023-11-14T04:54:41.1369858Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:54:41.1370473Z env: 2023-11-14T04:54:41.1370835Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:41.1371416Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:41.1372241Z GITHUB_TOKEN: *** 2023-11-14T04:54:41.1372624Z ##[endgroup] 2023-11-14T04:54:41.1400370Z + python3 .github/scripts/get_workflow_job_id.py 6859165800 i-0bfddc9f582173679 2023-11-14T04:54:44.3069577Z setting job-id=18651430132 2023-11-14T04:54:44.3070634Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2023-11-14T04:54:44.3309848Z Prepare all required actions 2023-11-14T04:54:44.3310475Z Getting action download info 2023-11-14T04:54:44.4649822Z ##[group]Run ./.github/actions/filter-test-configs 2023-11-14T04:54:44.4650386Z with: 2023-11-14T04:54:44.4651000Z github-token: *** 2023-11-14T04:54:44.4652831Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2023-11-14T04:54:44.4655175Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2023-11-14T04:54:44.4656097Z env: 2023-11-14T04:54:44.4656427Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:44.4657006Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:44.4657619Z ##[endgroup] 2023-11-14T04:54:44.4717529Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2023-11-14T04:54:44.4718214Z with: 2023-11-14T04:54:44.4718574Z shell: bash 2023-11-14T04:54:44.4718951Z timeout_minutes: 10 2023-11-14T04:54:44.4719342Z max_attempts: 5 2023-11-14T04:54:44.4719740Z retry_wait_seconds: 30 2023-11-14T04:54:44.4720392Z command: set -eux python3 -m pip install requests==2.26.0 pyyaml==6.0 2023-11-14T04:54:44.4721172Z polling_interval_seconds: 1 2023-11-14T04:54:44.4721652Z warning_on_retry: true 2023-11-14T04:54:44.4722098Z continue_on_error: false 2023-11-14T04:54:44.4722497Z env: 2023-11-14T04:54:44.4722842Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:44.4723407Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:44.4724235Z GITHUB_TOKEN: *** 2023-11-14T04:54:44.4724611Z ##[endgroup] 2023-11-14T04:54:44.5456111Z + python3 -m pip install requests==2.26.0 pyyaml==6.0 2023-11-14T04:54:44.8354482Z Defaulting to user installation because normal site-packages is not writeable 2023-11-14T04:54:44.8573818Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0) 2023-11-14T04:54:44.8757380Z Requirement already satisfied: pyyaml==6.0 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0) 2023-11-14T04:54:44.8769133Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12) 2023-11-14T04:54:44.8795684Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.4) 2023-11-14T04:54:44.8811128Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.18) 2023-11-14T04:54:44.9053921Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2023.7.22) 2023-11-14T04:54:45.5450360Z Command completed after 1 attempt(s). 2023-11-14T04:54:45.5509542Z ##[group]Run set -x 2023-11-14T04:54:45.5510047Z set -x 2023-11-14T04:54:45.5510617Z  2023-11-14T04:54:45.5511440Z # Use relative path here as this could be checked out anywhere, not necessarily 2023-11-14T04:54:45.5512378Z # in runner workspace 2023-11-14T04:54:45.5513227Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2023-11-14T04:54:45.5524360Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:54:45.5525223Z env: 2023-11-14T04:54:45.5525661Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:45.5526349Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:45.5527145Z ##[endgroup] 2023-11-14T04:54:45.5580953Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2023-11-14T04:54:45.5886627Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2023-11-14T04:54:45.5887510Z echo "Workflow: ${GITHUB_WORKFLOW}" 2023-11-14T04:54:45.5888144Z echo "Job name: ${JOB_NAME}" 2023-11-14T04:54:45.5888657Z  2023-11-14T04:54:45.5889354Z # Use relative path here as this could be checked out anywhere, not necessarily 2023-11-14T04:54:45.5890224Z # in runner workspace 2023-11-14T04:54:45.5890943Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2023-11-14T04:54:45.5891773Z  --workflow "${GITHUB_WORKFLOW}" \ 2023-11-14T04:54:45.5892370Z  --job-name "${JOB_NAME}" \ 2023-11-14T04:54:45.5894380Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" \ 2023-11-14T04:54:45.5896390Z  --pr-number "${PR_NUMBER}" \ 2023-11-14T04:54:45.5896950Z  --tag "${TAG}" \ 2023-11-14T04:54:45.5897456Z  --event-name "${EVENT_NAME}" \ 2023-11-14T04:54:45.5898023Z  --schedule "${SCHEDULE}" \ 2023-11-14T04:54:45.5898574Z  --branch "${HEAD_BRANCH}" 2023-11-14T04:54:45.5906827Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:54:45.5907474Z env: 2023-11-14T04:54:45.5907839Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:45.5908409Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:45.5909301Z GITHUB_TOKEN: *** 2023-11-14T04:54:45.5910151Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2023-11-14T04:54:45.5911092Z PR_NUMBER: 112098 2023-11-14T04:54:45.5911467Z TAG: 2023-11-14T04:54:45.5911835Z EVENT_NAME: pull_request 2023-11-14T04:54:45.5912284Z SCHEDULE: 2023-11-14T04:54:45.5912640Z HEAD_BRANCH: 2023-11-14T04:54:45.5913026Z ##[endgroup] 2023-11-14T04:54:45.5941172Z Workflow: pull 2023-11-14T04:54:45.5942125Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2023-11-14T04:54:45.9183289Z fatal: unknown commit main 2023-11-14T04:54:45.9209708Z /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/filter_test_configs.py:472: UserWarning: failed to get commit messages: Command '['git', 'cherry', '-v', 'main']' returned non-zero exit status 128. 2023-11-14T04:54:45.9211725Z warnings.warn(f"failed to get commit messages: {e}") 2023-11-14T04:54:45.9345355Z ##[group]Run echo "Filtered matrix:" 2023-11-14T04:54:45.9345937Z echo "Filtered matrix:" 2023-11-14T04:54:45.9347871Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" 2023-11-14T04:54:45.9349764Z  2023-11-14T04:54:45.9350143Z echo 2023-11-14T04:54:45.9350640Z echo "Is the current job unstable? False" 2023-11-14T04:54:45.9351220Z  2023-11-14T04:54:45.9351569Z echo 2023-11-14T04:54:45.9352024Z echo "Is keep-going label set? True" 2023-11-14T04:54:45.9352556Z  2023-11-14T04:54:45.9352908Z echo 2023-11-14T04:54:45.9353314Z echo "Renabled issues? " 2023-11-14T04:54:45.9361943Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:54:45.9362572Z env: 2023-11-14T04:54:45.9362939Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:45.9363523Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:45.9364126Z ##[endgroup] 2023-11-14T04:54:45.9393617Z Filtered matrix: 2023-11-14T04:54:45.9397105Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}]} 2023-11-14T04:54:45.9398940Z 2023-11-14T04:54:45.9399150Z Is the current job unstable? False 2023-11-14T04:54:45.9399537Z 2023-11-14T04:54:45.9399807Z Is keep-going label set? True 2023-11-14T04:54:45.9400159Z 2023-11-14T04:54:45.9400322Z Renabled issues? 2023-11-14T04:54:45.9461665Z Prepare all required actions 2023-11-14T04:54:45.9462251Z Getting action download info 2023-11-14T04:54:46.0684868Z ##[group]Run ./.github/actions/pytest-cache-download 2023-11-14T04:54:46.0685459Z with: 2023-11-14T04:54:46.0685826Z cache_dir: .pytest_cache 2023-11-14T04:54:46.0686517Z job_identifier: pull_linux-focal-cuda11.8-py3.10-gcc9_test_distributed 2023-11-14T04:54:46.0687220Z env: 2023-11-14T04:54:46.0687576Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:46.0688161Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:46.0688757Z ##[endgroup] 2023-11-14T04:54:46.0744101Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2023-11-14T04:54:46.0744818Z with: 2023-11-14T04:54:46.0745168Z shell: bash 2023-11-14T04:54:46.0745544Z timeout_minutes: 5 2023-11-14T04:54:46.0745961Z max_attempts: 5 2023-11-14T04:54:46.0746370Z retry_wait_seconds: 30 2023-11-14T04:54:46.0746997Z command: set -eu python3 -m pip install boto3==1.19.12 2023-11-14T04:54:46.0747630Z polling_interval_seconds: 1 2023-11-14T04:54:46.0748108Z warning_on_retry: true 2023-11-14T04:54:46.0748560Z continue_on_error: false 2023-11-14T04:54:46.0748967Z env: 2023-11-14T04:54:46.0749317Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:46.0749890Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:46.0750481Z ##[endgroup] 2023-11-14T04:54:46.4381145Z Defaulting to user installation because normal site-packages is not writeable 2023-11-14T04:54:46.4586114Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12) 2023-11-14T04:54:46.4656126Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12) 2023-11-14T04:54:46.4718477Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0) 2023-11-14T04:54:46.4734952Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2) 2023-11-14T04:54:46.4770338Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2) 2023-11-14T04:54:46.4804782Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.18) 2023-11-14T04:54:46.5045707Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0) 2023-11-14T04:54:47.1496699Z Command completed after 1 attempt(s). 2023-11-14T04:54:47.1557834Z ##[group]Run python3 .github/scripts/pytest_cache.py \ 2023-11-14T04:54:47.1558609Z python3 .github/scripts/pytest_cache.py \ 2023-11-14T04:54:47.1559228Z  --download \ 2023-11-14T04:54:47.1559803Z  --cache_dir $GITHUB_WORKSPACE/$CACHE_DIR \ 2023-11-14T04:54:47.1560443Z  --pr_identifier $GITHUB_REF \ 2023-11-14T04:54:47.1561064Z  --job_identifier $JOB_IDENTIFIER \ 2023-11-14T04:54:47.1561662Z  --temp_dir $RUNNER_TEMP \ 2023-11-14T04:54:47.1562163Z  --repo $REPO \ 2023-11-14T04:54:47.1571269Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:54:47.1571931Z env: 2023-11-14T04:54:47.1572304Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:47.1572871Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:47.1573524Z CACHE_DIR: .pytest_cache 2023-11-14T04:54:47.1574214Z JOB_IDENTIFIER: pull_linux-focal-cuda11.8-py3.10-gcc9_test_distributed 2023-11-14T04:54:47.1574983Z REPO: pytorch/pytorch 2023-11-14T04:54:47.1575411Z ##[endgroup] 2023-11-14T04:54:47.6387722Z PR identifier for `refs/pull/112098/merge` is `2e5132e7254b4b60102d80422aad461b` 2023-11-14T04:54:47.6391803Z Downloading cache with args Namespace(bucket=None, cache_dir='/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache', download=True, job_identifier='pull_linux-focal-cuda11.8-py3.10-gcc9_test_distributed', pr_identifier='refs/pull/112098/merge', repo='pytorch/pytorch', shard=None, temp_dir='/home/ec2-user/actions-runner/_work/_temp', upload=False) 2023-11-14T04:54:47.6395816Z Downloading s3://gha-artifacts/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/1.zip 2023-11-14T04:54:47.6399318Z to /home/ec2-user/actions-runner/_work/_temp/cache-zip-downloads/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/1.zip 2023-11-14T04:54:47.6402687Z Downloading s3://gha-artifacts/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/2.zip 2023-11-14T04:54:47.6406116Z to /home/ec2-user/actions-runner/_work/_temp/cache-zip-downloads/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/2.zip 2023-11-14T04:54:47.6410629Z Unzipping /home/ec2-user/actions-runner/_work/_temp/cache-zip-downloads/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/1.zip 2023-11-14T04:54:47.6414308Z to /home/ec2-user/actions-runner/_work/_temp/unzipped-caches/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/1/.pytest_cache 2023-11-14T04:54:47.6417384Z Merging cache for job_identifier `pull_linux-focal-cuda11.8-py3.10-gcc9_test_distributed`, shard `1` into `/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache` 2023-11-14T04:54:47.6421198Z Unzipping /home/ec2-user/actions-runner/_work/_temp/cache-zip-downloads/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/2.zip 2023-11-14T04:54:47.6424879Z to /home/ec2-user/actions-runner/_work/_temp/unzipped-caches/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/2/.pytest_cache 2023-11-14T04:54:47.6427529Z Merging cache for job_identifier `pull_linux-focal-cuda11.8-py3.10-gcc9_test_distributed`, shard `2` into `/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache` 2023-11-14T04:54:47.6859996Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2023-11-14T04:54:47.6860954Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2023-11-14T04:54:47.6870279Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T04:54:47.6870927Z env: 2023-11-14T04:54:47.6871266Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:47.6871845Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:47.6872476Z JOB_TIMEOUT: 360 2023-11-14T04:54:47.6872848Z ##[endgroup] 2023-11-14T04:54:47.6978437Z ##[group]Run set -x 2023-11-14T04:54:47.6979017Z set -x 2023-11-14T04:54:47.6979406Z  2023-11-14T04:54:47.6979864Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2023-11-14T04:54:47.6980553Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2023-11-14T04:54:47.6981282Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2023-11-14T04:54:47.6981964Z  TEST_COMMAND=.ci/onnx/test.sh 2023-11-14T04:54:47.6982477Z else 2023-11-14T04:54:47.6982924Z  TEST_COMMAND=.ci/pytorch/test.sh 2023-11-14T04:54:47.6983473Z fi 2023-11-14T04:54:47.6983814Z  2023-11-14T04:54:47.6984418Z # detached container should get cleaned up by teardown_ec2_linux 2023-11-14T04:54:47.6985402Z # TODO: Stop building test binaries as part of the build phase 2023-11-14T04:54:47.6986253Z # Used for GPU_FLAG since that doesn't play nice 2023-11-14T04:54:47.6987028Z # shellcheck disable=SC2086,SC2090 2023-11-14T04:54:47.6987604Z container_name=$(docker run \ 2023-11-14T04:54:47.6988149Z  ${GPU_FLAG:-} \ 2023-11-14T04:54:47.6988633Z  -e BUILD_ENVIRONMENT \ 2023-11-14T04:54:47.6989134Z  -e PR_NUMBER \ 2023-11-14T04:54:47.6989602Z  -e GITHUB_ACTIONS \ 2023-11-14T04:54:47.6990106Z  -e GITHUB_REPOSITORY \ 2023-11-14T04:54:47.6990618Z  -e GITHUB_WORKFLOW \ 2023-11-14T04:54:47.6991118Z  -e GITHUB_JOB \ 2023-11-14T04:54:47.6991590Z  -e GITHUB_RUN_ID \ 2023-11-14T04:54:47.6992086Z  -e GITHUB_RUN_NUMBER \ 2023-11-14T04:54:47.6992593Z  -e GITHUB_RUN_ATTEMPT \ 2023-11-14T04:54:47.6993108Z  -e JOB_ID \ 2023-11-14T04:54:47.6993549Z  -e BASE_SHA \ 2023-11-14T04:54:47.6993973Z  -e BRANCH \ 2023-11-14T04:54:47.6994409Z  -e SHA1 \ 2023-11-14T04:54:47.6995113Z  -e AWS_DEFAULT_REGION \ 2023-11-14T04:54:47.6995631Z  -e IN_WHEEL_TEST \ 2023-11-14T04:54:47.6996128Z  -e SHARD_NUMBER \ 2023-11-14T04:54:47.6996605Z  -e TEST_CONFIG \ 2023-11-14T04:54:47.6997065Z  -e NUM_TEST_SHARDS \ 2023-11-14T04:54:47.6997572Z  -e REENABLED_ISSUES \ 2023-11-14T04:54:47.6998103Z  -e CONTINUE_THROUGH_ERROR \ 2023-11-14T04:54:47.6998656Z  -e PYTORCH_RETRY_TEST_CASES \ 2023-11-14T04:54:47.6999494Z  -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \ 2023-11-14T04:54:47.7000071Z  -e PR_LABELS \ 2023-11-14T04:54:47.7000571Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2023-11-14T04:54:47.7001165Z  -e SCCACHE_BUCKET \ 2023-11-14T04:54:47.7001675Z  -e SCCACHE_S3_KEY_PREFIX \ 2023-11-14T04:54:47.7002177Z  -e XLA_CUDA \ 2023-11-14T04:54:47.7002704Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2023-11-14T04:54:47.7003364Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2023-11-14T04:54:47.7004023Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2023-11-14T04:54:47.7004686Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2023-11-14T04:54:47.7005298Z  -e HUGGING_FACE_HUB_TOKEN \ 2023-11-14T04:54:47.7005819Z  -e DASHBOARD_TAG \ 2023-11-14T04:54:47.7006431Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2023-11-14T04:54:47.7007116Z  --ulimit stack=10485760:83886080 \ 2023-11-14T04:54:47.7007752Z  --security-opt seccomp=unconfined \ 2023-11-14T04:54:47.7008328Z  --cap-add=SYS_PTRACE \ 2023-11-14T04:54:47.7008837Z  --ipc=host \ 2023-11-14T04:54:47.7009301Z  --shm-size="${SHM_SIZE}" \ 2023-11-14T04:54:47.7009788Z  --tty \ 2023-11-14T04:54:47.7010202Z  --detach \ 2023-11-14T04:54:47.7010668Z  --name="${container_name}" \ 2023-11-14T04:54:47.7011186Z  --user jenkins \ 2023-11-14T04:54:47.7011816Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2023-11-14T04:54:47.7012673Z  -w /var/lib/jenkins/workspace \ 2023-11-14T04:54:47.7013237Z  "${DOCKER_IMAGE}" 2023-11-14T04:54:47.7013690Z ) 2023-11-14T04:54:47.7014220Z # Propagate download.pytorch.org IP to container 2023-11-14T04:54:47.7015384Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2023-11-14T04:54:47.7016640Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2023-11-14T04:54:47.7017817Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2023-11-14T04:54:47.7026291Z shell: /usr/bin/bash -e {0} 2023-11-14T04:54:47.7026733Z env: 2023-11-14T04:54:47.7027092Z GIT_DEFAULT_BRANCH: main 2023-11-14T04:54:47.7027683Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:54:47.7028410Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2023-11-14T04:54:47.7029035Z PR_NUMBER: 112098 2023-11-14T04:54:47.7029489Z GITHUB_REPOSITORY: pytorch/pytorch 2023-11-14T04:54:47.7030028Z GITHUB_WORKFLOW: pull 2023-11-14T04:54:47.7030444Z GITHUB_JOB: test 2023-11-14T04:54:47.7030850Z GITHUB_RUN_ID: 6859165800 2023-11-14T04:54:47.7031309Z GITHUB_RUN_NUMBER: 155684 2023-11-14T04:54:47.7031749Z GITHUB_RUN_ATTEMPT: 1 2023-11-14T04:54:47.7032182Z JOB_ID: 18651430132 2023-11-14T04:54:47.7032606Z BRANCH: pull/112098 2023-11-14T04:54:47.7033084Z SHA1: fe705c0d795479e12d728ea79f708a42b978ffae 2023-11-14T04:54:47.7033743Z BASE_SHA: e1c872e009e8aa07281f38723353dc43baf2fc12 2023-11-14T04:54:47.7034353Z PYTORCH_RETRY_TEST_CASES: 1 2023-11-14T04:54:47.7035150Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2023-11-14T04:54:47.7035673Z TEST_CONFIG: distributed 2023-11-14T04:54:47.7036117Z SHARD_NUMBER: 2 2023-11-14T04:54:47.7036493Z NUM_TEST_SHARDS: 3 2023-11-14T04:54:47.7036912Z REENABLED_ISSUES: 2023-11-14T04:54:47.7037359Z CONTINUE_THROUGH_ERROR: True 2023-11-14T04:54:47.7037923Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2023-11-14T04:54:47.7038560Z SCCACHE_S3_KEY_PREFIX: pull 2023-11-14T04:54:47.7039028Z SHM_SIZE: 2g 2023-11-14T04:54:47.7040243Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:54:47.7041576Z XLA_CUDA: 2023-11-14T04:54:47.7042198Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2023-11-14T04:54:47.7043171Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2023-11-14T04:54:47.7043731Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2023-11-14T04:54:47.7044256Z DASHBOARD_TAG: 2023-11-14T04:54:47.7044645Z HUGGING_FACE_HUB_TOKEN: 2023-11-14T04:54:47.7045087Z ##[endgroup] 2023-11-14T04:54:47.7072594Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2023-11-14T04:54:47.7073426Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2023-11-14T04:54:47.7074068Z + TEST_COMMAND=.ci/pytorch/test.sh 2023-11-14T04:54:47.7084423Z +++ nproc --ignore=2 2023-11-14T04:54:47.7103716Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_6859165800 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T04:55:03.8695920Z + container_name=390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T04:55:03.8697505Z + grep download.pytorch.org /etc/hosts 2023-11-14T04:55:03.8700223Z + docker exec -i 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 sudo bash -c '/bin/cat >> /etc/hosts' 2023-11-14T04:55:03.9513218Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2023-11-14T04:55:03.9567182Z + echo DOCKER_CONTAINER_ID=390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T04:55:03.9572468Z ++ echo dist/torch-2.2.0a0+gitfe705c0-cp310-cp310-linux_x86_64.whl 2023-11-14T04:55:03.9574983Z + docker exec -t 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 sh -c 'pip install dist/torch-2.2.0a0+gitfe705c0-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2023-11-14T04:55:04.5195918Z Processing ./dist/torch-2.2.0a0+gitfe705c0-cp310-cp310-linux_x86_64.whl 2023-11-14T04:55:05.6029891Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+gitfe705c0) (3.9.0) 2023-11-14T04:55:05.6032310Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+gitfe705c0) (4.8.0) 2023-11-14T04:55:05.6037444Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+gitfe705c0) (1.12) 2023-11-14T04:55:05.6043048Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+gitfe705c0) (2.8.8) 2023-11-14T04:55:05.6047899Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+gitfe705c0) (3.1.2) 2023-11-14T04:55:05.6053108Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+gitfe705c0) (2023.4.0) 2023-11-14T04:55:05.6069853Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+gitfe705c0) (3.3.0) 2023-11-14T04:55:05.6155116Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.2.0a0+gitfe705c0) (1.21.2) 2023-11-14T04:55:05.6737723Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.2.0a0+gitfe705c0) (2.1.3) 2023-11-14T04:55:05.6964546Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.2.0a0+gitfe705c0) (1.3.0) 2023-11-14T04:55:06.7437668Z Installing collected packages: torch 2023-11-14T04:55:18.1300643Z Successfully installed torch-2.2.0a0+gitfe705c0 2023-11-14T04:55:18.2047035Z + echo 'Environment variables:' 2023-11-14T04:55:18.2048053Z Environment variables: 2023-11-14T04:55:18.2048513Z + env 2023-11-14T04:55:18.2054407Z INSTALLED_DB=yes 2023-11-14T04:55:18.2055456Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2023-11-14T04:55:18.2056106Z NVIDIA_VISIBLE_DEVICES=all 2023-11-14T04:55:18.2058537Z NV_NVML_DEV_VERSION=11.8.86-1 2023-11-14T04:55:18.2059498Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2023-11-14T04:55:18.2060255Z CONTINUE_THROUGH_ERROR=True 2023-11-14T04:55:18.2060930Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2023-11-14T04:55:18.2061502Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2023-11-14T04:55:18.2062203Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2023-11-14T04:55:18.2062858Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1 2023-11-14T04:55:18.2063357Z HOSTNAME=390ec749fb05 2023-11-14T04:55:18.2064434Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ded93807-a2ed-4f56-ac04-f7ae50e12174 2023-11-14T04:55:18.2065503Z GITHUB_ACTION=__self 2023-11-14T04:55:18.2065962Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2023-11-14T04:55:18.2073722Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=510,driver<511 brand=unknown,driver>=510,driver<511 brand=nvidia,driver>=510,driver<511 brand=nvidiartx,driver>=510,driver<511 brand=geforce,driver>=510,driver<511 brand=geforcertx,driver>=510,driver<511 brand=quadro,driver>=510,driver<511 brand=quadrortx,driver>=510,driver<511 brand=titan,driver>=510,driver<511 brand=titanrtx,driver>=510,driver<511 brand=tesla,driver>=515,driver<516 brand=unknown,driver>=515,driver<516 brand=nvidia,driver>=515,driver<516 brand=nvidiartx,driver>=515,driver<516 brand=geforce,driver>=515,driver<516 brand=geforcertx,driver>=515,driver<516 brand=quadro,driver>=515,driver<516 brand=quadrortx,driver>=515,driver<516 brand=titan,driver>=515,driver<516 brand=titanrtx,driver>=515,driver<516 2023-11-14T04:55:18.2082819Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2023-11-14T04:55:18.2083489Z NV_NVTX_VERSION=11.8.86-1 2023-11-14T04:55:18.2083950Z GITHUB_RUN_NUMBER=155684 2023-11-14T04:55:18.2084393Z TEST_CONFIG=distributed 2023-11-14T04:55:18.2084941Z GITHUB_REPOSITORY_OWNER_ID=21003710 2023-11-14T04:55:18.2085532Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2023-11-14T04:55:18.2086130Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2023-11-14T04:55:18.2086677Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2023-11-14T04:55:18.2087224Z NV_LIBNPP_VERSION=11.8.0.86-1 2023-11-14T04:55:18.2087723Z GITHUB_TRIGGERING_ACTOR=clee2000 2023-11-14T04:55:18.2088293Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2023-11-14T04:55:18.2088897Z GITHUB_REF_TYPE=branch 2023-11-14T04:55:18.2089345Z TORCH_CUDA_ARCH_LIST=Maxwell 2023-11-14T04:55:18.2089820Z NCCL_VERSION=2.16.2-1 2023-11-14T04:55:18.2090332Z BASE_SHA=e1c872e009e8aa07281f38723353dc43baf2fc12 2023-11-14T04:55:18.2090905Z XLA_CUDA= 2023-11-14T04:55:18.2091259Z HUGGING_FACE_HUB_TOKEN= 2023-11-14T04:55:18.2093998Z *** 2023-11-14T04:55:18.2094378Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2023-11-14T04:55:18.2094874Z GITHUB_REPOSITORY_ID=65600975 2023-11-14T04:55:18.2095353Z GITHUB_ACTIONS=true 2023-11-14T04:55:18.2096073Z NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:55:18.2096709Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2023-11-14T04:55:18.2097405Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2023-11-14T04:55:18.2098010Z SHA1=fe705c0d795479e12d728ea79f708a42b978ffae 2023-11-14T04:55:18.2098648Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2023-11-14T04:55:18.2099290Z GITHUB_SHA=35b9c00c6974fbcfd0eb53491c53c61a6c439831 2023-11-14T04:55:18.2100232Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/112098/merge 2023-11-14T04:55:18.2101015Z UCC_HOME=/usr 2023-11-14T04:55:18.2101472Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2023-11-14T04:55:18.2102003Z NVIDIA_PRODUCT_NAME=CUDA 2023-11-14T04:55:18.2102579Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2023-11-14T04:55:18.2103191Z GITHUB_REF=refs/pull/112098/merge 2023-11-14T04:55:18.2103738Z NV_CUDA_CUDART_VERSION=11.8.89-1 2023-11-14T04:55:18.2104204Z SHARD_NUMBER=2 2023-11-14T04:55:18.2104604Z GITHUB_REF_PROTECTED=false 2023-11-14T04:55:18.2105088Z HOME=/var/lib/jenkins 2023-11-14T04:55:18.2105518Z GITHUB_API_URL=https://api.github.com 2023-11-14T04:55:18.2106087Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2023-11-14T04:55:18.2106697Z UCX_COMMIT=00bcc6bb18fc282eb160623b4c0d300147f579af 2023-11-14T04:55:18.2107292Z SCCACHE_S3_KEY_PREFIX=pull 2023-11-14T04:55:18.2107763Z CUDA_VERSION=11.8.0 2023-11-14T04:55:18.2108300Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2023-11-14T04:55:18.2108853Z NUM_TEST_SHARDS=3 2023-11-14T04:55:18.2109250Z UCX_HOME=/usr 2023-11-14T04:55:18.2110082Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2023-11-14T04:55:18.2111492Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ded93807-a2ed-4f56-ac04-f7ae50e12174 2023-11-14T04:55:18.2112574Z PYTORCH_RETRY_TEST_CASES=1 2023-11-14T04:55:18.2113689Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ded93807-a2ed-4f56-ac04-f7ae50e12174 2023-11-14T04:55:18.2115364Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2023-11-14T04:55:18.2116248Z GITHUB_EVENT_NAME=pull_request 2023-11-14T04:55:18.2116725Z DASHBOARD_TAG= 2023-11-14T04:55:18.2117099Z GITHUB_RUN_ID=6859165800 2023-11-14T04:55:18.2117698Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2023-11-14T04:55:18.2118396Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2023-11-14T04:55:18.2119690Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ded93807-a2ed-4f56-ac04-f7ae50e12174 2023-11-14T04:55:18.2120795Z GITHUB_ACTOR=clee2000 2023-11-14T04:55:18.2121285Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2023-11-14T04:55:18.2121786Z PR_NUMBER=112098 2023-11-14T04:55:18.2122213Z GITHUB_RUN_ATTEMPT=1 2023-11-14T04:55:18.2122659Z ANACONDA_PYTHON_VERSION=3.10 2023-11-14T04:55:18.2123232Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2023-11-14T04:55:18.2123805Z TERM=xterm 2023-11-14T04:55:18.2124245Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2023-11-14T04:55:18.2124788Z INSTALLED_VISION=yes 2023-11-14T04:55:18.2125189Z BRANCH=pull/112098 2023-11-14T04:55:18.2125621Z OPENSSL_ROOT_DIR=/opt/openssl 2023-11-14T04:55:18.2126141Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2023-11-14T04:55:18.2126764Z CUDA_PATH=/usr/local/cuda 2023-11-14T04:55:18.2128003Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2023-11-14T04:55:18.2129093Z GITHUB_SERVER_URL=https://github.com 2023-11-14T04:55:18.2129771Z UCC_COMMIT=7cb07a76ccedad7e56ceb136b865eb9319c258ea 2023-11-14T04:55:18.2130583Z REENABLED_ISSUES= 2023-11-14T04:55:18.2131090Z SHLVL=1 2023-11-14T04:55:18.2131498Z MAX_JOBS=30 2023-11-14T04:55:18.2132071Z NV_CUDA_LIB_VERSION=11.8.0-1 2023-11-14T04:55:18.2132638Z NVARCH=x86_64 2023-11-14T04:55:18.2133084Z GITHUB_ACTOR_ID=44682903 2023-11-14T04:55:18.2133857Z GITHUB_WORKFLOW_SHA=35b9c00c6974fbcfd0eb53491c53c61a6c439831 2023-11-14T04:55:18.2134637Z GITHUB_REF_NAME=112098/merge 2023-11-14T04:55:18.2135261Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2023-11-14T04:55:18.2136488Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2023-11-14T04:55:18.2137346Z GITHUB_JOB=test 2023-11-14T04:55:18.2137939Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2023-11-14T04:55:18.2138823Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2023-11-14T04:55:18.2168207Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2023-11-14T04:55:18.2168884Z GITHUB_REPOSITORY=pytorch/pytorch 2023-11-14T04:55:18.2169456Z NV_NVPROF_VERSION=11.8.87-1 2023-11-14T04:55:18.2169927Z GITHUB_RETENTION_DAYS=90 2023-11-14T04:55:18.2170350Z OPENSSL_DIR=/opt/openssl 2023-11-14T04:55:18.2170796Z GITHUB_ACTION_REPOSITORY= 2023-11-14T04:55:18.2172145Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2023-11-14T04:55:18.2173527Z GITHUB_BASE_REF=main 2023-11-14T04:55:18.2173946Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2023-11-14T04:55:18.2174418Z CI=true 2023-11-14T04:55:18.2174804Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2023-11-14T04:55:18.2175286Z GITHUB_REPOSITORY_OWNER=pytorch 2023-11-14T04:55:18.2175732Z JOB_ID=18651430132 2023-11-14T04:55:18.2176090Z INSTALLED_PROTOBUF=yes 2023-11-14T04:55:18.2176472Z GITHUB_HEAD_REF=csl/test3 2023-11-14T04:55:18.2176905Z GITHUB_ACTION_REF= 2023-11-14T04:55:18.2177452Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2023-11-14T04:55:18.2177999Z GITHUB_WORKFLOW=pull 2023-11-14T04:55:18.2178392Z DEBIAN_FRONTEND=noninteractive 2023-11-14T04:55:18.2179724Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ded93807-a2ed-4f56-ac04-f7ae50e12174 2023-11-14T04:55:18.2180832Z SKIP_SCCACHE_INITIALIZATION=1 2023-11-14T04:55:18.2181297Z _=/usr/bin/env 2023-11-14T04:55:18.2181876Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2023-11-14T04:55:18.2277459Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2023-11-14T04:55:18.2278658Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2023-11-14T04:55:18.2280015Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2023-11-14T04:55:18.2281083Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2023-11-14T04:55:18.2281864Z + BUILD_DIR=build 2023-11-14T04:55:18.2282268Z + BUILD_RENAMED_DIR=build_renamed 2023-11-14T04:55:18.2282772Z + BUILD_BIN_DIR=build/bin 2023-11-14T04:55:18.2283209Z + export VALGRIND=ON 2023-11-14T04:55:18.2283610Z + VALGRIND=ON 2023-11-14T04:55:18.2284219Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2023-11-14T04:55:18.2284764Z + [[ 0 == \1 ]] 2023-11-14T04:55:18.2285149Z + [[ True == \1 ]] 2023-11-14T04:55:18.2285731Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2023-11-14T04:55:18.2286345Z ++ realpath build/custom_test_artifacts 2023-11-14T04:55:18.2294546Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2023-11-14T04:55:18.2298097Z ++ dirname .ci/pytorch/test.sh 2023-11-14T04:55:18.2306190Z + source .ci/pytorch/common.sh 2023-11-14T04:55:18.2309801Z +++ dirname .ci/pytorch/common.sh 2023-11-14T04:55:18.2318956Z ++ source .ci/pytorch/common_utils.sh 2023-11-14T04:55:18.2321071Z +++ declare -f -t trap_add 2023-11-14T04:55:18.2328800Z ++ set -ex 2023-11-14T04:55:18.2329350Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2023-11-14T04:55:18.2329896Z ++ BUILD_TEST_LIBTORCH=0 2023-11-14T04:55:18.2330394Z + echo 'Environment variables' 2023-11-14T04:55:18.2330855Z Environment variables 2023-11-14T04:55:18.2331214Z + env 2023-11-14T04:55:18.2337011Z INSTALLED_DB=yes 2023-11-14T04:55:18.2337641Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2023-11-14T04:55:18.2338255Z NVIDIA_VISIBLE_DEVICES=all 2023-11-14T04:55:18.2338709Z NV_NVML_DEV_VERSION=11.8.86-1 2023-11-14T04:55:18.2339519Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2023-11-14T04:55:18.2340225Z CONTINUE_THROUGH_ERROR=True 2023-11-14T04:55:18.2340829Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2023-11-14T04:55:18.2341731Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2023-11-14T04:55:18.2342433Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2023-11-14T04:55:18.2343038Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1 2023-11-14T04:55:18.2343591Z HOSTNAME=390ec749fb05 2023-11-14T04:55:18.2344674Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ded93807-a2ed-4f56-ac04-f7ae50e12174 2023-11-14T04:55:18.2345969Z GITHUB_ACTION=__self 2023-11-14T04:55:18.2346441Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2023-11-14T04:55:18.2358455Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=510,driver<511 brand=unknown,driver>=510,driver<511 brand=nvidia,driver>=510,driver<511 brand=nvidiartx,driver>=510,driver<511 brand=geforce,driver>=510,driver<511 brand=geforcertx,driver>=510,driver<511 brand=quadro,driver>=510,driver<511 brand=quadrortx,driver>=510,driver<511 brand=titan,driver>=510,driver<511 brand=titanrtx,driver>=510,driver<511 brand=tesla,driver>=515,driver<516 brand=unknown,driver>=515,driver<516 brand=nvidia,driver>=515,driver<516 brand=nvidiartx,driver>=515,driver<516 brand=geforce,driver>=515,driver<516 brand=geforcertx,driver>=515,driver<516 brand=quadro,driver>=515,driver<516 brand=quadrortx,driver>=515,driver<516 brand=titan,driver>=515,driver<516 brand=titanrtx,driver>=515,driver<516 2023-11-14T04:55:18.2374881Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2023-11-14T04:55:18.2376211Z NV_NVTX_VERSION=11.8.86-1 2023-11-14T04:55:18.2377130Z GITHUB_RUN_NUMBER=155684 2023-11-14T04:55:18.2377975Z TEST_CONFIG=distributed 2023-11-14T04:55:18.2378854Z GITHUB_REPOSITORY_OWNER_ID=21003710 2023-11-14T04:55:18.2380080Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2023-11-14T04:55:18.2381270Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2023-11-14T04:55:18.2382402Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2023-11-14T04:55:18.2383484Z NV_LIBNPP_VERSION=11.8.0.86-1 2023-11-14T04:55:18.2384451Z GITHUB_TRIGGERING_ACTOR=clee2000 2023-11-14T04:55:18.2385608Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2023-11-14T04:55:18.2386839Z GITHUB_REF_TYPE=branch 2023-11-14T04:55:18.2387679Z TORCH_CUDA_ARCH_LIST=Maxwell 2023-11-14T04:55:18.2388678Z NCCL_VERSION=2.16.2-1 2023-11-14T04:55:18.2389653Z BASE_SHA=e1c872e009e8aa07281f38723353dc43baf2fc12 2023-11-14T04:55:18.2390805Z XLA_CUDA= 2023-11-14T04:55:18.2391519Z HUGGING_FACE_HUB_TOKEN= 2023-11-14T04:55:18.2392495Z *** 2023-11-14T04:55:18.2393148Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2023-11-14T04:55:18.2394161Z GITHUB_REPOSITORY_ID=65600975 2023-11-14T04:55:18.2395431Z GITHUB_ACTIONS=true 2023-11-14T04:55:18.2396244Z NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T04:55:18.2397543Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2023-11-14T04:55:18.2398925Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2023-11-14T04:55:18.2400097Z SHA1=fe705c0d795479e12d728ea79f708a42b978ffae 2023-11-14T04:55:18.2401435Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2023-11-14T04:55:18.2402774Z GITHUB_SHA=35b9c00c6974fbcfd0eb53491c53c61a6c439831 2023-11-14T04:55:18.2404632Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/112098/merge 2023-11-14T04:55:18.2406260Z UCC_HOME=/usr 2023-11-14T04:55:18.2407155Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2023-11-14T04:55:18.2408245Z NVIDIA_PRODUCT_NAME=CUDA 2023-11-14T04:55:18.2409410Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2023-11-14T04:55:18.2410633Z GITHUB_REF=refs/pull/112098/merge 2023-11-14T04:55:18.2411741Z NV_CUDA_CUDART_VERSION=11.8.89-1 2023-11-14T04:55:18.2412648Z SHARD_NUMBER=2 2023-11-14T04:55:18.2413434Z GITHUB_REF_PROTECTED=false 2023-11-14T04:55:18.2414310Z HOME=/var/lib/jenkins 2023-11-14T04:55:18.2415195Z GITHUB_API_URL=https://api.github.com 2023-11-14T04:55:18.2416647Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2023-11-14T04:55:18.2417838Z UCX_COMMIT=00bcc6bb18fc282eb160623b4c0d300147f579af 2023-11-14T04:55:18.2419036Z SCCACHE_S3_KEY_PREFIX=pull 2023-11-14T04:55:18.2419898Z CUDA_VERSION=11.8.0 2023-11-14T04:55:18.2420939Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2023-11-14T04:55:18.2422077Z NUM_TEST_SHARDS=3 2023-11-14T04:55:18.2422828Z UCX_HOME=/usr 2023-11-14T04:55:18.2424125Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2023-11-14T04:55:18.2426940Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ded93807-a2ed-4f56-ac04-f7ae50e12174 2023-11-14T04:55:18.2429125Z PYTORCH_RETRY_TEST_CASES=1 2023-11-14T04:55:18.2431391Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ded93807-a2ed-4f56-ac04-f7ae50e12174 2023-11-14T04:55:18.2434337Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2023-11-14T04:55:18.2436448Z GITHUB_EVENT_NAME=pull_request 2023-11-14T04:55:18.2437382Z DASHBOARD_TAG= 2023-11-14T04:55:18.2438089Z GITHUB_RUN_ID=6859165800 2023-11-14T04:55:18.2439272Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2023-11-14T04:55:18.2440674Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2023-11-14T04:55:18.2443256Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ded93807-a2ed-4f56-ac04-f7ae50e12174 2023-11-14T04:55:18.2445548Z GITHUB_ACTOR=clee2000 2023-11-14T04:55:18.2446522Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2023-11-14T04:55:18.2447791Z PR_NUMBER=112098 2023-11-14T04:55:18.2448576Z GITHUB_RUN_ATTEMPT=1 2023-11-14T04:55:18.2449474Z VALGRIND=ON 2023-11-14T04:55:18.2450193Z ANACONDA_PYTHON_VERSION=3.10 2023-11-14T04:55:18.2451332Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2023-11-14T04:55:18.2452522Z TERM=xterm 2023-11-14T04:55:18.2453362Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2023-11-14T04:55:18.2454408Z INSTALLED_VISION=yes 2023-11-14T04:55:18.2455249Z BRANCH=pull/112098 2023-11-14T04:55:18.2456039Z OPENSSL_ROOT_DIR=/opt/openssl 2023-11-14T04:55:18.2457068Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2023-11-14T04:55:18.2458186Z CUDA_PATH=/usr/local/cuda 2023-11-14T04:55:18.2460116Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2023-11-14T04:55:18.2462091Z GITHUB_SERVER_URL=https://github.com 2023-11-14T04:55:18.2463340Z UCC_COMMIT=7cb07a76ccedad7e56ceb136b865eb9319c258ea 2023-11-14T04:55:18.2464522Z REENABLED_ISSUES= 2023-11-14T04:55:18.2465279Z SHLVL=1 2023-11-14T04:55:18.2465980Z MAX_JOBS=30 2023-11-14T04:55:18.2466790Z NV_CUDA_LIB_VERSION=11.8.0-1 2023-11-14T04:55:18.2467685Z NVARCH=x86_64 2023-11-14T04:55:18.2468455Z GITHUB_ACTOR_ID=44682903 2023-11-14T04:55:18.2469594Z GITHUB_WORKFLOW_SHA=35b9c00c6974fbcfd0eb53491c53c61a6c439831 2023-11-14T04:55:18.2470899Z GITHUB_REF_NAME=112098/merge 2023-11-14T04:55:18.2472042Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2023-11-14T04:55:18.2473770Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2023-11-14T04:55:18.2475526Z GITHUB_JOB=test 2023-11-14T04:55:18.2476564Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2023-11-14T04:55:18.2477960Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2023-11-14T04:55:18.2479452Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2023-11-14T04:55:18.2480566Z GITHUB_REPOSITORY=pytorch/pytorch 2023-11-14T04:55:18.2481616Z NV_NVPROF_VERSION=11.8.87-1 2023-11-14T04:55:18.2482554Z GITHUB_RETENTION_DAYS=90 2023-11-14T04:55:18.2483458Z OPENSSL_DIR=/opt/openssl 2023-11-14T04:55:18.2484340Z GITHUB_ACTION_REPOSITORY= 2023-11-14T04:55:18.2487094Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2023-11-14T04:55:18.2489966Z GITHUB_BASE_REF=main 2023-11-14T04:55:18.2490856Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2023-11-14T04:55:18.2491790Z CI=true 2023-11-14T04:55:18.2492621Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2023-11-14T04:55:18.2493985Z GITHUB_REPOSITORY_OWNER=pytorch 2023-11-14T04:55:18.2494946Z JOB_ID=18651430132 2023-11-14T04:55:18.2495756Z INSTALLED_PROTOBUF=yes 2023-11-14T04:55:18.2496572Z GITHUB_HEAD_REF=csl/test3 2023-11-14T04:55:18.2497429Z GITHUB_ACTION_REF= 2023-11-14T04:55:18.2498565Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2023-11-14T04:55:18.2499745Z GITHUB_WORKFLOW=pull 2023-11-14T04:55:18.2500630Z DEBIAN_FRONTEND=noninteractive 2023-11-14T04:55:18.2503019Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ded93807-a2ed-4f56-ac04-f7ae50e12174 2023-11-14T04:55:18.2505240Z SKIP_SCCACHE_INITIALIZATION=1 2023-11-14T04:55:18.2506149Z _=/usr/bin/env 2023-11-14T04:55:18.2507022Z + echo 'Testing pytorch' 2023-11-14T04:55:18.2507863Z Testing pytorch 2023-11-14T04:55:18.2508703Z + export LANG=C.UTF-8 2023-11-14T04:55:18.2509559Z + LANG=C.UTF-8 2023-11-14T04:55:18.2510238Z + PR_NUMBER=112098 2023-11-14T04:55:18.2511048Z + [[ distributed == \d\e\f\a\u\l\t ]] 2023-11-14T04:55:18.2512191Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2023-11-14T04:55:18.2513583Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2023-11-14T04:55:18.2515175Z + [[ distributed == \s\l\o\w ]] 2023-11-14T04:55:18.2516639Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2023-11-14T04:55:18.2518237Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2023-11-14T04:55:18.2519538Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2023-11-14T04:55:18.2520737Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2023-11-14T04:55:18.2521829Z + [[ distributed == *crossref* ]] 2023-11-14T04:55:18.2523466Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2023-11-14T04:55:18.2525082Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2023-11-14T04:55:18.2526472Z + pip_install --user ninja==1.10.2 2023-11-14T04:55:18.2527824Z + pip install --progress-bar off --user ninja==1.10.2 2023-11-14T04:55:18.8025274Z Collecting ninja==1.10.2 2023-11-14T04:55:18.8497682Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2023-11-14T04:55:19.9363163Z Installing collected packages: ninja 2023-11-14T04:55:19.9468616Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2023-11-14T04:55:19.9470266Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2023-11-14T04:55:19.9524229Z Successfully installed ninja-1.10.2 2023-11-14T04:55:20.0328777Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2023-11-14T04:55:20.0331617Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2023-11-14T04:55:20.0333586Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2023-11-14T04:55:20.0334322Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2023-11-14T04:55:20.0335091Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2023-11-14T04:55:20.0336246Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2023-11-14T04:55:20.0337606Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2023-11-14T04:55:20.0338591Z + cd test 2023-11-14T04:55:20.0339170Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2023-11-14T04:55:21.7186820Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2023-11-14T04:55:21.7187746Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2023-11-14T04:55:21.7188385Z + DYNAMO_BENCHMARK_FLAGS=() 2023-11-14T04:55:21.7188842Z + [[ distributed == *dynamo_eager* ]] 2023-11-14T04:55:21.7189407Z + [[ distributed == *aot_eager* ]] 2023-11-14T04:55:21.7189951Z + [[ distributed == *aot_inductor* ]] 2023-11-14T04:55:21.7190480Z + [[ distributed == *inductor* ]] 2023-11-14T04:55:21.7191522Z + [[ distributed == *dynamic* ]] 2023-11-14T04:55:21.7192057Z + [[ distributed == *cpu_accuracy* ]] 2023-11-14T04:55:21.7192832Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2023-11-14T04:55:21.7202357Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *tbb* ]] 2023-11-14T04:55:21.7219304Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2023-11-14T04:55:21.7220149Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2023-11-14T04:55:21.7222586Z + cd test 2023-11-14T04:55:21.7223213Z + python -c 'import torch; print(torch.__config__.show())' 2023-11-14T04:55:23.1878735Z PyTorch built with: 2023-11-14T04:55:23.1880055Z - GCC 9.4 2023-11-14T04:55:23.1880693Z - C++ Version: 201703 2023-11-14T04:55:23.1881789Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2023-11-14T04:55:23.1883158Z - Intel(R) MKL-DNN v3.1.1 (Git Hash 64f6bcbcbab628e96f33a62c3e975f8535a7bde4) 2023-11-14T04:55:23.1884041Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2023-11-14T04:55:23.1884756Z - LAPACK is enabled (usually provided by MKL) 2023-11-14T04:55:23.1885370Z - NNPACK is enabled 2023-11-14T04:55:23.1885881Z - CPU capability usage: AVX2 2023-11-14T04:55:23.1886421Z - CUDA Runtime 11.8 2023-11-14T04:55:23.1887076Z - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52 2023-11-14T04:55:23.1887802Z - CuDNN 8.7 2023-11-14T04:55:23.1888228Z - Magma 2.6.1 2023-11-14T04:55:23.1897644Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.2.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 2023-11-14T04:55:23.1906123Z 2023-11-14T04:55:23.4371781Z + cd test 2023-11-14T04:55:23.4372807Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2023-11-14T04:55:24.8373570Z ATen/Parallel: 2023-11-14T04:55:24.8374706Z at::get_num_threads() : 16 2023-11-14T04:55:24.8375886Z at::get_num_interop_threads() : 32 2023-11-14T04:55:24.8377217Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2023-11-14T04:55:24.8378381Z omp_get_max_threads() : 16 2023-11-14T04:55:24.8381143Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2023-11-14T04:55:24.8383589Z mkl_get_max_threads() : 16 2023-11-14T04:55:24.8385403Z Intel(R) MKL-DNN v3.1.1 (Git Hash 64f6bcbcbab628e96f33a62c3e975f8535a7bde4) 2023-11-14T04:55:24.8387232Z std::thread::hardware_concurrency() : 32 2023-11-14T04:55:24.8388499Z Environment variables: 2023-11-14T04:55:24.8389549Z OMP_NUM_THREADS : [not set] 2023-11-14T04:55:24.8390462Z MKL_NUM_THREADS : [not set] 2023-11-14T04:55:24.8391427Z ATen parallel backend: OpenMP 2023-11-14T04:55:24.8392050Z 2023-11-14T04:55:25.0549731Z + [[ distributed == *backward* ]] 2023-11-14T04:55:25.0550430Z + [[ distributed == *xla* ]] 2023-11-14T04:55:25.0550972Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2023-11-14T04:55:25.0551947Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2023-11-14T04:55:25.0553130Z + [[ distributed == distributed ]] 2023-11-14T04:55:25.0553647Z + test_distributed 2023-11-14T04:55:25.0554191Z + echo 'Testing distributed python tests' 2023-11-14T04:55:25.0555483Z Testing distributed python tests 2023-11-14T04:55:25.0556313Z + python test/run_test.py --distributed-tests --shard 2 3 --verbose 2023-11-14T04:55:25.1738266Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2023-11-14T04:55:25.1739720Z import pkg_resources 2023-11-14T04:55:26.9143402Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2023-11-14T04:55:26.9977099Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2023-11-14T04:55:27.0789306Z Ignoring disabled issues: [''] 2023-11-14T04:55:27.0797134Z Received 182 tests to prioritize 2023-11-14T04:55:27.0798337Z distributed/_composable/fully_shard/test_fully_shard_compile 2023-11-14T04:55:27.0799169Z distributed/_composable/fully_shard/test_fully_shard_init 2023-11-14T04:55:27.0799993Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 2023-11-14T04:55:27.0801218Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 2023-11-14T04:55:27.0802154Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 2023-11-14T04:55:27.0803314Z distributed/_composable/fully_shard/test_fully_shard_runtime 2023-11-14T04:55:27.0804133Z distributed/_composable/fully_shard/test_fully_shard_util 2023-11-14T04:55:27.0804903Z distributed/_composable/test_checkpoint 2023-11-14T04:55:27.0805502Z distributed/_composable/test_compose 2023-11-14T04:55:27.0806061Z distributed/_composable/test_contract 2023-11-14T04:55:27.0806590Z distributed/_composable/test_replicate 2023-11-14T04:55:27.0807296Z distributed/_shard/sharded_optim/test_sharded_optim 2023-11-14T04:55:27.0807995Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 2023-11-14T04:55:27.0808703Z distributed/_shard/sharded_tensor/ops/test_embedding 2023-11-14T04:55:27.0809453Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2023-11-14T04:55:27.0810178Z distributed/_shard/sharded_tensor/ops/test_init 2023-11-14T04:55:27.0810855Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2023-11-14T04:55:27.0811552Z distributed/_shard/sharded_tensor/test_logger 2023-11-14T04:55:27.0812249Z distributed/_shard/sharded_tensor/test_sharded_tensor 2023-11-14T04:55:27.0813003Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2023-11-14T04:55:27.0813786Z distributed/_shard/sharding_plan/test_sharding_plan 2023-11-14T04:55:27.0814819Z distributed/_shard/sharding_spec/test_sharding_spec 2023-11-14T04:55:27.0815443Z distributed/_shard/test_sharder 2023-11-14T04:55:27.0816011Z distributed/_spmd/test_data_parallel 2023-11-14T04:55:27.0816601Z distributed/_spmd/test_graph_utils 2023-11-14T04:55:27.0817125Z distributed/_spmd/test_tracing 2023-11-14T04:55:27.0817660Z distributed/_spmd/test_transformation 2023-11-14T04:55:27.0818368Z distributed/_tensor/debug/test_op_coverage 2023-11-14T04:55:27.0819417Z distributed/_tensor/experimental/test_tp_transform 2023-11-14T04:55:27.0820013Z distributed/_tensor/test_api 2023-11-14T04:55:27.0820791Z distributed/_tensor/test_common_rules 2023-11-14T04:55:27.0821553Z distributed/_tensor/test_dtensor 2023-11-14T04:55:27.0822085Z distributed/_tensor/test_dtensor_compile 2023-11-14T04:55:27.0822740Z distributed/_tensor/test_dtensor_ops 2023-11-14T04:55:27.0823273Z distributed/_tensor/test_embedding_ops 2023-11-14T04:55:27.0823834Z distributed/_tensor/test_init 2023-11-14T04:55:27.0824760Z distributed/_tensor/test_math_ops 2023-11-14T04:55:27.0825706Z distributed/_tensor/test_matrix_ops 2023-11-14T04:55:27.0826722Z distributed/_tensor/test_op_strategy 2023-11-14T04:55:27.0827590Z distributed/_tensor/test_optimizers 2023-11-14T04:55:27.0828713Z distributed/_tensor/test_pointwise_ops 2023-11-14T04:55:27.0829615Z distributed/_tensor/test_random_ops 2023-11-14T04:55:27.0830528Z distributed/_tensor/test_redistribute 2023-11-14T04:55:27.0831565Z distributed/_tensor/test_tensor_ops 2023-11-14T04:55:27.0832538Z distributed/_tensor/test_utils 2023-11-14T04:55:27.0833532Z distributed/_tensor/test_view_ops 2023-11-14T04:55:27.0834547Z distributed/_tensor/test_xla_integration 2023-11-14T04:55:27.0836145Z distributed/_tools/test_memory_tracker 2023-11-14T04:55:27.0837097Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2023-11-14T04:55:27.0838358Z distributed/algorithms/quantization/test_quantization 2023-11-14T04:55:27.0839546Z distributed/algorithms/test_join 2023-11-14T04:55:27.0840553Z distributed/checkpoint/e2e/test_e2e_save_and_load 2023-11-14T04:55:27.0841257Z distributed/checkpoint/e2e/test_fine_tuning 2023-11-14T04:55:27.0842336Z distributed/checkpoint/e2e/test_pipeline 2023-11-14T04:55:27.0843385Z distributed/checkpoint/test_checkpoint 2023-11-14T04:55:27.0844431Z distributed/checkpoint/test_dedup_tensors 2023-11-14T04:55:27.0845277Z distributed/checkpoint/test_dtensor_checkpoint 2023-11-14T04:55:27.0846165Z distributed/checkpoint/test_dtensor_resharding 2023-11-14T04:55:27.0847097Z distributed/checkpoint/test_file_system_checkpoint 2023-11-14T04:55:27.0848045Z distributed/checkpoint/test_file_system_checkpoint_cpu 2023-11-14T04:55:27.0848973Z distributed/checkpoint/test_fsdp_model_state 2023-11-14T04:55:27.0849840Z distributed/checkpoint/test_fsdp_optim_state 2023-11-14T04:55:27.0850583Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 2023-11-14T04:55:27.0851280Z distributed/checkpoint/test_fsspec 2023-11-14T04:55:27.0851877Z distributed/checkpoint/test_hsdp_checkpoint 2023-11-14T04:55:27.0852731Z distributed/checkpoint/test_nested_dict 2023-11-14T04:55:27.0853557Z distributed/checkpoint/test_planner 2023-11-14T04:55:27.0854479Z distributed/checkpoint/test_state_dict 2023-11-14T04:55:27.0855403Z distributed/checkpoint/test_state_dict_utils 2023-11-14T04:55:27.0856455Z distributed/checkpoint/test_tp_checkpoint 2023-11-14T04:55:27.0857477Z distributed/checkpoint/test_traverse 2023-11-14T04:55:27.0858339Z distributed/checkpoint/test_utils 2023-11-14T04:55:27.0859301Z distributed/elastic/events/lib_test 2023-11-14T04:55:27.0860109Z distributed/elastic/metrics/api_test 2023-11-14T04:55:27.0860958Z distributed/elastic/multiprocessing/api_test 2023-11-14T04:55:27.0861592Z distributed/elastic/timer/api_test 2023-11-14T04:55:27.0862199Z distributed/elastic/timer/local_timer_example 2023-11-14T04:55:27.0862817Z distributed/elastic/timer/local_timer_test 2023-11-14T04:55:27.0863453Z distributed/elastic/utils/distributed_test 2023-11-14T04:55:27.0864061Z distributed/elastic/utils/logging_test 2023-11-14T04:55:27.0864605Z distributed/elastic/utils/util_test 2023-11-14T04:55:27.0865196Z distributed/fsdp/test_checkpoint_wrapper 2023-11-14T04:55:27.0865827Z distributed/fsdp/test_distributed_checkpoint 2023-11-14T04:55:27.0866435Z distributed/fsdp/test_fsdp_apply 2023-11-14T04:55:27.0866991Z distributed/fsdp/test_fsdp_backward_prefetch 2023-11-14T04:55:27.0867603Z distributed/fsdp/test_fsdp_checkpoint 2023-11-14T04:55:27.0868165Z distributed/fsdp/test_fsdp_clip_grad_norm 2023-11-14T04:55:27.0868749Z distributed/fsdp/test_fsdp_comm 2023-11-14T04:55:27.0869287Z distributed/fsdp/test_fsdp_comm_hooks 2023-11-14T04:55:27.0869815Z distributed/fsdp/test_fsdp_core 2023-11-14T04:55:27.0870394Z distributed/fsdp/test_fsdp_dtensor_state_dict 2023-11-14T04:55:27.0871011Z distributed/fsdp/test_fsdp_exec_order 2023-11-14T04:55:27.0871549Z distributed/fsdp/test_fsdp_fine_tune 2023-11-14T04:55:27.0872130Z distributed/fsdp/test_fsdp_flatten_params 2023-11-14T04:55:27.0872769Z distributed/fsdp/test_fsdp_freezing_weights 2023-11-14T04:55:27.0873362Z distributed/fsdp/test_fsdp_fx 2023-11-14T04:55:27.0874120Z distributed/fsdp/test_fsdp_grad_acc 2023-11-14T04:55:27.0874927Z distributed/fsdp/test_fsdp_hybrid_shard 2023-11-14T04:55:27.0875558Z distributed/fsdp/test_fsdp_ignored_modules 2023-11-14T04:55:27.0876125Z distributed/fsdp/test_fsdp_input 2023-11-14T04:55:27.0876659Z distributed/fsdp/test_fsdp_memory 2023-11-14T04:55:27.0877193Z distributed/fsdp/test_fsdp_meta 2023-11-14T04:55:27.0877687Z distributed/fsdp/test_fsdp_misc 2023-11-14T04:55:27.0878251Z distributed/fsdp/test_fsdp_mixed_precision 2023-11-14T04:55:27.0878889Z distributed/fsdp/test_fsdp_multiple_forward 2023-11-14T04:55:27.0879516Z distributed/fsdp/test_fsdp_multiple_wrapping 2023-11-14T04:55:27.0880140Z distributed/fsdp/test_fsdp_optim_state 2023-11-14T04:55:27.0880711Z distributed/fsdp/test_fsdp_overlap 2023-11-14T04:55:27.0881246Z distributed/fsdp/test_fsdp_pure_fp16 2023-11-14T04:55:27.0881849Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2023-11-14T04:55:27.0882477Z distributed/fsdp/test_fsdp_state_dict 2023-11-14T04:55:27.0883048Z distributed/fsdp/test_fsdp_tp_integration 2023-11-14T04:55:27.0883647Z distributed/fsdp/test_fsdp_traversal 2023-11-14T04:55:27.0884198Z distributed/fsdp/test_fsdp_uneven 2023-11-14T04:55:27.0884734Z distributed/fsdp/test_fsdp_unshard_params 2023-11-14T04:55:27.0885356Z distributed/fsdp/test_fsdp_use_orig_params 2023-11-14T04:55:27.0886183Z distributed/fsdp/test_hsdp_dtensor_state_dict 2023-11-14T04:55:27.0886760Z distributed/fsdp/test_shard_utils 2023-11-14T04:55:27.0887287Z distributed/fsdp/test_utils 2023-11-14T04:55:27.0887918Z distributed/fsdp/test_wrap 2023-11-14T04:55:27.0888428Z distributed/nn/jit/test_instantiator 2023-11-14T04:55:27.0889067Z distributed/optim/test_apply_optimizer_in_backward 2023-11-14T04:55:27.0889725Z distributed/optim/test_named_optimizer 2023-11-14T04:55:27.0890334Z distributed/optim/test_zero_redundancy_optimizer 2023-11-14T04:55:27.0890986Z distributed/pipeline/sync/skip/test_api 2023-11-14T04:55:27.0891605Z distributed/pipeline/sync/skip/test_gpipe 2023-11-14T04:55:27.0892281Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2023-11-14T04:55:27.0893114Z distributed/pipeline/sync/skip/test_leak 2023-11-14T04:55:27.0893713Z distributed/pipeline/sync/skip/test_portal 2023-11-14T04:55:27.0894363Z distributed/pipeline/sync/skip/test_stash_pop 2023-11-14T04:55:27.0895009Z distributed/pipeline/sync/skip/test_tracker 2023-11-14T04:55:27.0895683Z distributed/pipeline/sync/skip/test_verify_skippables 2023-11-14T04:55:27.0896358Z distributed/pipeline/sync/test_balance 2023-11-14T04:55:27.0896945Z distributed/pipeline/sync/test_bugs 2023-11-14T04:55:27.0897502Z distributed/pipeline/sync/test_checkpoint 2023-11-14T04:55:27.0898096Z distributed/pipeline/sync/test_copy 2023-11-14T04:55:27.0898731Z distributed/pipeline/sync/test_deferred_batch_norm 2023-11-14T04:55:27.0899376Z distributed/pipeline/sync/test_dependency 2023-11-14T04:55:27.0899979Z distributed/pipeline/sync/test_inplace 2023-11-14T04:55:27.0900585Z distributed/pipeline/sync/test_microbatch 2023-11-14T04:55:27.0901154Z distributed/pipeline/sync/test_phony 2023-11-14T04:55:27.0901721Z distributed/pipeline/sync/test_pipe 2023-11-14T04:55:27.0902292Z distributed/pipeline/sync/test_pipeline 2023-11-14T04:55:27.0902850Z distributed/pipeline/sync/test_stream 2023-11-14T04:55:27.0903454Z distributed/pipeline/sync/test_transparency 2023-11-14T04:55:27.0904066Z distributed/pipeline/sync/test_worker 2023-11-14T04:55:27.0904642Z distributed/rpc/cuda/test_tensorpipe_agent 2023-11-14T04:55:27.0905245Z distributed/rpc/test_faulty_agent 2023-11-14T04:55:27.0905791Z distributed/rpc/test_share_memory 2023-11-14T04:55:27.0906311Z distributed/rpc/test_tensorpipe_agent 2023-11-14T04:55:27.0906941Z distributed/tensor/parallel/test_ddp_2d_parallel 2023-11-14T04:55:27.0907642Z distributed/tensor/parallel/test_fsdp_2d_parallel 2023-11-14T04:55:27.0908318Z distributed/tensor/parallel/test_parallelize_api 2023-11-14T04:55:27.0909167Z distributed/tensor/parallel/test_tp_examples 2023-11-14T04:55:27.0909838Z distributed/tensor/parallel/test_tp_random_state 2023-11-14T04:55:27.0910469Z distributed/tensor/parallel/test_tp_style 2023-11-14T04:55:27.0911180Z distributed/tensor/parallel/test_view_sharding_dim_change 2023-11-14T04:55:27.0911852Z distributed/test_c10d_common 2023-11-14T04:55:27.0912345Z distributed/test_c10d_functional_native 2023-11-14T04:55:27.0912904Z distributed/test_c10d_gloo 2023-11-14T04:55:27.0913384Z distributed/test_c10d_logger 2023-11-14T04:55:27.0913846Z distributed/test_c10d_nccl 2023-11-14T04:55:27.0914368Z distributed/test_c10d_object_collectives 2023-11-14T04:55:27.0915143Z distributed/test_c10d_pypg 2023-11-14T04:55:27.0915605Z distributed/test_c10d_spawn_gloo 2023-11-14T04:55:27.0916129Z distributed/test_c10d_spawn_nccl 2023-11-14T04:55:27.0916653Z distributed/test_c10d_spawn_ucc 2023-11-14T04:55:27.0917144Z distributed/test_c10d_ucc 2023-11-14T04:55:27.0917634Z distributed/test_collective_utils 2023-11-14T04:55:27.0918215Z distributed/test_compute_comm_reordering 2023-11-14T04:55:27.0918764Z distributed/test_data_parallel 2023-11-14T04:55:27.0919273Z distributed/test_device_mesh 2023-11-14T04:55:27.0919785Z distributed/test_distributed_spawn 2023-11-14T04:55:27.0920319Z distributed/test_dynamo_distributed 2023-11-14T04:55:27.0920858Z distributed/test_fake_pg 2023-11-14T04:55:27.0921333Z distributed/test_functional_api 2023-11-14T04:55:27.0921843Z distributed/test_inductor_collectives 2023-11-14T04:55:27.0922437Z distributed/test_launcher 2023-11-14T04:55:27.0923064Z distributed/test_multi_threaded_pg 2023-11-14T04:55:27.0923585Z distributed/test_nccl 2023-11-14T04:55:27.0924042Z distributed/test_pg_wrapper 2023-11-14T04:55:27.0924521Z distributed/test_store 2023-11-14T04:55:27.0925197Z Heuristic PreviouslyFailedInPR identified 1 tests to prioritize (0.55%%) 2023-11-14T04:55:27.0925980Z High Relevance tests (1): 2023-11-14T04:55:27.0926468Z distributed/test_distributed_spawn 2023-11-14T04:55:27.0926997Z Unranked Relevance tests (181): 2023-11-14T04:55:27.0927653Z distributed/_composable/fully_shard/test_fully_shard_compile 2023-11-14T04:55:27.0928465Z distributed/_composable/fully_shard/test_fully_shard_init 2023-11-14T04:55:27.0929319Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 2023-11-14T04:55:27.0930226Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 2023-11-14T04:55:27.0931169Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 2023-11-14T04:55:27.0932054Z distributed/_composable/fully_shard/test_fully_shard_runtime 2023-11-14T04:55:27.0932834Z distributed/_composable/fully_shard/test_fully_shard_util 2023-11-14T04:55:27.0933528Z distributed/_composable/test_checkpoint 2023-11-14T04:55:27.0934114Z distributed/_composable/test_compose 2023-11-14T04:55:27.0934653Z distributed/_composable/test_contract 2023-11-14T04:55:27.0935233Z distributed/_composable/test_replicate 2023-11-14T04:55:27.0935895Z distributed/_shard/sharded_optim/test_sharded_optim 2023-11-14T04:55:27.0936597Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 2023-11-14T04:55:27.0937331Z distributed/_shard/sharded_tensor/ops/test_embedding 2023-11-14T04:55:27.0938092Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2023-11-14T04:55:27.0938788Z distributed/_shard/sharded_tensor/ops/test_init 2023-11-14T04:55:27.0939488Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2023-11-14T04:55:27.0940185Z distributed/_shard/sharded_tensor/test_logger 2023-11-14T04:55:27.0940850Z distributed/_shard/sharded_tensor/test_sharded_tensor 2023-11-14T04:55:27.0941630Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2023-11-14T04:55:27.0942411Z distributed/_shard/sharding_plan/test_sharding_plan 2023-11-14T04:55:27.0943103Z distributed/_shard/sharding_spec/test_sharding_spec 2023-11-14T04:55:27.0943733Z distributed/_shard/test_sharder 2023-11-14T04:55:27.0944271Z distributed/_spmd/test_data_parallel 2023-11-14T04:55:27.0944994Z distributed/_spmd/test_graph_utils 2023-11-14T04:55:27.0945539Z distributed/_spmd/test_tracing 2023-11-14T04:55:27.0946065Z distributed/_spmd/test_transformation 2023-11-14T04:55:27.0946636Z distributed/_tensor/debug/test_op_coverage 2023-11-14T04:55:27.0947305Z distributed/_tensor/experimental/test_tp_transform 2023-11-14T04:55:27.0947938Z distributed/_tensor/test_api 2023-11-14T04:55:27.0948454Z distributed/_tensor/test_common_rules 2023-11-14T04:55:27.0948982Z distributed/_tensor/test_dtensor 2023-11-14T04:55:27.0950028Z distributed/_tensor/test_dtensor_compile 2023-11-14T04:55:27.0950679Z distributed/_tensor/test_dtensor_ops 2023-11-14T04:55:27.0951229Z distributed/_tensor/test_embedding_ops 2023-11-14T04:55:27.0951788Z distributed/_tensor/test_init 2023-11-14T04:55:27.0952296Z distributed/_tensor/test_math_ops 2023-11-14T04:55:27.0952811Z distributed/_tensor/test_matrix_ops 2023-11-14T04:55:27.0953366Z distributed/_tensor/test_op_strategy 2023-11-14T04:55:27.0953937Z distributed/_tensor/test_optimizers 2023-11-14T04:55:27.0954476Z distributed/_tensor/test_pointwise_ops 2023-11-14T04:55:27.0955268Z distributed/_tensor/test_random_ops 2023-11-14T04:55:27.0955823Z distributed/_tensor/test_redistribute 2023-11-14T04:55:27.0956364Z distributed/_tensor/test_tensor_ops 2023-11-14T04:55:27.0956905Z distributed/_tensor/test_utils 2023-11-14T04:55:27.0957421Z distributed/_tensor/test_view_ops 2023-11-14T04:55:27.0957951Z distributed/_tensor/test_xla_integration 2023-11-14T04:55:27.0958724Z distributed/_tools/test_memory_tracker 2023-11-14T04:55:27.0959406Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2023-11-14T04:55:27.0960128Z distributed/algorithms/quantization/test_quantization 2023-11-14T04:55:27.0960788Z distributed/algorithms/test_join 2023-11-14T04:55:27.0961384Z distributed/checkpoint/e2e/test_e2e_save_and_load 2023-11-14T04:55:27.0962031Z distributed/checkpoint/e2e/test_fine_tuning 2023-11-14T04:55:27.0962664Z distributed/checkpoint/e2e/test_pipeline 2023-11-14T04:55:27.0963266Z distributed/checkpoint/test_checkpoint 2023-11-14T04:55:27.0963840Z distributed/checkpoint/test_dedup_tensors 2023-11-14T04:55:27.0964485Z distributed/checkpoint/test_dtensor_checkpoint 2023-11-14T04:55:27.0965150Z distributed/checkpoint/test_dtensor_resharding 2023-11-14T04:55:27.0965817Z distributed/checkpoint/test_file_system_checkpoint 2023-11-14T04:55:27.0966564Z distributed/checkpoint/test_file_system_checkpoint_cpu 2023-11-14T04:55:27.0967272Z distributed/checkpoint/test_fsdp_model_state 2023-11-14T04:55:27.0967898Z distributed/checkpoint/test_fsdp_optim_state 2023-11-14T04:55:27.0968628Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 2023-11-14T04:55:27.0969322Z distributed/checkpoint/test_fsspec 2023-11-14T04:55:27.0969885Z distributed/checkpoint/test_hsdp_checkpoint 2023-11-14T04:55:27.0970507Z distributed/checkpoint/test_nested_dict 2023-11-14T04:55:27.0971086Z distributed/checkpoint/test_planner 2023-11-14T04:55:27.0971635Z distributed/checkpoint/test_state_dict 2023-11-14T04:55:27.0972247Z distributed/checkpoint/test_state_dict_utils 2023-11-14T04:55:27.0972873Z distributed/checkpoint/test_tp_checkpoint 2023-11-14T04:55:27.0973449Z distributed/checkpoint/test_traverse 2023-11-14T04:55:27.0974004Z distributed/checkpoint/test_utils 2023-11-14T04:55:27.0974544Z distributed/elastic/events/lib_test 2023-11-14T04:55:27.0975076Z distributed/elastic/metrics/api_test 2023-11-14T04:55:27.0975687Z distributed/elastic/multiprocessing/api_test 2023-11-14T04:55:27.0976306Z distributed/elastic/timer/api_test 2023-11-14T04:55:27.0976878Z distributed/elastic/timer/local_timer_example 2023-11-14T04:55:27.0977529Z distributed/elastic/timer/local_timer_test 2023-11-14T04:55:27.0978165Z distributed/elastic/utils/distributed_test 2023-11-14T04:55:27.0978748Z distributed/elastic/utils/logging_test 2023-11-14T04:55:27.0979318Z distributed/elastic/utils/util_test 2023-11-14T04:55:27.0980062Z distributed/fsdp/test_checkpoint_wrapper 2023-11-14T04:55:27.0980696Z distributed/fsdp/test_distributed_checkpoint 2023-11-14T04:55:27.0981273Z distributed/fsdp/test_fsdp_apply 2023-11-14T04:55:27.0981852Z distributed/fsdp/test_fsdp_backward_prefetch 2023-11-14T04:55:27.0982459Z distributed/fsdp/test_fsdp_checkpoint 2023-11-14T04:55:27.0983033Z distributed/fsdp/test_fsdp_clip_grad_norm 2023-11-14T04:55:27.0983612Z distributed/fsdp/test_fsdp_comm 2023-11-14T04:55:27.0984144Z distributed/fsdp/test_fsdp_comm_hooks 2023-11-14T04:55:27.0984685Z distributed/fsdp/test_fsdp_core 2023-11-14T04:55:27.0985261Z distributed/fsdp/test_fsdp_dtensor_state_dict 2023-11-14T04:55:27.0985876Z distributed/fsdp/test_fsdp_exec_order 2023-11-14T04:55:27.0986416Z distributed/fsdp/test_fsdp_fine_tune 2023-11-14T04:55:27.0986993Z distributed/fsdp/test_fsdp_flatten_params 2023-11-14T04:55:27.0987627Z distributed/fsdp/test_fsdp_freezing_weights 2023-11-14T04:55:27.0988189Z distributed/fsdp/test_fsdp_fx 2023-11-14T04:55:27.0988720Z distributed/fsdp/test_fsdp_grad_acc 2023-11-14T04:55:27.0989290Z distributed/fsdp/test_fsdp_hybrid_shard 2023-11-14T04:55:27.0989871Z distributed/fsdp/test_fsdp_ignored_modules 2023-11-14T04:55:27.0990461Z distributed/fsdp/test_fsdp_input 2023-11-14T04:55:27.0990988Z distributed/fsdp/test_fsdp_memory 2023-11-14T04:55:27.0991503Z distributed/fsdp/test_fsdp_meta 2023-11-14T04:55:27.0992024Z distributed/fsdp/test_fsdp_misc 2023-11-14T04:55:27.0992579Z distributed/fsdp/test_fsdp_mixed_precision 2023-11-14T04:55:27.0993314Z distributed/fsdp/test_fsdp_multiple_forward 2023-11-14T04:55:27.0993977Z distributed/fsdp/test_fsdp_multiple_wrapping 2023-11-14T04:55:27.0994769Z distributed/fsdp/test_fsdp_optim_state 2023-11-14T04:55:27.0995330Z distributed/fsdp/test_fsdp_overlap 2023-11-14T04:55:27.0995884Z distributed/fsdp/test_fsdp_pure_fp16 2023-11-14T04:55:27.0996500Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2023-11-14T04:55:27.0997115Z distributed/fsdp/test_fsdp_state_dict 2023-11-14T04:55:27.0997708Z distributed/fsdp/test_fsdp_tp_integration 2023-11-14T04:55:27.0998311Z distributed/fsdp/test_fsdp_traversal 2023-11-14T04:55:27.0998840Z distributed/fsdp/test_fsdp_uneven 2023-11-14T04:55:27.0999407Z distributed/fsdp/test_fsdp_unshard_params 2023-11-14T04:55:27.1000034Z distributed/fsdp/test_fsdp_use_orig_params 2023-11-14T04:55:27.1000653Z distributed/fsdp/test_hsdp_dtensor_state_dict 2023-11-14T04:55:27.1001255Z distributed/fsdp/test_shard_utils 2023-11-14T04:55:27.1001792Z distributed/fsdp/test_utils 2023-11-14T04:55:27.1002265Z distributed/fsdp/test_wrap 2023-11-14T04:55:27.1002773Z distributed/nn/jit/test_instantiator 2023-11-14T04:55:27.1003413Z distributed/optim/test_apply_optimizer_in_backward 2023-11-14T04:55:27.1004048Z distributed/optim/test_named_optimizer 2023-11-14T04:55:27.1004679Z distributed/optim/test_zero_redundancy_optimizer 2023-11-14T04:55:27.1005333Z distributed/pipeline/sync/skip/test_api 2023-11-14T04:55:27.1005922Z distributed/pipeline/sync/skip/test_gpipe 2023-11-14T04:55:27.1006626Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2023-11-14T04:55:27.1007330Z distributed/pipeline/sync/skip/test_leak 2023-11-14T04:55:27.1007932Z distributed/pipeline/sync/skip/test_portal 2023-11-14T04:55:27.1008588Z distributed/pipeline/sync/skip/test_stash_pop 2023-11-14T04:55:27.1009239Z distributed/pipeline/sync/skip/test_tracker 2023-11-14T04:55:27.1009911Z distributed/pipeline/sync/skip/test_verify_skippables 2023-11-14T04:55:27.1010590Z distributed/pipeline/sync/test_balance 2023-11-14T04:55:27.1011179Z distributed/pipeline/sync/test_bugs 2023-11-14T04:55:27.1011738Z distributed/pipeline/sync/test_checkpoint 2023-11-14T04:55:27.1012337Z distributed/pipeline/sync/test_copy 2023-11-14T04:55:27.1012968Z distributed/pipeline/sync/test_deferred_batch_norm 2023-11-14T04:55:27.1013619Z distributed/pipeline/sync/test_dependency 2023-11-14T04:55:27.1014224Z distributed/pipeline/sync/test_inplace 2023-11-14T04:55:27.1014993Z distributed/pipeline/sync/test_microbatch 2023-11-14T04:55:27.1015588Z distributed/pipeline/sync/test_phony 2023-11-14T04:55:27.1016125Z distributed/pipeline/sync/test_pipe 2023-11-14T04:55:27.1016704Z distributed/pipeline/sync/test_pipeline 2023-11-14T04:55:27.1017287Z distributed/pipeline/sync/test_stream 2023-11-14T04:55:27.1017869Z distributed/pipeline/sync/test_transparency 2023-11-14T04:55:27.1018480Z distributed/pipeline/sync/test_worker 2023-11-14T04:55:27.1019083Z distributed/rpc/cuda/test_tensorpipe_agent 2023-11-14T04:55:27.1019662Z distributed/rpc/test_faulty_agent 2023-11-14T04:55:27.1020204Z distributed/rpc/test_share_memory 2023-11-14T04:55:27.1020761Z distributed/rpc/test_tensorpipe_agent 2023-11-14T04:55:27.1021361Z distributed/tensor/parallel/test_ddp_2d_parallel 2023-11-14T04:55:27.1022061Z distributed/tensor/parallel/test_fsdp_2d_parallel 2023-11-14T04:55:27.1022822Z distributed/tensor/parallel/test_parallelize_api 2023-11-14T04:55:27.1023487Z distributed/tensor/parallel/test_tp_examples 2023-11-14T04:55:27.1024155Z distributed/tensor/parallel/test_tp_random_state 2023-11-14T04:55:27.1024810Z distributed/tensor/parallel/test_tp_style 2023-11-14T04:55:27.1025499Z distributed/tensor/parallel/test_view_sharding_dim_change 2023-11-14T04:55:27.1026176Z distributed/test_c10d_common 2023-11-14T04:55:27.1026703Z distributed/test_c10d_functional_native 2023-11-14T04:55:27.1027232Z distributed/test_c10d_gloo 2023-11-14T04:55:27.1027712Z distributed/test_c10d_logger 2023-11-14T04:55:27.1028325Z distributed/test_c10d_nccl 2023-11-14T04:55:27.1028840Z distributed/test_c10d_object_collectives 2023-11-14T04:55:27.1029413Z distributed/test_c10d_pypg 2023-11-14T04:55:27.1029904Z distributed/test_c10d_spawn_gloo 2023-11-14T04:55:27.1030400Z distributed/test_c10d_spawn_nccl 2023-11-14T04:55:27.1030924Z distributed/test_c10d_spawn_ucc 2023-11-14T04:55:27.1031439Z distributed/test_c10d_ucc 2023-11-14T04:55:27.1031913Z distributed/test_collective_utils 2023-11-14T04:55:27.1032484Z distributed/test_compute_comm_reordering 2023-11-14T04:55:27.1033069Z distributed/test_data_parallel 2023-11-14T04:55:27.1033560Z distributed/test_device_mesh 2023-11-14T04:55:27.1034069Z distributed/test_dynamo_distributed 2023-11-14T04:55:27.1034778Z distributed/test_fake_pg 2023-11-14T04:55:27.1035252Z distributed/test_functional_api 2023-11-14T04:55:27.1035784Z distributed/test_inductor_collectives 2023-11-14T04:55:27.1036330Z distributed/test_launcher 2023-11-14T04:55:27.1036806Z distributed/test_multi_threaded_pg 2023-11-14T04:55:27.1037335Z distributed/test_nccl 2023-11-14T04:55:27.1037795Z distributed/test_pg_wrapper 2023-11-14T04:55:27.1038248Z distributed/test_store 2023-11-14T04:55:27.1038899Z Heuristic EditedByPR identified 0 tests to prioritize (0.00%%) 2023-11-14T04:55:27.3199615Z Heuristic CorrelatedWithHistoricalFailures identified 49 tests to prioritize (26.92%%) 2023-11-14T04:55:27.3201145Z Probable Relevance tests (49): 2023-11-14T04:55:27.3201980Z distributed/test_distributed_spawn 2023-11-14T04:55:27.3202540Z distributed/fsdp/test_fsdp_optim_state 2023-11-14T04:55:27.3203145Z distributed/fsdp/test_fsdp_core 2023-11-14T04:55:27.3203637Z distributed/rpc/test_tensorpipe_agent 2023-11-14T04:55:27.3204641Z distributed/test_multi_threaded_pg 2023-11-14T04:55:27.3205641Z distributed/test_c10d_nccl 2023-11-14T04:55:27.3206583Z distributed/tensor/parallel/test_fsdp_2d_parallel 2023-11-14T04:55:27.3207346Z distributed/_tensor/test_pointwise_ops 2023-11-14T04:55:27.3208175Z distributed/_tensor/test_dtensor 2023-11-14T04:55:27.3209086Z distributed/test_c10d_ucc 2023-11-14T04:55:27.3209740Z distributed/test_pg_wrapper 2023-11-14T04:55:27.3210433Z distributed/test_dynamo_distributed 2023-11-14T04:55:27.3211382Z distributed/fsdp/test_fsdp_use_orig_params 2023-11-14T04:55:27.3212339Z distributed/fsdp/test_fsdp_unshard_params 2023-11-14T04:55:27.3213351Z distributed/test_c10d_gloo 2023-11-14T04:55:27.3214809Z distributed/test_launcher 2023-11-14T04:55:27.3215447Z distributed/test_inductor_collectives 2023-11-14T04:55:27.3216432Z distributed/_shard/sharded_optim/test_sharded_optim 2023-11-14T04:55:27.3217529Z distributed/_tensor/test_dtensor_ops 2023-11-14T04:55:27.3218619Z distributed/rpc/cuda/test_tensorpipe_agent 2023-11-14T04:55:27.3219260Z distributed/test_c10d_common 2023-11-14T04:55:27.3220438Z distributed/_composable/fully_shard/test_fully_shard_runtime 2023-11-14T04:55:27.3221396Z distributed/fsdp/test_fsdp_state_dict 2023-11-14T04:55:27.3222437Z distributed/fsdp/test_hsdp_dtensor_state_dict 2023-11-14T04:55:27.3223119Z distributed/test_c10d_object_collectives 2023-11-14T04:55:27.3224287Z distributed/fsdp/test_fsdp_grad_acc 2023-11-14T04:55:27.3225640Z distributed/_composable/fully_shard/test_fully_shard_compile 2023-11-14T04:55:27.3226808Z distributed/_tensor/test_tensor_ops 2023-11-14T04:55:27.3227429Z distributed/fsdp/test_fsdp_flatten_params 2023-11-14T04:55:27.3228282Z distributed/fsdp/test_fsdp_exec_order 2023-11-14T04:55:27.3229000Z distributed/fsdp/test_checkpoint_wrapper 2023-11-14T04:55:27.3229750Z distributed/test_data_parallel 2023-11-14T04:55:27.3230706Z distributed/_spmd/test_tracing 2023-11-14T04:55:27.3231525Z distributed/fsdp/test_fsdp_mixed_precision 2023-11-14T04:55:27.3232319Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 2023-11-14T04:55:27.3233092Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2023-11-14T04:55:27.3233703Z distributed/fsdp/test_fsdp_fx 2023-11-14T04:55:27.3234474Z distributed/fsdp/test_fsdp_memory 2023-11-14T04:55:27.3235486Z distributed/_tensor/test_dtensor_compile 2023-11-14T04:55:27.3236098Z distributed/test_compute_comm_reordering 2023-11-14T04:55:27.3236687Z distributed/_tensor/debug/test_op_coverage 2023-11-14T04:55:27.3237314Z distributed/fsdp/test_fsdp_ignored_modules 2023-11-14T04:55:27.3237951Z distributed/elastic/utils/distributed_test 2023-11-14T04:55:27.3238551Z distributed/checkpoint/test_state_dict 2023-11-14T04:55:27.3239119Z distributed/_tensor/test_utils 2023-11-14T04:55:27.3239721Z distributed/checkpoint/test_file_system_checkpoint 2023-11-14T04:55:27.3240444Z distributed/checkpoint/test_file_system_checkpoint_cpu 2023-11-14T04:55:27.3241129Z distributed/checkpoint/test_planner 2023-11-14T04:55:27.3241802Z distributed/_composable/fully_shard/test_fully_shard_init 2023-11-14T04:55:27.3242448Z Unranked Relevance tests (133): 2023-11-14T04:55:27.3243163Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 2023-11-14T04:55:27.3244111Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 2023-11-14T04:55:27.3244974Z distributed/_composable/fully_shard/test_fully_shard_util 2023-11-14T04:55:27.3245647Z distributed/_composable/test_checkpoint 2023-11-14T04:55:27.3246230Z distributed/_composable/test_compose 2023-11-14T04:55:27.3246798Z distributed/_composable/test_contract 2023-11-14T04:55:27.3247367Z distributed/_composable/test_replicate 2023-11-14T04:55:27.3248025Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 2023-11-14T04:55:27.3248763Z distributed/_shard/sharded_tensor/ops/test_embedding 2023-11-14T04:55:27.3249494Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2023-11-14T04:55:27.3250215Z distributed/_shard/sharded_tensor/ops/test_init 2023-11-14T04:55:27.3250921Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2023-11-14T04:55:27.3251588Z distributed/_shard/sharded_tensor/test_logger 2023-11-14T04:55:27.3252285Z distributed/_shard/sharded_tensor/test_sharded_tensor 2023-11-14T04:55:27.3253064Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2023-11-14T04:55:27.3253815Z distributed/_shard/sharding_plan/test_sharding_plan 2023-11-14T04:55:27.3254529Z distributed/_shard/sharding_spec/test_sharding_spec 2023-11-14T04:55:27.3255161Z distributed/_shard/test_sharder 2023-11-14T04:55:27.3255669Z distributed/_spmd/test_data_parallel 2023-11-14T04:55:27.3256416Z distributed/_spmd/test_graph_utils 2023-11-14T04:55:27.3256983Z distributed/_spmd/test_transformation 2023-11-14T04:55:27.3257608Z distributed/_tensor/experimental/test_tp_transform 2023-11-14T04:55:27.3258244Z distributed/_tensor/test_api 2023-11-14T04:55:27.3258755Z distributed/_tensor/test_common_rules 2023-11-14T04:55:27.3259304Z distributed/_tensor/test_embedding_ops 2023-11-14T04:55:27.3259859Z distributed/_tensor/test_init 2023-11-14T04:55:27.3260375Z distributed/_tensor/test_math_ops 2023-11-14T04:55:27.3260892Z distributed/_tensor/test_matrix_ops 2023-11-14T04:55:27.3261454Z distributed/_tensor/test_op_strategy 2023-11-14T04:55:27.3262016Z distributed/_tensor/test_optimizers 2023-11-14T04:55:27.3262539Z distributed/_tensor/test_random_ops 2023-11-14T04:55:27.3263087Z distributed/_tensor/test_redistribute 2023-11-14T04:55:27.3263646Z distributed/_tensor/test_view_ops 2023-11-14T04:55:27.3264185Z distributed/_tensor/test_xla_integration 2023-11-14T04:55:27.3264784Z distributed/_tools/test_memory_tracker 2023-11-14T04:55:27.3265436Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2023-11-14T04:55:27.3266157Z distributed/algorithms/quantization/test_quantization 2023-11-14T04:55:27.3266816Z distributed/algorithms/test_join 2023-11-14T04:55:27.3267415Z distributed/checkpoint/e2e/test_e2e_save_and_load 2023-11-14T04:55:27.3268064Z distributed/checkpoint/e2e/test_fine_tuning 2023-11-14T04:55:27.3268685Z distributed/checkpoint/e2e/test_pipeline 2023-11-14T04:55:27.3269287Z distributed/checkpoint/test_checkpoint 2023-11-14T04:55:27.3269994Z distributed/checkpoint/test_dedup_tensors 2023-11-14T04:55:27.3270664Z distributed/checkpoint/test_dtensor_checkpoint 2023-11-14T04:55:27.3271338Z distributed/checkpoint/test_dtensor_resharding 2023-11-14T04:55:27.3271997Z distributed/checkpoint/test_fsdp_model_state 2023-11-14T04:55:27.3272615Z distributed/checkpoint/test_fsdp_optim_state 2023-11-14T04:55:27.3273343Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 2023-11-14T04:55:27.3274042Z distributed/checkpoint/test_fsspec 2023-11-14T04:55:27.3274807Z distributed/checkpoint/test_hsdp_checkpoint 2023-11-14T04:55:27.3275445Z distributed/checkpoint/test_nested_dict 2023-11-14T04:55:27.3276061Z distributed/checkpoint/test_state_dict_utils 2023-11-14T04:55:27.3276663Z distributed/checkpoint/test_tp_checkpoint 2023-11-14T04:55:27.3277255Z distributed/checkpoint/test_traverse 2023-11-14T04:55:27.3277816Z distributed/checkpoint/test_utils 2023-11-14T04:55:27.3278340Z distributed/elastic/events/lib_test 2023-11-14T04:55:27.3278904Z distributed/elastic/metrics/api_test 2023-11-14T04:55:27.3279515Z distributed/elastic/multiprocessing/api_test 2023-11-14T04:55:27.3280091Z distributed/elastic/timer/api_test 2023-11-14T04:55:27.3280685Z distributed/elastic/timer/local_timer_example 2023-11-14T04:55:27.3281323Z distributed/elastic/timer/local_timer_test 2023-11-14T04:55:27.3281900Z distributed/elastic/utils/logging_test 2023-11-14T04:55:27.3282485Z distributed/elastic/utils/util_test 2023-11-14T04:55:27.3283087Z distributed/fsdp/test_distributed_checkpoint 2023-11-14T04:55:27.3283664Z distributed/fsdp/test_fsdp_apply 2023-11-14T04:55:27.3284246Z distributed/fsdp/test_fsdp_backward_prefetch 2023-11-14T04:55:27.3284907Z distributed/fsdp/test_fsdp_checkpoint 2023-11-14T04:55:27.3285501Z distributed/fsdp/test_fsdp_clip_grad_norm 2023-11-14T04:55:27.3286061Z distributed/fsdp/test_fsdp_comm 2023-11-14T04:55:27.3286603Z distributed/fsdp/test_fsdp_comm_hooks 2023-11-14T04:55:27.3287216Z distributed/fsdp/test_fsdp_dtensor_state_dict 2023-11-14T04:55:27.3287803Z distributed/fsdp/test_fsdp_fine_tune 2023-11-14T04:55:27.3288407Z distributed/fsdp/test_fsdp_freezing_weights 2023-11-14T04:55:27.3289027Z distributed/fsdp/test_fsdp_hybrid_shard 2023-11-14T04:55:27.3289572Z distributed/fsdp/test_fsdp_input 2023-11-14T04:55:27.3290106Z distributed/fsdp/test_fsdp_meta 2023-11-14T04:55:27.3290633Z distributed/fsdp/test_fsdp_misc 2023-11-14T04:55:27.3291355Z distributed/fsdp/test_fsdp_multiple_forward 2023-11-14T04:55:27.3292001Z distributed/fsdp/test_fsdp_multiple_wrapping 2023-11-14T04:55:27.3292605Z distributed/fsdp/test_fsdp_overlap 2023-11-14T04:55:27.3293137Z distributed/fsdp/test_fsdp_pure_fp16 2023-11-14T04:55:27.3293720Z distributed/fsdp/test_fsdp_tp_integration 2023-11-14T04:55:27.3294316Z distributed/fsdp/test_fsdp_traversal 2023-11-14T04:55:27.3294846Z distributed/fsdp/test_fsdp_uneven 2023-11-14T04:55:27.3295392Z distributed/fsdp/test_shard_utils 2023-11-14T04:55:27.3295927Z distributed/fsdp/test_utils 2023-11-14T04:55:27.3296395Z distributed/fsdp/test_wrap 2023-11-14T04:55:27.3296900Z distributed/nn/jit/test_instantiator 2023-11-14T04:55:27.3297532Z distributed/optim/test_apply_optimizer_in_backward 2023-11-14T04:55:27.3298162Z distributed/optim/test_named_optimizer 2023-11-14T04:55:27.3298794Z distributed/optim/test_zero_redundancy_optimizer 2023-11-14T04:55:27.3299537Z distributed/pipeline/sync/skip/test_api 2023-11-14T04:55:27.3300133Z distributed/pipeline/sync/skip/test_gpipe 2023-11-14T04:55:27.3300836Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2023-11-14T04:55:27.3301541Z distributed/pipeline/sync/skip/test_leak 2023-11-14T04:55:27.3302139Z distributed/pipeline/sync/skip/test_portal 2023-11-14T04:55:27.3302785Z distributed/pipeline/sync/skip/test_stash_pop 2023-11-14T04:55:27.3303437Z distributed/pipeline/sync/skip/test_tracker 2023-11-14T04:55:27.3304130Z distributed/pipeline/sync/skip/test_verify_skippables 2023-11-14T04:55:27.3304905Z distributed/pipeline/sync/test_balance 2023-11-14T04:55:27.3305504Z distributed/pipeline/sync/test_bugs 2023-11-14T04:55:27.3306083Z distributed/pipeline/sync/test_checkpoint 2023-11-14T04:55:27.3306659Z distributed/pipeline/sync/test_copy 2023-11-14T04:55:27.3307290Z distributed/pipeline/sync/test_deferred_batch_norm 2023-11-14T04:55:27.3307959Z distributed/pipeline/sync/test_dependency 2023-11-14T04:55:27.3308546Z distributed/pipeline/sync/test_inplace 2023-11-14T04:55:27.3309142Z distributed/pipeline/sync/test_microbatch 2023-11-14T04:55:27.3309738Z distributed/pipeline/sync/test_phony 2023-11-14T04:55:27.3310275Z distributed/pipeline/sync/test_pipe 2023-11-14T04:55:27.3310850Z distributed/pipeline/sync/test_pipeline 2023-11-14T04:55:27.3311431Z distributed/pipeline/sync/test_stream 2023-11-14T04:55:27.3312008Z distributed/pipeline/sync/test_transparency 2023-11-14T04:55:27.3312621Z distributed/pipeline/sync/test_worker 2023-11-14T04:55:27.3313182Z distributed/rpc/test_faulty_agent 2023-11-14T04:55:27.3313701Z distributed/rpc/test_share_memory 2023-11-14T04:55:27.3314306Z distributed/tensor/parallel/test_ddp_2d_parallel 2023-11-14T04:55:27.3315209Z distributed/tensor/parallel/test_parallelize_api 2023-11-14T04:55:27.3315866Z distributed/tensor/parallel/test_tp_examples 2023-11-14T04:55:27.3316530Z distributed/tensor/parallel/test_tp_random_state 2023-11-14T04:55:27.3317202Z distributed/tensor/parallel/test_tp_style 2023-11-14T04:55:27.3317889Z distributed/tensor/parallel/test_view_sharding_dim_change 2023-11-14T04:55:27.3318589Z distributed/test_c10d_functional_native 2023-11-14T04:55:27.3319154Z distributed/test_c10d_logger 2023-11-14T04:55:27.3319617Z distributed/test_c10d_pypg 2023-11-14T04:55:27.3320106Z distributed/test_c10d_spawn_gloo 2023-11-14T04:55:27.3320631Z distributed/test_c10d_spawn_nccl 2023-11-14T04:55:27.3321125Z distributed/test_c10d_spawn_ucc 2023-11-14T04:55:27.3321651Z distributed/test_collective_utils 2023-11-14T04:55:27.3322198Z distributed/test_device_mesh 2023-11-14T04:55:27.3322720Z distributed/test_fake_pg 2023-11-14T04:55:27.3323199Z distributed/test_functional_api 2023-11-14T04:55:27.3323701Z distributed/test_nccl 2023-11-14T04:55:27.3324118Z distributed/test_store 2023-11-14T04:55:27.3873520Z Heuristic HistorialEditedFiles identified 63 tests to prioritize (34.62%%) 2023-11-14T04:55:27.3874369Z Probable Relevance tests (63): 2023-11-14T04:55:27.3875358Z distributed/test_c10d_ucc 2023-11-14T04:55:27.3875862Z distributed/test_c10d_spawn_ucc 2023-11-14T04:55:27.3876377Z distributed/test_c10d_nccl 2023-11-14T04:55:27.3876844Z distributed/test_data_parallel 2023-11-14T04:55:27.3877335Z distributed/test_nccl 2023-11-14T04:55:27.3877832Z distributed/rpc/cuda/test_tensorpipe_agent 2023-11-14T04:55:27.3878370Z distributed/test_store 2023-11-14T04:55:27.3878940Z distributed/algorithms/quantization/test_quantization 2023-11-14T04:55:27.3879646Z distributed/fsdp/test_fsdp_mixed_precision 2023-11-14T04:55:27.3880317Z distributed/optim/test_zero_redundancy_optimizer 2023-11-14T04:55:27.3880945Z distributed/test_c10d_gloo 2023-11-14T04:55:27.3881424Z distributed/test_c10d_spawn_gloo 2023-11-14T04:55:27.3881980Z distributed/test_c10d_common 2023-11-14T04:55:27.3882439Z distributed/fsdp/test_fsdp_comm_hooks 2023-11-14T04:55:27.3883074Z distributed/_shard/sharded_tensor/test_sharded_tensor 2023-11-14T04:55:27.3883830Z distributed/_shard/sharding_spec/test_sharding_spec 2023-11-14T04:55:27.3884489Z distributed/elastic/multiprocessing/api_test 2023-11-14T04:55:27.3885180Z distributed/elastic/timer/local_timer_example 2023-11-14T04:55:27.3885781Z distributed/fsdp/test_fsdp_meta 2023-11-14T04:55:27.3886261Z distributed/test_c10d_spawn_nccl 2023-11-14T04:55:27.3886823Z distributed/elastic/events/lib_test 2023-11-14T04:55:27.3887394Z distributed/pipeline/sync/skip/test_api 2023-11-14T04:55:27.3887973Z distributed/pipeline/sync/skip/test_gpipe 2023-11-14T04:55:27.3888829Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2023-11-14T04:55:27.3889558Z distributed/pipeline/sync/skip/test_leak 2023-11-14T04:55:27.3890127Z distributed/pipeline/sync/skip/test_portal 2023-11-14T04:55:27.3890797Z distributed/pipeline/sync/skip/test_stash_pop 2023-11-14T04:55:27.3891450Z distributed/pipeline/sync/skip/test_tracker 2023-11-14T04:55:27.3892118Z distributed/pipeline/sync/skip/test_verify_skippables 2023-11-14T04:55:27.3892810Z distributed/pipeline/sync/test_balance 2023-11-14T04:55:27.3893389Z distributed/pipeline/sync/test_bugs 2023-11-14T04:55:27.3893940Z distributed/pipeline/sync/test_checkpoint 2023-11-14T04:55:27.3894509Z distributed/pipeline/sync/test_copy 2023-11-14T04:55:27.3895147Z distributed/pipeline/sync/test_deferred_batch_norm 2023-11-14T04:55:27.3895792Z distributed/pipeline/sync/test_dependency 2023-11-14T04:55:27.3896397Z distributed/pipeline/sync/test_inplace 2023-11-14T04:55:27.3897000Z distributed/pipeline/sync/test_microbatch 2023-11-14T04:55:27.3897582Z distributed/pipeline/sync/test_phony 2023-11-14T04:55:27.3898144Z distributed/pipeline/sync/test_pipe 2023-11-14T04:55:27.3898723Z distributed/pipeline/sync/test_pipeline 2023-11-14T04:55:27.3899310Z distributed/pipeline/sync/test_stream 2023-11-14T04:55:27.3899888Z distributed/pipeline/sync/test_transparency 2023-11-14T04:55:27.3900529Z distributed/pipeline/sync/test_worker 2023-11-14T04:55:27.3901092Z distributed/test_dynamo_distributed 2023-11-14T04:55:27.3901644Z distributed/fsdp/test_fsdp_state_dict 2023-11-14T04:55:27.3902198Z distributed/fsdp/test_fsdp_optim_state 2023-11-14T04:55:27.3902765Z distributed/_tensor/test_view_ops 2023-11-14T04:55:27.3903276Z distributed/test_launcher 2023-11-14T04:55:27.3903784Z distributed/_composable/test_replicate 2023-11-14T04:55:27.3904451Z distributed/_shard/sharded_optim/test_sharded_optim 2023-11-14T04:55:27.3905041Z distributed/_tensor/test_dtensor 2023-11-14T04:55:27.3905664Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2023-11-14T04:55:27.3906315Z distributed/algorithms/test_join 2023-11-14T04:55:27.3906847Z distributed/checkpoint/test_checkpoint 2023-11-14T04:55:27.3907494Z distributed/checkpoint/test_file_system_checkpoint 2023-11-14T04:55:27.3908238Z distributed/checkpoint/test_file_system_checkpoint_cpu 2023-11-14T04:55:27.3908907Z distributed/fsdp/test_checkpoint_wrapper 2023-11-14T04:55:27.3909499Z distributed/rpc/test_share_memory 2023-11-14T04:55:27.3910270Z distributed/tensor/parallel/test_parallelize_api 2023-11-14T04:55:27.3910928Z distributed/tensor/parallel/test_tp_examples 2023-11-14T04:55:27.3911556Z distributed/test_c10d_object_collectives 2023-11-14T04:55:27.3912125Z distributed/test_c10d_pypg 2023-11-14T04:55:27.3912664Z distributed/test_pg_wrapper 2023-11-14T04:55:27.3913142Z Unranked Relevance tests (119): 2023-11-14T04:55:27.3913805Z distributed/_composable/fully_shard/test_fully_shard_compile 2023-11-14T04:55:27.3914903Z distributed/_composable/fully_shard/test_fully_shard_init 2023-11-14T04:55:27.3915768Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 2023-11-14T04:55:27.3916720Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 2023-11-14T04:55:27.3917654Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 2023-11-14T04:55:27.3918502Z distributed/_composable/fully_shard/test_fully_shard_runtime 2023-11-14T04:55:27.3919318Z distributed/_composable/fully_shard/test_fully_shard_util 2023-11-14T04:55:27.3920017Z distributed/_composable/test_checkpoint 2023-11-14T04:55:27.3920575Z distributed/_composable/test_compose 2023-11-14T04:55:27.3921137Z distributed/_composable/test_contract 2023-11-14T04:55:27.3921789Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 2023-11-14T04:55:27.3922573Z distributed/_shard/sharded_tensor/ops/test_embedding 2023-11-14T04:55:27.3923296Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2023-11-14T04:55:27.3924018Z distributed/_shard/sharded_tensor/ops/test_init 2023-11-14T04:55:27.3924857Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2023-11-14T04:55:27.3925559Z distributed/_shard/sharded_tensor/test_logger 2023-11-14T04:55:27.3926293Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2023-11-14T04:55:27.3927076Z distributed/_shard/sharding_plan/test_sharding_plan 2023-11-14T04:55:27.3927693Z distributed/_shard/test_sharder 2023-11-14T04:55:27.3928243Z distributed/_spmd/test_data_parallel 2023-11-14T04:55:27.3928807Z distributed/_spmd/test_graph_utils 2023-11-14T04:55:27.3929319Z distributed/_spmd/test_tracing 2023-11-14T04:55:27.3929855Z distributed/_spmd/test_transformation 2023-11-14T04:55:27.3930445Z distributed/_tensor/debug/test_op_coverage 2023-11-14T04:55:27.3931088Z distributed/_tensor/experimental/test_tp_transform 2023-11-14T04:55:27.3931717Z distributed/_tensor/test_api 2023-11-14T04:55:27.3932225Z distributed/_tensor/test_common_rules 2023-11-14T04:55:27.3932779Z distributed/_tensor/test_dtensor_compile 2023-11-14T04:55:27.3933373Z distributed/_tensor/test_dtensor_ops 2023-11-14T04:55:27.3933947Z distributed/_tensor/test_embedding_ops 2023-11-14T04:55:27.3934474Z distributed/_tensor/test_init 2023-11-14T04:55:27.3934979Z distributed/_tensor/test_math_ops 2023-11-14T04:55:27.3935516Z distributed/_tensor/test_matrix_ops 2023-11-14T04:55:27.3936046Z distributed/_tensor/test_op_strategy 2023-11-14T04:55:27.3936617Z distributed/_tensor/test_optimizers 2023-11-14T04:55:27.3937182Z distributed/_tensor/test_pointwise_ops 2023-11-14T04:55:27.3937723Z distributed/_tensor/test_random_ops 2023-11-14T04:55:27.3938271Z distributed/_tensor/test_redistribute 2023-11-14T04:55:27.3938832Z distributed/_tensor/test_tensor_ops 2023-11-14T04:55:27.3939344Z distributed/_tensor/test_utils 2023-11-14T04:55:27.3939875Z distributed/_tensor/test_xla_integration 2023-11-14T04:55:27.3940467Z distributed/_tools/test_memory_tracker 2023-11-14T04:55:27.3941067Z distributed/checkpoint/e2e/test_e2e_save_and_load 2023-11-14T04:55:27.3941743Z distributed/checkpoint/e2e/test_fine_tuning 2023-11-14T04:55:27.3942365Z distributed/checkpoint/e2e/test_pipeline 2023-11-14T04:55:27.3942950Z distributed/checkpoint/test_dedup_tensors 2023-11-14T04:55:27.3943598Z distributed/checkpoint/test_dtensor_checkpoint 2023-11-14T04:55:27.3944266Z distributed/checkpoint/test_dtensor_resharding 2023-11-14T04:55:27.3944900Z distributed/checkpoint/test_fsdp_model_state 2023-11-14T04:55:27.3945698Z distributed/checkpoint/test_fsdp_optim_state 2023-11-14T04:55:27.3946419Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 2023-11-14T04:55:27.3947082Z distributed/checkpoint/test_fsspec 2023-11-14T04:55:27.3947676Z distributed/checkpoint/test_hsdp_checkpoint 2023-11-14T04:55:27.3948293Z distributed/checkpoint/test_nested_dict 2023-11-14T04:55:27.3948849Z distributed/checkpoint/test_planner 2023-11-14T04:55:27.3949408Z distributed/checkpoint/test_state_dict 2023-11-14T04:55:27.3950018Z distributed/checkpoint/test_state_dict_utils 2023-11-14T04:55:27.3950655Z distributed/checkpoint/test_tp_checkpoint 2023-11-14T04:55:27.3951233Z distributed/checkpoint/test_traverse 2023-11-14T04:55:27.3951788Z distributed/checkpoint/test_utils 2023-11-14T04:55:27.3952331Z distributed/elastic/metrics/api_test 2023-11-14T04:55:27.3952858Z distributed/elastic/timer/api_test 2023-11-14T04:55:27.3953439Z distributed/elastic/timer/local_timer_test 2023-11-14T04:55:27.3954082Z distributed/elastic/utils/distributed_test 2023-11-14T04:55:27.3954894Z distributed/elastic/utils/logging_test 2023-11-14T04:55:27.3955448Z distributed/elastic/utils/util_test 2023-11-14T04:55:27.3956042Z distributed/fsdp/test_distributed_checkpoint 2023-11-14T04:55:27.3956615Z distributed/fsdp/test_fsdp_apply 2023-11-14T04:55:27.3957194Z distributed/fsdp/test_fsdp_backward_prefetch 2023-11-14T04:55:27.3957798Z distributed/fsdp/test_fsdp_checkpoint 2023-11-14T04:55:27.3958357Z distributed/fsdp/test_fsdp_clip_grad_norm 2023-11-14T04:55:27.3959092Z distributed/fsdp/test_fsdp_comm 2023-11-14T04:55:27.3959646Z distributed/fsdp/test_fsdp_core 2023-11-14T04:55:27.3960185Z distributed/fsdp/test_fsdp_dtensor_state_dict 2023-11-14T04:55:27.3960800Z distributed/fsdp/test_fsdp_exec_order 2023-11-14T04:55:27.3961362Z distributed/fsdp/test_fsdp_fine_tune 2023-11-14T04:55:27.3961915Z distributed/fsdp/test_fsdp_flatten_params 2023-11-14T04:55:27.3962545Z distributed/fsdp/test_fsdp_freezing_weights 2023-11-14T04:55:27.3963148Z distributed/fsdp/test_fsdp_fx 2023-11-14T04:55:27.3963641Z distributed/fsdp/test_fsdp_grad_acc 2023-11-14T04:55:27.3964204Z distributed/fsdp/test_fsdp_hybrid_shard 2023-11-14T04:55:27.3964812Z distributed/fsdp/test_fsdp_ignored_modules 2023-11-14T04:55:27.3965377Z distributed/fsdp/test_fsdp_input 2023-11-14T04:55:27.3965909Z distributed/fsdp/test_fsdp_memory 2023-11-14T04:55:27.3966440Z distributed/fsdp/test_fsdp_misc 2023-11-14T04:55:27.3966975Z distributed/fsdp/test_fsdp_multiple_forward 2023-11-14T04:55:27.3967624Z distributed/fsdp/test_fsdp_multiple_wrapping 2023-11-14T04:55:27.3968225Z distributed/fsdp/test_fsdp_overlap 2023-11-14T04:55:27.3968756Z distributed/fsdp/test_fsdp_pure_fp16 2023-11-14T04:55:27.3969363Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2023-11-14T04:55:27.3970002Z distributed/fsdp/test_fsdp_tp_integration 2023-11-14T04:55:27.3970569Z distributed/fsdp/test_fsdp_traversal 2023-11-14T04:55:27.3971136Z distributed/fsdp/test_fsdp_uneven 2023-11-14T04:55:27.3971704Z distributed/fsdp/test_fsdp_unshard_params 2023-11-14T04:55:27.3972300Z distributed/fsdp/test_fsdp_use_orig_params 2023-11-14T04:55:27.3972940Z distributed/fsdp/test_hsdp_dtensor_state_dict 2023-11-14T04:55:27.3973538Z distributed/fsdp/test_shard_utils 2023-11-14T04:55:27.3974038Z distributed/fsdp/test_utils 2023-11-14T04:55:27.3974525Z distributed/fsdp/test_wrap 2023-11-14T04:55:27.3975024Z distributed/nn/jit/test_instantiator 2023-11-14T04:55:27.3975632Z distributed/optim/test_apply_optimizer_in_backward 2023-11-14T04:55:27.3976295Z distributed/optim/test_named_optimizer 2023-11-14T04:55:27.3976858Z distributed/rpc/test_faulty_agent 2023-11-14T04:55:27.3977381Z distributed/rpc/test_tensorpipe_agent 2023-11-14T04:55:27.3978005Z distributed/tensor/parallel/test_ddp_2d_parallel 2023-11-14T04:55:27.3978698Z distributed/tensor/parallel/test_fsdp_2d_parallel 2023-11-14T04:55:27.3979372Z distributed/tensor/parallel/test_tp_random_state 2023-11-14T04:55:27.3980189Z distributed/tensor/parallel/test_tp_style 2023-11-14T04:55:27.3980896Z distributed/tensor/parallel/test_view_sharding_dim_change 2023-11-14T04:55:27.3981568Z distributed/test_c10d_functional_native 2023-11-14T04:55:27.3982130Z distributed/test_c10d_logger 2023-11-14T04:55:27.3982630Z distributed/test_collective_utils 2023-11-14T04:55:27.3983168Z distributed/test_compute_comm_reordering 2023-11-14T04:55:27.3983733Z distributed/test_device_mesh 2023-11-14T04:55:27.3984235Z distributed/test_distributed_spawn 2023-11-14T04:55:27.3984749Z distributed/test_fake_pg 2023-11-14T04:55:27.3985228Z distributed/test_functional_api 2023-11-14T04:55:27.3985759Z distributed/test_inductor_collectives 2023-11-14T04:55:27.3986298Z distributed/test_multi_threaded_pg 2023-11-14T04:55:27.4875515Z Heuristic Profiling identified 133 tests to prioritize (73.08%%) 2023-11-14T04:55:27.4876692Z Probable Relevance tests (133): 2023-11-14T04:55:27.4878003Z distributed/_composable/fully_shard/test_fully_shard_compile 2023-11-14T04:55:27.4879483Z distributed/_composable/fully_shard/test_fully_shard_init 2023-11-14T04:55:27.4880805Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 2023-11-14T04:55:27.4882554Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 2023-11-14T04:55:27.4884329Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 2023-11-14T04:55:27.4885804Z distributed/_composable/fully_shard/test_fully_shard_runtime 2023-11-14T04:55:27.4887563Z distributed/_composable/fully_shard/test_fully_shard_util 2023-11-14T04:55:27.4888554Z distributed/_composable/test_checkpoint 2023-11-14T04:55:27.4889522Z distributed/_composable/test_compose 2023-11-14T04:55:27.4890274Z distributed/_composable/test_contract 2023-11-14T04:55:27.4890995Z distributed/_composable/test_replicate 2023-11-14T04:55:27.4892131Z distributed/_shard/sharded_optim/test_sharded_optim 2023-11-14T04:55:27.4893377Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 2023-11-14T04:55:27.4894672Z distributed/_shard/sharded_tensor/ops/test_embedding 2023-11-14T04:55:27.4896076Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2023-11-14T04:55:27.4897441Z distributed/_shard/sharded_tensor/ops/test_init 2023-11-14T04:55:27.4898261Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2023-11-14T04:55:27.4899328Z distributed/_shard/sharded_tensor/test_logger 2023-11-14T04:55:27.4900386Z distributed/_shard/sharded_tensor/test_sharded_tensor 2023-11-14T04:55:27.4901577Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2023-11-14T04:55:27.4902599Z distributed/_shard/sharding_plan/test_sharding_plan 2023-11-14T04:55:27.4903537Z distributed/_shard/sharding_spec/test_sharding_spec 2023-11-14T04:55:27.4904558Z distributed/_tensor/debug/test_op_coverage 2023-11-14T04:55:27.4905581Z distributed/_tensor/test_api 2023-11-14T04:55:27.4906106Z distributed/_tensor/test_common_rules 2023-11-14T04:55:27.4906646Z distributed/_tensor/test_dtensor 2023-11-14T04:55:27.4907170Z distributed/_tensor/test_dtensor_compile 2023-11-14T04:55:27.4907851Z distributed/_tensor/test_embedding_ops 2023-11-14T04:55:27.4908382Z distributed/_tensor/test_init 2023-11-14T04:55:27.4908897Z distributed/_tensor/test_math_ops 2023-11-14T04:55:27.4909445Z distributed/_tensor/test_matrix_ops 2023-11-14T04:55:27.4909984Z distributed/_tensor/test_pointwise_ops 2023-11-14T04:55:27.4910546Z distributed/_tensor/test_random_ops 2023-11-14T04:55:27.4911095Z distributed/_tensor/test_redistribute 2023-11-14T04:55:27.4911643Z distributed/_tensor/test_tensor_ops 2023-11-14T04:55:27.4912182Z distributed/_tensor/test_utils 2023-11-14T04:55:27.4912697Z distributed/_tensor/test_view_ops 2023-11-14T04:55:27.4913224Z distributed/_tools/test_memory_tracker 2023-11-14T04:55:27.4934935Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2023-11-14T04:55:27.4935823Z distributed/algorithms/quantization/test_quantization 2023-11-14T04:55:27.4936823Z distributed/algorithms/test_join 2023-11-14T04:55:27.4937390Z distributed/checkpoint/test_checkpoint 2023-11-14T04:55:27.4937965Z distributed/checkpoint/test_dedup_tensors 2023-11-14T04:55:27.4938616Z distributed/checkpoint/test_dtensor_checkpoint 2023-11-14T04:55:27.4939314Z distributed/checkpoint/test_file_system_checkpoint 2023-11-14T04:55:27.4940029Z distributed/checkpoint/test_file_system_checkpoint_cpu 2023-11-14T04:55:27.4940739Z distributed/checkpoint/test_fsdp_model_state 2023-11-14T04:55:27.4941387Z distributed/checkpoint/test_fsdp_optim_state 2023-11-14T04:55:27.4941983Z distributed/checkpoint/test_fsspec 2023-11-14T04:55:27.4942561Z distributed/checkpoint/test_nested_dict 2023-11-14T04:55:27.4943140Z distributed/checkpoint/test_planner 2023-11-14T04:55:27.4943672Z distributed/checkpoint/test_traverse 2023-11-14T04:55:27.4944230Z distributed/checkpoint/test_utils 2023-11-14T04:55:27.4944776Z distributed/elastic/events/lib_test 2023-11-14T04:55:27.4945321Z distributed/elastic/metrics/api_test 2023-11-14T04:55:27.4945930Z distributed/elastic/multiprocessing/api_test 2023-11-14T04:55:27.4946580Z distributed/elastic/timer/local_timer_example 2023-11-14T04:55:27.4947198Z distributed/elastic/timer/local_timer_test 2023-11-14T04:55:27.4947826Z distributed/elastic/utils/distributed_test 2023-11-14T04:55:27.4948440Z distributed/elastic/utils/logging_test 2023-11-14T04:55:27.4948983Z distributed/elastic/utils/util_test 2023-11-14T04:55:27.4949555Z distributed/fsdp/test_checkpoint_wrapper 2023-11-14T04:55:27.4950326Z distributed/fsdp/test_distributed_checkpoint 2023-11-14T04:55:27.4950928Z distributed/fsdp/test_fsdp_apply 2023-11-14T04:55:27.4951510Z distributed/fsdp/test_fsdp_backward_prefetch 2023-11-14T04:55:27.4952120Z distributed/fsdp/test_fsdp_checkpoint 2023-11-14T04:55:27.4952691Z distributed/fsdp/test_fsdp_clip_grad_norm 2023-11-14T04:55:27.4953276Z distributed/fsdp/test_fsdp_comm 2023-11-14T04:55:27.4953808Z distributed/fsdp/test_fsdp_comm_hooks 2023-11-14T04:55:27.4954350Z distributed/fsdp/test_fsdp_core 2023-11-14T04:55:27.4955255Z distributed/fsdp/test_fsdp_dtensor_state_dict 2023-11-14T04:55:27.4955881Z distributed/fsdp/test_fsdp_exec_order 2023-11-14T04:55:27.4956425Z distributed/fsdp/test_fsdp_fine_tune 2023-11-14T04:55:27.4957010Z distributed/fsdp/test_fsdp_flatten_params 2023-11-14T04:55:27.4957647Z distributed/fsdp/test_fsdp_freezing_weights 2023-11-14T04:55:27.4958210Z distributed/fsdp/test_fsdp_fx 2023-11-14T04:55:27.4958726Z distributed/fsdp/test_fsdp_grad_acc 2023-11-14T04:55:27.4959299Z distributed/fsdp/test_fsdp_hybrid_shard 2023-11-14T04:55:27.4959885Z distributed/fsdp/test_fsdp_ignored_modules 2023-11-14T04:55:27.4960478Z distributed/fsdp/test_fsdp_input 2023-11-14T04:55:27.4961012Z distributed/fsdp/test_fsdp_memory 2023-11-14T04:55:27.4961518Z distributed/fsdp/test_fsdp_meta 2023-11-14T04:55:27.4962038Z distributed/fsdp/test_fsdp_misc 2023-11-14T04:55:27.4962598Z distributed/fsdp/test_fsdp_mixed_precision 2023-11-14T04:55:27.4963223Z distributed/fsdp/test_fsdp_multiple_forward 2023-11-14T04:55:27.4963859Z distributed/fsdp/test_fsdp_multiple_wrapping 2023-11-14T04:55:27.4964481Z distributed/fsdp/test_fsdp_optim_state 2023-11-14T04:55:27.4965028Z distributed/fsdp/test_fsdp_overlap 2023-11-14T04:55:27.4965581Z distributed/fsdp/test_fsdp_pure_fp16 2023-11-14T04:55:27.4966206Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2023-11-14T04:55:27.4966831Z distributed/fsdp/test_fsdp_state_dict 2023-11-14T04:55:27.4967399Z distributed/fsdp/test_fsdp_tp_integration 2023-11-14T04:55:27.4968004Z distributed/fsdp/test_fsdp_traversal 2023-11-14T04:55:27.4968561Z distributed/fsdp/test_fsdp_uneven 2023-11-14T04:55:27.4969097Z distributed/fsdp/test_fsdp_unshard_params 2023-11-14T04:55:27.4969719Z distributed/fsdp/test_fsdp_use_orig_params 2023-11-14T04:55:27.4970298Z distributed/fsdp/test_shard_utils 2023-11-14T04:55:27.4970797Z distributed/fsdp/test_utils 2023-11-14T04:55:27.4971474Z distributed/fsdp/test_wrap 2023-11-14T04:55:27.4971985Z distributed/nn/jit/test_instantiator 2023-11-14T04:55:27.4972582Z distributed/optim/test_zero_redundancy_optimizer 2023-11-14T04:55:27.4973257Z distributed/rpc/cuda/test_tensorpipe_agent 2023-11-14T04:55:27.4973862Z distributed/rpc/test_faulty_agent 2023-11-14T04:55:27.4974374Z distributed/rpc/test_share_memory 2023-11-14T04:55:27.4974926Z distributed/rpc/test_tensorpipe_agent 2023-11-14T04:55:27.4975555Z distributed/tensor/parallel/test_ddp_2d_parallel 2023-11-14T04:55:27.4976241Z distributed/tensor/parallel/test_fsdp_2d_parallel 2023-11-14T04:55:27.4976947Z distributed/tensor/parallel/test_parallelize_api 2023-11-14T04:55:27.4977623Z distributed/tensor/parallel/test_tp_examples 2023-11-14T04:55:27.4978268Z distributed/tensor/parallel/test_tp_random_state 2023-11-14T04:55:27.4978924Z distributed/tensor/parallel/test_tp_style 2023-11-14T04:55:27.4979639Z distributed/tensor/parallel/test_view_sharding_dim_change 2023-11-14T04:55:27.4980301Z distributed/test_c10d_common 2023-11-14T04:55:27.4980796Z distributed/test_c10d_gloo 2023-11-14T04:55:27.4981280Z distributed/test_c10d_logger 2023-11-14T04:55:27.4981743Z distributed/test_c10d_nccl 2023-11-14T04:55:27.4982256Z distributed/test_c10d_object_collectives 2023-11-14T04:55:27.4982826Z distributed/test_c10d_pypg 2023-11-14T04:55:27.4983285Z distributed/test_c10d_spawn_gloo 2023-11-14T04:55:27.4983810Z distributed/test_c10d_spawn_nccl 2023-11-14T04:55:27.4984332Z distributed/test_c10d_spawn_ucc 2023-11-14T04:55:27.4984812Z distributed/test_c10d_ucc 2023-11-14T04:55:27.4985424Z distributed/test_data_parallel 2023-11-14T04:55:27.4985980Z distributed/test_distributed_spawn 2023-11-14T04:55:27.4986511Z distributed/test_dynamo_distributed 2023-11-14T04:55:27.4987049Z distributed/test_fake_pg 2023-11-14T04:55:27.4987527Z distributed/test_functional_api 2023-11-14T04:55:27.4988035Z distributed/test_inductor_collectives 2023-11-14T04:55:27.4988596Z distributed/test_launcher 2023-11-14T04:55:27.4989092Z distributed/test_multi_threaded_pg 2023-11-14T04:55:27.4989591Z distributed/test_nccl 2023-11-14T04:55:27.4990049Z distributed/test_pg_wrapper 2023-11-14T04:55:27.4990528Z distributed/test_store 2023-11-14T04:55:27.4990975Z Unranked Relevance tests (49): 2023-11-14T04:55:27.4991477Z distributed/_shard/test_sharder 2023-11-14T04:55:27.4992002Z distributed/_spmd/test_data_parallel 2023-11-14T04:55:27.4992535Z distributed/_spmd/test_graph_utils 2023-11-14T04:55:27.4993072Z distributed/_spmd/test_tracing 2023-11-14T04:55:27.4993616Z distributed/_spmd/test_transformation 2023-11-14T04:55:27.4994236Z distributed/_tensor/experimental/test_tp_transform 2023-11-14T04:55:27.4995096Z distributed/_tensor/test_dtensor_ops 2023-11-14T04:55:27.4995668Z distributed/_tensor/test_op_strategy 2023-11-14T04:55:27.4996201Z distributed/_tensor/test_optimizers 2023-11-14T04:55:27.4996775Z distributed/_tensor/test_xla_integration 2023-11-14T04:55:27.4997436Z distributed/checkpoint/e2e/test_e2e_save_and_load 2023-11-14T04:55:27.4998088Z distributed/checkpoint/e2e/test_fine_tuning 2023-11-14T04:55:27.4998708Z distributed/checkpoint/e2e/test_pipeline 2023-11-14T04:55:27.4999341Z distributed/checkpoint/test_dtensor_resharding 2023-11-14T04:55:27.5000060Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 2023-11-14T04:55:27.5000797Z distributed/checkpoint/test_hsdp_checkpoint 2023-11-14T04:55:27.5001419Z distributed/checkpoint/test_state_dict 2023-11-14T04:55:27.5002006Z distributed/checkpoint/test_state_dict_utils 2023-11-14T04:55:27.5002643Z distributed/checkpoint/test_tp_checkpoint 2023-11-14T04:55:27.5003235Z distributed/elastic/timer/api_test 2023-11-14T04:55:27.5003807Z distributed/fsdp/test_hsdp_dtensor_state_dict 2023-11-14T04:55:27.5004494Z distributed/optim/test_apply_optimizer_in_backward 2023-11-14T04:55:27.5005152Z distributed/optim/test_named_optimizer 2023-11-14T04:55:27.5005718Z distributed/pipeline/sync/skip/test_api 2023-11-14T04:55:27.5006516Z distributed/pipeline/sync/skip/test_gpipe 2023-11-14T04:55:27.5007225Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2023-11-14T04:55:27.5007895Z distributed/pipeline/sync/skip/test_leak 2023-11-14T04:55:27.5008526Z distributed/pipeline/sync/skip/test_portal 2023-11-14T04:55:27.5009174Z distributed/pipeline/sync/skip/test_stash_pop 2023-11-14T04:55:27.5009797Z distributed/pipeline/sync/skip/test_tracker 2023-11-14T04:55:27.5010486Z distributed/pipeline/sync/skip/test_verify_skippables 2023-11-14T04:55:27.5011168Z distributed/pipeline/sync/test_balance 2023-11-14T04:55:27.5011749Z distributed/pipeline/sync/test_bugs 2023-11-14T04:55:27.5012303Z distributed/pipeline/sync/test_checkpoint 2023-11-14T04:55:27.5012902Z distributed/pipeline/sync/test_copy 2023-11-14T04:55:27.5013533Z distributed/pipeline/sync/test_deferred_batch_norm 2023-11-14T04:55:27.5014519Z distributed/pipeline/sync/test_dependency 2023-11-14T04:55:27.5015674Z distributed/pipeline/sync/test_inplace 2023-11-14T04:55:27.5016798Z distributed/pipeline/sync/test_microbatch 2023-11-14T04:55:27.5017613Z distributed/pipeline/sync/test_phony 2023-11-14T04:55:27.5018325Z distributed/pipeline/sync/test_pipe 2023-11-14T04:55:27.5019151Z distributed/pipeline/sync/test_pipeline 2023-11-14T04:55:27.5020067Z distributed/pipeline/sync/test_stream 2023-11-14T04:55:27.5021266Z distributed/pipeline/sync/test_transparency 2023-11-14T04:55:27.5022456Z distributed/pipeline/sync/test_worker 2023-11-14T04:55:27.5023571Z distributed/test_c10d_functional_native 2023-11-14T04:55:27.5024852Z distributed/test_collective_utils 2023-11-14T04:55:27.5025994Z distributed/test_compute_comm_reordering 2023-11-14T04:55:27.5027053Z distributed/test_device_mesh 2023-11-14T04:55:27.5028096Z High Relevance tests (1): 2023-11-14T04:55:27.5029037Z distributed/test_distributed_spawn 2023-11-14T04:55:27.5030037Z Probable Relevance tests (48): 2023-11-14T04:55:27.5031000Z distributed/fsdp/test_fsdp_optim_state 2023-11-14T04:55:27.5032141Z distributed/fsdp/test_fsdp_core 2023-11-14T04:55:27.5032820Z distributed/rpc/test_tensorpipe_agent 2023-11-14T04:55:27.5033492Z distributed/test_multi_threaded_pg 2023-11-14T04:55:27.5034462Z distributed/test_c10d_nccl 2023-11-14T04:55:27.5035529Z distributed/tensor/parallel/test_fsdp_2d_parallel 2023-11-14T04:55:27.5036501Z distributed/_tensor/test_pointwise_ops 2023-11-14T04:55:27.5037521Z distributed/_tensor/test_dtensor 2023-11-14T04:55:27.5038573Z distributed/test_c10d_ucc 2023-11-14T04:55:27.5039387Z distributed/test_pg_wrapper 2023-11-14T04:55:27.5040411Z distributed/test_dynamo_distributed 2023-11-14T04:55:27.5041209Z distributed/fsdp/test_fsdp_use_orig_params 2023-11-14T04:55:27.5041883Z distributed/fsdp/test_fsdp_unshard_params 2023-11-14T04:55:27.5042770Z distributed/test_c10d_gloo 2023-11-14T04:55:27.5043274Z distributed/test_launcher 2023-11-14T04:55:27.5044172Z distributed/test_inductor_collectives 2023-11-14T04:55:27.5045112Z distributed/_shard/sharded_optim/test_sharded_optim 2023-11-14T04:55:27.5046056Z distributed/_tensor/test_dtensor_ops 2023-11-14T04:55:27.5046866Z distributed/rpc/cuda/test_tensorpipe_agent 2023-11-14T04:55:27.5047719Z distributed/test_c10d_common 2023-11-14T04:55:27.5048593Z distributed/_composable/fully_shard/test_fully_shard_runtime 2023-11-14T04:55:27.5049534Z distributed/fsdp/test_fsdp_state_dict 2023-11-14T04:55:27.5050126Z distributed/fsdp/test_hsdp_dtensor_state_dict 2023-11-14T04:55:27.5051089Z distributed/test_c10d_object_collectives 2023-11-14T04:55:27.5051764Z distributed/fsdp/test_fsdp_grad_acc 2023-11-14T04:55:27.5052755Z distributed/_composable/fully_shard/test_fully_shard_compile 2023-11-14T04:55:27.5053743Z distributed/_tensor/test_tensor_ops 2023-11-14T04:55:27.5054309Z distributed/fsdp/test_fsdp_flatten_params 2023-11-14T04:55:27.5054915Z distributed/fsdp/test_fsdp_exec_order 2023-11-14T04:55:27.5055507Z distributed/fsdp/test_checkpoint_wrapper 2023-11-14T04:55:27.5056284Z distributed/test_data_parallel 2023-11-14T04:55:27.5056805Z distributed/_spmd/test_tracing 2023-11-14T04:55:27.5057363Z distributed/fsdp/test_fsdp_mixed_precision 2023-11-14T04:55:27.5058114Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 2023-11-14T04:55:27.5058914Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2023-11-14T04:55:27.5059516Z distributed/fsdp/test_fsdp_fx 2023-11-14T04:55:27.5060011Z distributed/fsdp/test_fsdp_memory 2023-11-14T04:55:27.5060568Z distributed/_tensor/test_dtensor_compile 2023-11-14T04:55:27.5061173Z distributed/test_compute_comm_reordering 2023-11-14T04:55:27.5061764Z distributed/_tensor/debug/test_op_coverage 2023-11-14T04:55:27.5062385Z distributed/fsdp/test_fsdp_ignored_modules 2023-11-14T04:55:27.5063013Z distributed/elastic/utils/distributed_test 2023-11-14T04:55:27.5063596Z distributed/checkpoint/test_state_dict 2023-11-14T04:55:27.5064152Z distributed/_tensor/test_utils 2023-11-14T04:55:27.5064769Z distributed/checkpoint/test_file_system_checkpoint 2023-11-14T04:55:27.5065492Z distributed/checkpoint/test_file_system_checkpoint_cpu 2023-11-14T04:55:27.5066166Z distributed/checkpoint/test_planner 2023-11-14T04:55:27.5066836Z distributed/_composable/fully_shard/test_fully_shard_init 2023-11-14T04:55:27.5067510Z Unranked Relevance tests (133): 2023-11-14T04:55:27.5068199Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 2023-11-14T04:55:27.5069139Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 2023-11-14T04:55:27.5070117Z distributed/_composable/fully_shard/test_fully_shard_util 2023-11-14T04:55:27.5070812Z distributed/_composable/test_checkpoint 2023-11-14T04:55:27.5071399Z distributed/_composable/test_compose 2023-11-14T04:55:27.5071966Z distributed/_composable/test_contract 2023-11-14T04:55:27.5072517Z distributed/_composable/test_replicate 2023-11-14T04:55:27.5073174Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 2023-11-14T04:55:27.5073915Z distributed/_shard/sharded_tensor/ops/test_embedding 2023-11-14T04:55:27.5074850Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2023-11-14T04:55:27.5075596Z distributed/_shard/sharded_tensor/ops/test_init 2023-11-14T04:55:27.5076304Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2023-11-14T04:55:27.5076969Z distributed/_shard/sharded_tensor/test_logger 2023-11-14T04:55:27.5077647Z distributed/_shard/sharded_tensor/test_sharded_tensor 2023-11-14T04:55:27.5078425Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2023-11-14T04:55:27.5079185Z distributed/_shard/sharding_plan/test_sharding_plan 2023-11-14T04:55:27.5079903Z distributed/_shard/sharding_spec/test_sharding_spec 2023-11-14T04:55:27.5080537Z distributed/_shard/test_sharder 2023-11-14T04:55:27.5081048Z distributed/_spmd/test_data_parallel 2023-11-14T04:55:27.5081612Z distributed/_spmd/test_graph_utils 2023-11-14T04:55:27.5082171Z distributed/_spmd/test_transformation 2023-11-14T04:55:27.5082801Z distributed/_tensor/experimental/test_tp_transform 2023-11-14T04:55:27.5083429Z distributed/_tensor/test_api 2023-11-14T04:55:27.5083937Z distributed/_tensor/test_common_rules 2023-11-14T04:55:27.5084483Z distributed/_tensor/test_embedding_ops 2023-11-14T04:55:27.5085044Z distributed/_tensor/test_init 2023-11-14T04:55:27.5085561Z distributed/_tensor/test_math_ops 2023-11-14T04:55:27.5086084Z distributed/_tensor/test_matrix_ops 2023-11-14T04:55:27.5086632Z distributed/_tensor/test_op_strategy 2023-11-14T04:55:27.5087196Z distributed/_tensor/test_optimizers 2023-11-14T04:55:27.5087727Z distributed/_tensor/test_random_ops 2023-11-14T04:55:27.5088279Z distributed/_tensor/test_redistribute 2023-11-14T04:55:27.5088840Z distributed/_tensor/test_view_ops 2023-11-14T04:55:27.5089372Z distributed/_tensor/test_xla_integration 2023-11-14T04:55:27.5089961Z distributed/_tools/test_memory_tracker 2023-11-14T04:55:27.5090619Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2023-11-14T04:55:27.5091540Z distributed/algorithms/quantization/test_quantization 2023-11-14T04:55:27.5092175Z distributed/algorithms/test_join 2023-11-14T04:55:27.5092770Z distributed/checkpoint/e2e/test_e2e_save_and_load 2023-11-14T04:55:27.5093444Z distributed/checkpoint/e2e/test_fine_tuning 2023-11-14T04:55:27.5094036Z distributed/checkpoint/e2e/test_pipeline 2023-11-14T04:55:27.5094638Z distributed/checkpoint/test_checkpoint 2023-11-14T04:55:27.5095237Z distributed/checkpoint/test_dedup_tensors 2023-11-14T04:55:27.5095863Z distributed/checkpoint/test_dtensor_checkpoint 2023-11-14T04:55:27.5096533Z distributed/checkpoint/test_dtensor_resharding 2023-11-14T04:55:27.5097184Z distributed/checkpoint/test_fsdp_model_state 2023-11-14T04:55:27.5097804Z distributed/checkpoint/test_fsdp_optim_state 2023-11-14T04:55:27.5098586Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 2023-11-14T04:55:27.5098813Z distributed/checkpoint/test_fsspec 2023-11-14T04:55:27.5099038Z distributed/checkpoint/test_hsdp_checkpoint 2023-11-14T04:55:27.5099255Z distributed/checkpoint/test_nested_dict 2023-11-14T04:55:27.5099510Z distributed/checkpoint/test_state_dict_utils 2023-11-14T04:55:27.5099718Z distributed/checkpoint/test_tp_checkpoint 2023-11-14T04:55:27.5099906Z distributed/checkpoint/test_traverse 2023-11-14T04:55:27.5100115Z distributed/checkpoint/test_utils 2023-11-14T04:55:27.5100300Z distributed/elastic/events/lib_test 2023-11-14T04:55:27.5100489Z distributed/elastic/metrics/api_test 2023-11-14T04:55:27.5100748Z distributed/elastic/multiprocessing/api_test 2023-11-14T04:55:27.5101054Z distributed/elastic/timer/api_test 2023-11-14T04:55:27.5101312Z distributed/elastic/timer/local_timer_example 2023-11-14T04:55:27.5101554Z distributed/elastic/timer/local_timer_test 2023-11-14T04:55:27.5101751Z distributed/elastic/utils/logging_test 2023-11-14T04:55:27.5101965Z distributed/elastic/utils/util_test 2023-11-14T04:55:27.5102193Z distributed/fsdp/test_distributed_checkpoint 2023-11-14T04:55:27.5102379Z distributed/fsdp/test_fsdp_apply 2023-11-14T04:55:27.5102632Z distributed/fsdp/test_fsdp_backward_prefetch 2023-11-14T04:55:27.5102821Z distributed/fsdp/test_fsdp_checkpoint 2023-11-14T04:55:27.5103027Z distributed/fsdp/test_fsdp_clip_grad_norm 2023-11-14T04:55:27.5103227Z distributed/fsdp/test_fsdp_comm 2023-11-14T04:55:27.5103416Z distributed/fsdp/test_fsdp_comm_hooks 2023-11-14T04:55:27.5103643Z distributed/fsdp/test_fsdp_dtensor_state_dict 2023-11-14T04:55:27.5103856Z distributed/fsdp/test_fsdp_fine_tune 2023-11-14T04:55:27.5104085Z distributed/fsdp/test_fsdp_freezing_weights 2023-11-14T04:55:27.5104286Z distributed/fsdp/test_fsdp_hybrid_shard 2023-11-14T04:55:27.5104491Z distributed/fsdp/test_fsdp_input 2023-11-14T04:55:27.5104670Z distributed/fsdp/test_fsdp_meta 2023-11-14T04:55:27.5104849Z distributed/fsdp/test_fsdp_misc 2023-11-14T04:55:27.5105097Z distributed/fsdp/test_fsdp_multiple_forward 2023-11-14T04:55:27.5105323Z distributed/fsdp/test_fsdp_multiple_wrapping 2023-11-14T04:55:27.5105524Z distributed/fsdp/test_fsdp_overlap 2023-11-14T04:55:27.5105713Z distributed/fsdp/test_fsdp_pure_fp16 2023-11-14T04:55:27.5105919Z distributed/fsdp/test_fsdp_tp_integration 2023-11-14T04:55:27.5106131Z distributed/fsdp/test_fsdp_traversal 2023-11-14T04:55:27.5106311Z distributed/fsdp/test_fsdp_uneven 2023-11-14T04:55:27.5106491Z distributed/fsdp/test_shard_utils 2023-11-14T04:55:27.5106687Z distributed/fsdp/test_utils 2023-11-14T04:55:27.5106856Z distributed/fsdp/test_wrap 2023-11-14T04:55:27.5107041Z distributed/nn/jit/test_instantiator 2023-11-14T04:55:27.5107340Z distributed/optim/test_apply_optimizer_in_backward 2023-11-14T04:55:27.5107542Z distributed/optim/test_named_optimizer 2023-11-14T04:55:27.5107787Z distributed/optim/test_zero_redundancy_optimizer 2023-11-14T04:55:27.5108017Z distributed/pipeline/sync/skip/test_api 2023-11-14T04:55:27.5108227Z distributed/pipeline/sync/skip/test_gpipe 2023-11-14T04:55:27.5108658Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2023-11-14T04:55:27.5108892Z distributed/pipeline/sync/skip/test_leak 2023-11-14T04:55:27.5109112Z distributed/pipeline/sync/skip/test_portal 2023-11-14T04:55:27.5109346Z distributed/pipeline/sync/skip/test_stash_pop 2023-11-14T04:55:27.5109593Z distributed/pipeline/sync/skip/test_tracker 2023-11-14T04:55:27.5109869Z distributed/pipeline/sync/skip/test_verify_skippables 2023-11-14T04:55:27.5110088Z distributed/pipeline/sync/test_balance 2023-11-14T04:55:27.5110275Z distributed/pipeline/sync/test_bugs 2023-11-14T04:55:27.5110492Z distributed/pipeline/sync/test_checkpoint 2023-11-14T04:55:27.5110702Z distributed/pipeline/sync/test_copy 2023-11-14T04:55:27.5110965Z distributed/pipeline/sync/test_deferred_batch_norm 2023-11-14T04:55:27.5111173Z distributed/pipeline/sync/test_dependency 2023-11-14T04:55:27.5111392Z distributed/pipeline/sync/test_inplace 2023-11-14T04:55:27.5111596Z distributed/pipeline/sync/test_microbatch 2023-11-14T04:55:27.5111795Z distributed/pipeline/sync/test_phony 2023-11-14T04:55:27.5112003Z distributed/pipeline/sync/test_pipe 2023-11-14T04:55:27.5112204Z distributed/pipeline/sync/test_pipeline 2023-11-14T04:55:27.5112391Z distributed/pipeline/sync/test_stream 2023-11-14T04:55:27.5112644Z distributed/pipeline/sync/test_transparency 2023-11-14T04:55:27.5112834Z distributed/pipeline/sync/test_worker 2023-11-14T04:55:27.5113014Z distributed/rpc/test_faulty_agent 2023-11-14T04:55:27.5113219Z distributed/rpc/test_share_memory 2023-11-14T04:55:27.5113464Z distributed/tensor/parallel/test_ddp_2d_parallel 2023-11-14T04:55:27.5113841Z distributed/tensor/parallel/test_parallelize_api 2023-11-14T04:55:27.5114079Z distributed/tensor/parallel/test_tp_examples 2023-11-14T04:55:27.5114325Z distributed/tensor/parallel/test_tp_random_state 2023-11-14T04:55:27.5114796Z distributed/tensor/parallel/test_tp_style 2023-11-14T04:55:27.5115252Z distributed/tensor/parallel/test_view_sharding_dim_change 2023-11-14T04:55:27.5115687Z distributed/test_c10d_functional_native 2023-11-14T04:55:27.5115964Z distributed/test_c10d_logger 2023-11-14T04:55:27.5116180Z distributed/test_c10d_pypg 2023-11-14T04:55:27.5116359Z distributed/test_c10d_spawn_gloo 2023-11-14T04:55:27.5116565Z distributed/test_c10d_spawn_nccl 2023-11-14T04:55:27.5116740Z distributed/test_c10d_spawn_ucc 2023-11-14T04:55:27.5116918Z distributed/test_collective_utils 2023-11-14T04:55:27.5117114Z distributed/test_device_mesh 2023-11-14T04:55:27.5117278Z distributed/test_fake_pg 2023-11-14T04:55:27.5117445Z distributed/test_functional_api 2023-11-14T04:55:27.5117628Z distributed/test_nccl 2023-11-14T04:55:27.5117791Z distributed/test_store 2023-11-14T04:55:27.5118016Z Found test times from artifacts 2023-11-14T04:55:27.5357962Z Found test times from artifacts 2023-11-14T04:55:27.5411357Z Name: high_relevance 2023-11-14T04:55:27.5411562Z Parallel tests: 2023-11-14T04:55:27.5411712Z Serial tests: 2023-11-14T04:55:27.5411939Z distributed/test_distributed_spawn 2/2 2023-11-14T04:55:27.5412221Z Name: probable_relevance 2023-11-14T04:55:27.5412377Z Parallel tests: 2023-11-14T04:55:27.5412526Z Serial tests: 2023-11-14T04:55:27.5412753Z distributed/fsdp/test_fsdp_core 1/2 2023-11-14T04:55:27.5413088Z distributed/tensor/parallel/test_fsdp_2d_parallel 1/1 2023-11-14T04:55:27.5413255Z distributed/test_c10d_ucc 1/1 2023-11-14T04:55:27.5413496Z distributed/fsdp/test_fsdp_unshard_params 1/1 2023-11-14T04:55:27.5413701Z distributed/test_launcher 1/1 2023-11-14T04:55:27.5413925Z distributed/test_inductor_collectives 1/1 2023-11-14T04:55:27.5414270Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 2023-11-14T04:55:27.5414472Z distributed/_tensor/test_dtensor_ops 1/1 2023-11-14T04:55:27.5414744Z distributed/rpc/cuda/test_tensorpipe_agent 1/2 2023-11-14T04:55:27.5415095Z distributed/_composable/fully_shard/test_fully_shard_runtime 1/1 2023-11-14T04:55:27.5415299Z distributed/fsdp/test_fsdp_state_dict 2/2 2023-11-14T04:55:27.5416044Z distributed/fsdp/test_fsdp_fx 1/1 2023-11-14T04:55:27.5416418Z distributed/fsdp/test_fsdp_memory 1/1 2023-11-14T04:55:27.5416698Z distributed/_tensor/test_dtensor_compile 1/1 2023-11-14T04:55:27.5416928Z distributed/test_compute_comm_reordering 1/1 2023-11-14T04:55:27.5417196Z distributed/_tensor/debug/test_op_coverage 1/1 2023-11-14T04:55:27.5417437Z distributed/fsdp/test_fsdp_ignored_modules 1/1 2023-11-14T04:55:27.5417620Z distributed/_tensor/test_utils 1/1 2023-11-14T04:55:27.5417954Z distributed/checkpoint/test_file_system_checkpoint 1/1 2023-11-14T04:55:27.5418634Z distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 2023-11-14T04:55:27.5418889Z Name: unranked_relevance 2023-11-14T04:55:27.5419054Z Parallel tests: 2023-11-14T04:55:27.5419263Z Serial tests: 2023-11-14T04:55:27.5419573Z distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 2023-11-14T04:55:27.5419746Z distributed/test_c10d_pypg 1/1 2023-11-14T04:55:27.5420032Z distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 2023-11-14T04:55:27.5420190Z distributed/test_device_mesh 1/1 2023-11-14T04:55:27.5420486Z distributed/fsdp/test_fsdp_freezing_weights 1/1 2023-11-14T04:55:27.5420697Z distributed/fsdp/test_fsdp_meta 1/1 2023-11-14T04:55:27.5420893Z distributed/_tensor/test_common_rules 1/1 2023-11-14T04:55:27.5421180Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 2023-11-14T04:55:27.5421357Z distributed/algorithms/test_join 1/1 2023-11-14T04:55:27.5421573Z distributed/test_c10d_spawn_gloo 1/1 2023-11-14T04:55:27.5421948Z distributed/_tensor/test_random_ops 1/1 2023-11-14T04:55:27.5422347Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 2023-11-14T04:55:27.5422591Z distributed/fsdp/test_fsdp_fine_tune 1/1 2023-11-14T04:55:27.5422833Z distributed/elastic/multiprocessing/api_test 1/1 2023-11-14T04:55:27.5423087Z distributed/checkpoint/test_hsdp_checkpoint 1/1 2023-11-14T04:55:27.5423413Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 2023-11-14T04:55:27.5423824Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 1/1 2023-11-14T04:55:27.5424121Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 2023-11-14T04:55:27.5424344Z distributed/fsdp/test_fsdp_input 1/1 2023-11-14T04:55:27.5424639Z distributed/_shard/sharding_plan/test_sharding_plan 1/1 2023-11-14T04:55:27.5424837Z distributed/_tensor/test_view_ops 1/1 2023-11-14T04:55:27.5425154Z distributed/_shard/sharded_tensor/ops/test_embedding 1/1 2023-11-14T04:55:27.5425480Z distributed/_composable/fully_shard/test_fully_shard_util 1/1 2023-11-14T04:55:27.5425724Z distributed/fsdp/test_fsdp_multiple_forward 1/1 2023-11-14T04:55:27.5425939Z distributed/fsdp/test_fsdp_uneven 1/1 2023-11-14T04:55:27.5426158Z distributed/pipeline/sync/test_balance 1/1 2023-11-14T04:55:27.5426460Z distributed/tensor/parallel/test_ddp_2d_parallel 1/1 2023-11-14T04:55:27.5426734Z distributed/tensor/parallel/test_tp_random_state 1/1 2023-11-14T04:55:27.5426980Z distributed/pipeline/sync/skip/test_gpipe 1/1 2023-11-14T04:55:27.5427181Z distributed/test_fake_pg 1/1 2023-11-14T04:55:27.5427413Z distributed/pipeline/sync/test_checkpoint 1/1 2023-11-14T04:55:27.5427579Z distributed/test_nccl 1/1 2023-11-14T04:55:27.5427764Z distributed/fsdp/test_utils 1/1 2023-11-14T04:55:27.5427984Z distributed/elastic/utils/logging_test 1/1 2023-11-14T04:55:27.5428209Z distributed/checkpoint/test_nested_dict 1/1 2023-11-14T04:55:27.5428457Z distributed/_tensor/test_xla_integration 1/1 2023-11-14T04:55:27.5428698Z distributed/checkpoint/test_dedup_tensors 1/1 2023-11-14T04:55:27.5428932Z distributed/pipeline/sync/test_dependency 1/1 2023-11-14T04:55:27.5429203Z distributed/pipeline/sync/skip/test_api 1/1 2023-11-14T04:55:27.5429389Z distributed/_shard/test_sharder 1/1 2023-11-14T04:55:27.5429634Z distributed/_spmd/test_transformation 1/1 2023-11-14T04:55:27.5430003Z distributed/checkpoint/e2e/test_pipeline 1/1 2023-11-14T04:55:27.5430225Z distributed/optim/test_named_optimizer 1/1 2023-11-14T04:55:27.5430413Z Name: unlikely_relevance 2023-11-14T04:55:27.5430570Z Parallel tests: 2023-11-14T04:55:27.5430718Z Serial tests: 2023-11-14T04:55:27.5430896Z Name: none_relevance 2023-11-14T04:55:27.5431049Z Parallel tests: 2023-11-14T04:55:27.5431197Z Serial tests: 2023-11-14T04:55:27.5431988Z Starting test batch 'high_relevance' 1.9073486328125e-06 seconds after initiating testing 2023-11-14T04:55:27.5432204Z With sharding, this batch will run 1 tests 2023-11-14T04:55:29.7269445Z Running distributed/test_distributed_spawn 2/2 ... [2023-11-14 04:55:29.726211] 2023-11-14T04:55:29.7278463Z /usr/bin/mpiexec 2023-11-14T04:55:29.7279097Z MPI not available -- MPI backend tests will be skipped 2023-11-14T04:55:29.7284707Z Running distributed tests for the test backend with env init_method 2023-11-14T04:55:29.7293021Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 04:55:29.728830] 2023-11-14T04:55:35.2203739Z 2023-11-14T04:55:35.2206096Z distributed/test_distributed_spawn 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.2_525f18ae3bfe5473_.log 2023-11-14T04:55:35.2207464Z 2023-11-14T04:55:35.2217841Z Running distributed tests for the test backend with file init_method 2023-11-14T04:55:35.2225472Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 04:55:35.222063] 2023-11-14T04:55:40.7367541Z 2023-11-14T04:55:40.7371878Z distributed/test_distributed_spawn 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.2_321c95eedf29904e_.log 2023-11-14T04:55:40.7374629Z 2023-11-14T04:55:40.7382598Z Running distributed tests for the nccl backend with env init_method 2023-11-14T04:55:40.7390106Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 04:55:40.738495] 2023-11-14T05:18:31.6483227Z 2023-11-14T05:18:31.6488099Z distributed/test_distributed_spawn 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.2_059d461e6a272799_.log 2023-11-14T05:18:31.6490592Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2023-11-14T05:18:31.6494498Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2023-11-14T05:18:31.6496535Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2023-11-14T05:18:31.6498413Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2023-11-14T05:18:31.6500379Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2023-11-14T05:18:31.6502575Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value 2023-11-14T05:18:31.6504877Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2023-11-14T05:18:31.6509398Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2023-11-14T05:18:31.6513684Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2023-11-14T05:18:31.6517502Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2023-11-14T05:18:31.6521680Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2023-11-14T05:18:31.6524885Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2023-11-14T05:18:31.6526666Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2023-11-14T05:18:31.6528459Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync 2023-11-14T05:18:31.6530225Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2023-11-14T05:18:31.6532343Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2023-11-14T05:18:31.6534121Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2023-11-14T05:18:31.6535829Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2023-11-14T05:18:31.6537515Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2023-11-14T05:18:31.6539186Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2023-11-14T05:18:31.6540886Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_multigpu_complex 2023-11-14T05:18:31.6542646Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup 2023-11-14T05:18:31.6544326Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2023-11-14T05:18:31.6546098Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2023-11-14T05:18:31.6547953Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2023-11-14T05:18:31.6549825Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2023-11-14T05:18:31.6551568Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min 2023-11-14T05:18:31.6553319Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2023-11-14T05:18:31.6555495Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2023-11-14T05:18:31.6557258Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2023-11-14T05:18:31.6558967Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2023-11-14T05:18:31.6560901Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2023-11-14T05:18:31.6562565Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2023-11-14T05:18:31.6564195Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_multigpu 2023-11-14T05:18:31.6565891Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_multigpu_complex 2023-11-14T05:18:31.6567607Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2023-11-14T05:18:31.6569286Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_result_cuda 2023-11-14T05:18:31.6570960Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum 2023-11-14T05:18:31.6572615Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2023-11-14T05:18:31.6574218Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2023-11-14T05:18:31.6575841Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group 2023-11-14T05:18:31.6577613Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2023-11-14T05:18:31.6579408Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group 2023-11-14T05:18:31.6581268Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2023-11-14T05:18:31.6583139Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2023-11-14T05:18:31.6585068Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2023-11-14T05:18:31.6587007Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2023-11-14T05:18:31.6588924Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda 2023-11-14T05:18:31.6590726Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group 2023-11-14T05:18:31.6592291Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier 2023-11-14T05:18:31.6593910Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2023-11-14T05:18:31.6595997Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2023-11-14T05:18:31.6597626Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2023-11-14T05:18:31.6599280Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group 2023-11-14T05:18:31.6600986Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2023-11-14T05:18:31.6602721Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2023-11-14T05:18:31.6604535Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err 2023-11-14T05:18:31.6606463Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2023-11-14T05:18:31.6608247Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2023-11-14T05:18:31.6610050Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2023-11-14T05:18:31.6611717Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2023-11-14T05:18:31.6613606Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2023-11-14T05:18:31.6615569Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2023-11-14T05:18:31.6617412Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2023-11-14T05:18:31.6619281Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping 2023-11-14T05:18:31.6621078Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2023-11-14T05:18:31.6623000Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2023-11-14T05:18:31.6624903Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2023-11-14T05:18:31.6626611Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2023-11-14T05:18:31.6628208Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2023-11-14T05:18:31.6629857Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook 2023-11-14T05:18:31.6631555Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2023-11-14T05:18:31.6633221Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2023-11-14T05:18:31.6635320Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2023-11-14T05:18:31.6637093Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2023-11-14T05:18:31.6638815Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg 2023-11-14T05:18:31.6640600Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2023-11-14T05:18:31.6642499Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2023-11-14T05:18:31.6644247Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2023-11-14T05:18:31.6646151Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2023-11-14T05:18:31.6648189Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params 2023-11-14T05:18:31.6650080Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2023-11-14T05:18:31.6652090Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2023-11-14T05:18:31.6653861Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged 2023-11-14T05:18:31.6655582Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2023-11-14T05:18:31.6657356Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2023-11-14T05:18:31.6659165Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2023-11-14T05:18:31.6660917Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states 2023-11-14T05:18:31.6662653Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2023-11-14T05:18:31.6664507Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn 2023-11-14T05:18:31.6666270Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2023-11-14T05:18:31.6667901Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2023-11-14T05:18:31.6669466Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2023-11-14T05:18:31.6671125Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group 2023-11-14T05:18:31.6672797Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2023-11-14T05:18:31.6674504Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params 2023-11-14T05:18:31.6676507Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future 2023-11-14T05:18:31.6678054Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2023-11-14T05:18:31.6679690Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2023-11-14T05:18:31.6681366Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2023-11-14T05:18:31.6682946Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv 2023-11-14T05:18:31.6684454Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2023-11-14T05:18:31.6686062Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler 2023-11-14T05:18:31.6687821Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2023-11-14T05:18:31.6689737Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2023-11-14T05:18:31.6691596Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo 2023-11-14T05:18:31.6693337Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup 2023-11-14T05:18:31.6695134Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2023-11-14T05:18:31.6697044Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2023-11-14T05:18:31.6698793Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2023-11-14T05:18:31.6700447Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort 2023-11-14T05:18:31.6702333Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2023-11-14T05:18:31.6704325Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2023-11-14T05:18:31.6706283Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size 2023-11-14T05:18:31.6708222Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2023-11-14T05:18:31.6709999Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2023-11-14T05:18:31.6711766Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2023-11-14T05:18:31.6713821Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2023-11-14T05:18:31.6716330Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2023-11-14T05:18:31.6718381Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2023-11-14T05:18:31.6720175Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2023-11-14T05:18:31.6721869Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2023-11-14T05:18:31.6723505Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2023-11-14T05:18:31.6725139Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2023-11-14T05:18:31.6726761Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2023-11-14T05:18:31.6728347Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2023-11-14T05:18:31.6729914Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_multigpu 2023-11-14T05:18:31.6731520Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2023-11-14T05:18:31.6733168Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2023-11-14T05:18:31.6734787Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2023-11-14T05:18:31.6736377Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2023-11-14T05:18:31.6738013Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2023-11-14T05:18:31.6739619Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2023-11-14T05:18:31.6741374Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda 2023-11-14T05:18:31.6742979Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2023-11-14T05:18:31.6744643Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2023-11-14T05:18:31.6746262Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2023-11-14T05:18:31.6747906Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2023-11-14T05:18:31.6749647Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2023-11-14T05:18:31.6751490Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2023-11-14T05:18:31.6753292Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2023-11-14T05:18:31.6755451Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2023-11-14T05:18:31.6757442Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2023-11-14T05:18:31.6759231Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2023-11-14T05:18:31.6760960Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2023-11-14T05:18:31.6762664Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2023-11-14T05:18:31.6763646Z 2023-11-14T05:18:31.6764030Z Running distributed tests for the nccl backend with file init_method 2023-11-14T05:18:31.6766489Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 05:18:31.653896] 2023-11-14T05:41:22.0582060Z 2023-11-14T05:41:22.0584418Z distributed/test_distributed_spawn 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.2_ad3bea52ae63b2cd_.log 2023-11-14T05:41:22.0588274Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2023-11-14T05:41:22.0591650Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2023-11-14T05:41:22.0594246Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2023-11-14T05:41:22.0597060Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2023-11-14T05:41:22.0599495Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2023-11-14T05:41:22.0602212Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value 2023-11-14T05:41:22.0605897Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2023-11-14T05:41:22.0608964Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2023-11-14T05:41:22.0611139Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2023-11-14T05:41:22.0613462Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2023-11-14T05:41:22.0616656Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2023-11-14T05:41:22.0618882Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2023-11-14T05:41:22.0620710Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2023-11-14T05:41:22.0622571Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync 2023-11-14T05:41:22.0624252Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2023-11-14T05:41:22.0626056Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2023-11-14T05:41:22.0628013Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2023-11-14T05:41:22.0629750Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2023-11-14T05:41:22.0631439Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2023-11-14T05:41:22.0633164Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2023-11-14T05:41:22.0635182Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_multigpu_complex 2023-11-14T05:41:22.0636914Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup 2023-11-14T05:41:22.0638533Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2023-11-14T05:41:22.0640295Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2023-11-14T05:41:22.0642189Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2023-11-14T05:41:22.0644067Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2023-11-14T05:41:22.0645819Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min 2023-11-14T05:41:22.0647581Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2023-11-14T05:41:22.0649352Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2023-11-14T05:41:22.0651129Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2023-11-14T05:41:22.0652863Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2023-11-14T05:41:22.0654571Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2023-11-14T05:41:22.0656499Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2023-11-14T05:41:22.0658037Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_multigpu 2023-11-14T05:41:22.0659738Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_multigpu_complex 2023-11-14T05:41:22.0661458Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2023-11-14T05:41:22.0663149Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_result_cuda 2023-11-14T05:41:22.0664807Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum 2023-11-14T05:41:22.0666437Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2023-11-14T05:41:22.0668052Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2023-11-14T05:41:22.0669680Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group 2023-11-14T05:41:22.0671320Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2023-11-14T05:41:22.0673244Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group 2023-11-14T05:41:22.0675387Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2023-11-14T05:41:22.0677215Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2023-11-14T05:41:22.0679152Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2023-11-14T05:41:22.0681097Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2023-11-14T05:41:22.0683021Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda 2023-11-14T05:41:22.0684826Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group 2023-11-14T05:41:22.0686390Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier 2023-11-14T05:41:22.0688019Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2023-11-14T05:41:22.0689691Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2023-11-14T05:41:22.0691320Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2023-11-14T05:41:22.0692971Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group 2023-11-14T05:41:22.0694688Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2023-11-14T05:41:22.0696421Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2023-11-14T05:41:22.0698233Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err 2023-11-14T05:41:22.0700162Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2023-11-14T05:41:22.0701953Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2023-11-14T05:41:22.0703765Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2023-11-14T05:41:22.0705463Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2023-11-14T05:41:22.0707335Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2023-11-14T05:41:22.0709306Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2023-11-14T05:41:22.0711157Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2023-11-14T05:41:22.0713064Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping 2023-11-14T05:41:22.0715070Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2023-11-14T05:41:22.0716979Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2023-11-14T05:41:22.0718941Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2023-11-14T05:41:22.0720660Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2023-11-14T05:41:22.0722252Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2023-11-14T05:41:22.0723925Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook 2023-11-14T05:41:22.0725624Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2023-11-14T05:41:22.0727291Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2023-11-14T05:41:22.0729045Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2023-11-14T05:41:22.0730804Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2023-11-14T05:41:22.0732549Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg 2023-11-14T05:41:22.0734333Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2023-11-14T05:41:22.0736248Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2023-11-14T05:41:22.0738013Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2023-11-14T05:41:22.0739923Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2023-11-14T05:41:22.0741969Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params 2023-11-14T05:41:22.0743866Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2023-11-14T05:41:22.0745876Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2023-11-14T05:41:22.0747670Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged 2023-11-14T05:41:22.0749390Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2023-11-14T05:41:22.0751184Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2023-11-14T05:41:22.0753002Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2023-11-14T05:41:22.0754994Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states 2023-11-14T05:41:22.0756688Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2023-11-14T05:41:22.0758551Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn 2023-11-14T05:41:22.0760321Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2023-11-14T05:41:22.0762015Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2023-11-14T05:41:22.0763570Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2023-11-14T05:41:22.0765191Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group 2023-11-14T05:41:22.0766878Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2023-11-14T05:41:22.0768594Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params 2023-11-14T05:41:22.0770240Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future 2023-11-14T05:41:22.0771797Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2023-11-14T05:41:22.0773441Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2023-11-14T05:41:22.0775123Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2023-11-14T05:41:22.0776713Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv 2023-11-14T05:41:22.0778218Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2023-11-14T05:41:22.0779849Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler 2023-11-14T05:41:22.0781608Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2023-11-14T05:41:22.0783536Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2023-11-14T05:41:22.0785408Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo 2023-11-14T05:41:22.0787173Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup 2023-11-14T05:41:22.0788936Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2023-11-14T05:41:22.0790840Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2023-11-14T05:41:22.0792596Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2023-11-14T05:41:22.0794259Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort 2023-11-14T05:41:22.0796347Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2023-11-14T05:41:22.0798353Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2023-11-14T05:41:22.0800309Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size 2023-11-14T05:41:22.0802258Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2023-11-14T05:41:22.0804117Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2023-11-14T05:41:22.0805837Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2023-11-14T05:41:22.0807920Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2023-11-14T05:41:22.0810231Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2023-11-14T05:41:22.0812177Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2023-11-14T05:41:22.0813974Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2023-11-14T05:41:22.0815671Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2023-11-14T05:41:22.0817317Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2023-11-14T05:41:22.0819011Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2023-11-14T05:41:22.0820642Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2023-11-14T05:41:22.0822247Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2023-11-14T05:41:22.0823823Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_multigpu 2023-11-14T05:41:22.0825441Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2023-11-14T05:41:22.0827094Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2023-11-14T05:41:22.0828721Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2023-11-14T05:41:22.0830332Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2023-11-14T05:41:22.0831980Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2023-11-14T05:41:22.0833601Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2023-11-14T05:41:22.0835552Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda 2023-11-14T05:41:22.0837169Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2023-11-14T05:41:22.0838849Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2023-11-14T05:41:22.0840487Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2023-11-14T05:41:22.0842107Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2023-11-14T05:41:22.0843885Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2023-11-14T05:41:22.0845737Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2023-11-14T05:41:22.0847566Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2023-11-14T05:41:22.0849374Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2023-11-14T05:41:22.0851234Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2023-11-14T05:41:22.0853135Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2023-11-14T05:41:22.0854893Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2023-11-14T05:41:22.0856599Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2023-11-14T05:41:22.0857600Z 2023-11-14T05:41:22.0857971Z Running distributed tests for the gloo backend with env init_method 2023-11-14T05:41:22.0860433Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 05:41:22.065364] 2023-11-14T06:07:13.7965373Z 2023-11-14T06:07:13.7967440Z PRINTING LOG FILE of distributed/test_distributed_spawn 2/2 (/var/lib/jenkins/workspace/test/test-reports/distributed.test_distributed_spawn_2.2_ddlahcw4_toprint.log) 2023-11-14T06:07:13.7969013Z Running 146 items in this shard 2023-11-14T06:07:13.7970669Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2023-11-14T06:07:13.7972423Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2023-11-14T06:07:13.7973946Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2023-11-14T06:07:13.7975334Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2023-11-14T06:07:13.7976882Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2023-11-14T06:07:13.7980136Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value 2023-11-14T06:07:13.7983709Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2023-11-14T06:07:13.7986592Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2023-11-14T06:07:13.7988871Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2023-11-14T06:07:13.7990732Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2023-11-14T06:07:13.7992406Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2023-11-14T06:07:13.7993894Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2023-11-14T06:07:13.7995868Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2023-11-14T06:07:13.7997290Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync 2023-11-14T06:07:13.7998684Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2023-11-14T06:07:13.8000125Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2023-11-14T06:07:13.8001511Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2023-11-14T06:07:13.8002837Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2023-11-14T06:07:13.8004512Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2023-11-14T06:07:13.8005810Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2023-11-14T06:07:13.8007414Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_multigpu_complex 2023-11-14T06:07:13.8008872Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup 2023-11-14T06:07:13.8010208Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2023-11-14T06:07:13.8011828Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2023-11-14T06:07:13.8013380Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2023-11-14T06:07:13.8014913Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2023-11-14T06:07:13.8016327Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min 2023-11-14T06:07:13.8017706Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2023-11-14T06:07:13.8019096Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2023-11-14T06:07:13.8020497Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2023-11-14T06:07:13.8021857Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2023-11-14T06:07:13.8023142Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2023-11-14T06:07:13.8024420Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2023-11-14T06:07:13.8025699Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_multigpu 2023-11-14T06:07:13.8027042Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_multigpu_complex 2023-11-14T06:07:13.8028347Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2023-11-14T06:07:13.8029695Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_result_cuda 2023-11-14T06:07:13.8030985Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum 2023-11-14T06:07:13.8032201Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2023-11-14T06:07:13.8033462Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2023-11-14T06:07:13.8035146Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group 2023-11-14T06:07:13.8036395Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2023-11-14T06:07:13.8037820Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group 2023-11-14T06:07:13.8039329Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2023-11-14T06:07:13.8040821Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2023-11-14T06:07:13.8042361Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2023-11-14T06:07:13.8043941Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2023-11-14T06:07:13.8045534Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda 2023-11-14T06:07:13.8046974Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group 2023-11-14T06:07:13.8048144Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier 2023-11-14T06:07:13.8049421Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2023-11-14T06:07:13.8050845Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2023-11-14T06:07:13.8052134Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2023-11-14T06:07:13.8053386Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group 2023-11-14T06:07:13.8054708Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2023-11-14T06:07:13.8056079Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2023-11-14T06:07:13.8057492Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err 2023-11-14T06:07:13.8058899Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2023-11-14T06:07:13.8060306Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2023-11-14T06:07:13.8061738Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2023-11-14T06:07:13.8063015Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2023-11-14T06:07:13.8064534Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2023-11-14T06:07:13.8066135Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2023-11-14T06:07:13.8067582Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2023-11-14T06:07:13.8069167Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping 2023-11-14T06:07:13.8070588Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2023-11-14T06:07:13.8072016Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2023-11-14T06:07:13.8073501Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2023-11-14T06:07:13.8075068Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2023-11-14T06:07:13.8076295Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2023-11-14T06:07:13.8077696Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook 2023-11-14T06:07:13.8079012Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2023-11-14T06:07:13.8080327Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2023-11-14T06:07:13.8081692Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2023-11-14T06:07:13.8083035Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2023-11-14T06:07:13.8084383Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg 2023-11-14T06:07:13.8085806Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2023-11-14T06:07:13.8087304Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2023-11-14T06:07:13.8088690Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2023-11-14T06:07:13.8090214Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2023-11-14T06:07:13.8091900Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params 2023-11-14T06:07:13.8093391Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2023-11-14T06:07:13.8094972Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2023-11-14T06:07:13.8096399Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged 2023-11-14T06:07:13.8097759Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2023-11-14T06:07:13.8099141Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2023-11-14T06:07:13.8100577Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2023-11-14T06:07:13.8101945Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states 2023-11-14T06:07:13.8103289Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2023-11-14T06:07:13.8104768Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn 2023-11-14T06:07:13.8106144Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2023-11-14T06:07:13.8107309Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2023-11-14T06:07:13.8108441Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2023-11-14T06:07:13.8109670Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group 2023-11-14T06:07:13.8110972Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2023-11-14T06:07:13.8112287Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params 2023-11-14T06:07:13.8113549Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future 2023-11-14T06:07:13.8114938Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2023-11-14T06:07:13.8116190Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2023-11-14T06:07:13.8117513Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2023-11-14T06:07:13.8118709Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv 2023-11-14T06:07:13.8119820Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2023-11-14T06:07:13.8121204Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler 2023-11-14T06:07:13.8122611Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2023-11-14T06:07:13.8124158Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2023-11-14T06:07:13.8125606Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo 2023-11-14T06:07:13.8126993Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup 2023-11-14T06:07:13.8128400Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2023-11-14T06:07:13.8129768Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2023-11-14T06:07:13.8131114Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2023-11-14T06:07:13.8132417Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort 2023-11-14T06:07:13.8133919Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2023-11-14T06:07:13.8135536Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2023-11-14T06:07:13.8137091Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size 2023-11-14T06:07:13.8138786Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2023-11-14T06:07:13.8140216Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2023-11-14T06:07:13.8141587Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2023-11-14T06:07:13.8143165Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2023-11-14T06:07:13.8144941Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2023-11-14T06:07:13.8146605Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2023-11-14T06:07:13.8147991Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2023-11-14T06:07:13.8149312Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2023-11-14T06:07:13.8150597Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2023-11-14T06:07:13.8151822Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2023-11-14T06:07:13.8153065Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2023-11-14T06:07:13.8154274Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2023-11-14T06:07:13.8155767Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_multigpu 2023-11-14T06:07:13.8156965Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2023-11-14T06:07:13.8158241Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2023-11-14T06:07:13.8159491Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2023-11-14T06:07:13.8160713Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2023-11-14T06:07:13.8161976Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2023-11-14T06:07:13.8163209Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2023-11-14T06:07:13.8164578Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda 2023-11-14T06:07:13.8165810Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2023-11-14T06:07:13.8167100Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2023-11-14T06:07:13.8168351Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2023-11-14T06:07:13.8169642Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2023-11-14T06:07:13.8171044Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2023-11-14T06:07:13.8172517Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2023-11-14T06:07:13.8173950Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2023-11-14T06:07:13.8175393Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2023-11-14T06:07:13.8176872Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2023-11-14T06:07:13.8178257Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2023-11-14T06:07:13.8179583Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2023-11-14T06:07:13.8181051Z test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2023-11-14T06:07:13.8181866Z 2023-11-14T06:07:13.8182040Z 284 tests collected in 0.03s 2023-11-14T06:07:13.8183468Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-504775bb20eb3b37.xml 2023-11-14T06:07:13.8184789Z ============================= test session starts ============================== 2023-11-14T06:07:13.8210362Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8211378Z cachedir: .pytest_cache 2023-11-14T06:07:13.8212576Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8213688Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8214215Z configfile: pytest.ini 2023-11-14T06:07:13.8215280Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8216358Z collecting ... collected 1 item 2023-11-14T06:07:13.8217796Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager 2023-11-14T06:07:13.8219042Z 2023-11-14T06:07:13.8221118Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8223331Z PASSED [6.2479s] [100%] 2023-11-14T06:07:13.8223641Z 2023-11-14T06:07:13.8224888Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-504775bb20eb3b37.xml - 2023-11-14T06:07:13.8226440Z ============================== 1 passed in 6.29s =============================== 2023-11-14T06:07:13.8227952Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6e2da40136798593.xml 2023-11-14T06:07:13.8229282Z ============================= test session starts ============================== 2023-11-14T06:07:13.8230414Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8231572Z cachedir: .pytest_cache 2023-11-14T06:07:13.8232688Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8233792Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8234321Z configfile: pytest.ini 2023-11-14T06:07:13.8235647Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8236701Z collecting ... collected 1 item 2023-11-14T06:07:13.8237896Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2023-11-14T06:07:13.8238927Z 2023-11-14T06:07:13.8240927Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Test requires world size of 4) [100%] 2023-11-14T06:07:13.8242847Z 2023-11-14T06:07:13.8244117Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6e2da40136798593.xml - 2023-11-14T06:07:13.8245652Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:13.8247147Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-97011c5f813edeaa.xml 2023-11-14T06:07:13.8248663Z ============================= test session starts ============================== 2023-11-14T06:07:13.8249796Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8250707Z cachedir: .pytest_cache 2023-11-14T06:07:13.8251854Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8252960Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8253475Z configfile: pytest.ini 2023-11-14T06:07:13.8254568Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8255620Z collecting ... collected 1 item 2023-11-14T06:07:13.8256728Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2023-11-14T06:07:13.8257728Z 2023-11-14T06:07:13.8259509Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8263514Z SKIPPED [0.3178s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [100%] 2023-11-14T06:07:13.8265750Z 2023-11-14T06:07:13.8267020Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-97011c5f813edeaa.xml - 2023-11-14T06:07:13.8268589Z ============================== 1 skipped in 0.35s ============================== 2023-11-14T06:07:13.8270101Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb3a98eab001f4eb.xml 2023-11-14T06:07:13.8271470Z ============================= test session starts ============================== 2023-11-14T06:07:13.8272606Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8273492Z cachedir: .pytest_cache 2023-11-14T06:07:13.8274933Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8276234Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8276764Z configfile: pytest.ini 2023-11-14T06:07:13.8277834Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8278892Z collecting ... collected 1 item 2023-11-14T06:07:13.8280053Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2023-11-14T06:07:13.8281044Z 2023-11-14T06:07:13.8282848Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8284780Z PASSED [5.9361s] [100%] 2023-11-14T06:07:13.8285086Z 2023-11-14T06:07:13.8286336Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb3a98eab001f4eb.xml - 2023-11-14T06:07:13.8287910Z ============================== 1 passed in 5.97s =============================== 2023-11-14T06:07:13.8289379Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e2416b9aa862ab3.xml 2023-11-14T06:07:13.8290729Z ============================= test session starts ============================== 2023-11-14T06:07:13.8291864Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8292926Z cachedir: .pytest_cache 2023-11-14T06:07:13.8294076Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8295183Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8295707Z configfile: pytest.ini 2023-11-14T06:07:13.8296763Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8297825Z collecting ... collected 1 item 2023-11-14T06:07:13.8299136Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last 2023-11-14T06:07:13.8300274Z 2023-11-14T06:07:13.8302232Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8304304Z PASSED [6.3361s] [100%] 2023-11-14T06:07:13.8304609Z 2023-11-14T06:07:13.8305860Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e2416b9aa862ab3.xml - 2023-11-14T06:07:13.8307422Z ============================== 1 passed in 6.37s =============================== 2023-11-14T06:07:13.8308900Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4125e6349aca78e.xml 2023-11-14T06:07:13.8311318Z ============================= test session starts ============================== 2023-11-14T06:07:13.8312460Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8313375Z cachedir: .pytest_cache 2023-11-14T06:07:13.8314490Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8315839Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8316369Z configfile: pytest.ini 2023-11-14T06:07:13.8317432Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8318488Z collecting ... collected 1 item 2023-11-14T06:07:13.8319893Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value 2023-11-14T06:07:13.8321289Z 2023-11-14T06:07:13.8323376Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8325504Z PASSED [6.4394s] [100%] 2023-11-14T06:07:13.8325811Z 2023-11-14T06:07:13.8327058Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4125e6349aca78e.xml - 2023-11-14T06:07:13.8328618Z ============================== 1 passed in 6.48s =============================== 2023-11-14T06:07:13.8330115Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-576c4b9d361f090e.xml 2023-11-14T06:07:13.8331458Z ============================= test session starts ============================== 2023-11-14T06:07:13.8332599Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8333511Z cachedir: .pytest_cache 2023-11-14T06:07:13.8334620Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8335730Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8336253Z configfile: pytest.ini 2023-11-14T06:07:13.8337474Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8338531Z collecting ... collected 1 item 2023-11-14T06:07:13.8339899Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2023-11-14T06:07:13.8341113Z 2023-11-14T06:07:13.8343164Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8345289Z PASSED [6.1432s] [100%] 2023-11-14T06:07:13.8345569Z 2023-11-14T06:07:13.8346811Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-576c4b9d361f090e.xml - 2023-11-14T06:07:13.8348372Z ============================== 1 passed in 6.18s =============================== 2023-11-14T06:07:13.8349879Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8cdbf6ee296b6475.xml 2023-11-14T06:07:13.8351221Z ============================= test session starts ============================== 2023-11-14T06:07:13.8352370Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8353285Z cachedir: .pytest_cache 2023-11-14T06:07:13.8354427Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8355774Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8356306Z configfile: pytest.ini 2023-11-14T06:07:13.8357403Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8358442Z collecting ... collected 1 item 2023-11-14T06:07:13.8359724Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2023-11-14T06:07:13.8360840Z 2023-11-14T06:07:13.8362795Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8364988Z PASSED [6.2389s] [100%] 2023-11-14T06:07:13.8365270Z 2023-11-14T06:07:13.8366520Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8cdbf6ee296b6475.xml - 2023-11-14T06:07:13.8368093Z ============================== 1 passed in 6.28s =============================== 2023-11-14T06:07:13.8369648Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fdde95d1d18bb8cb.xml 2023-11-14T06:07:13.8371009Z ============================= test session starts ============================== 2023-11-14T06:07:13.8372110Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8373020Z cachedir: .pytest_cache 2023-11-14T06:07:13.8374170Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8375256Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8375783Z configfile: pytest.ini 2023-11-14T06:07:13.8376872Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8377899Z collecting ... collected 1 item 2023-11-14T06:07:13.8379397Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2023-11-14T06:07:13.8380649Z 2023-11-14T06:07:13.8382675Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8384817Z PASSED [6.1350s] [100%] 2023-11-14T06:07:13.8385099Z 2023-11-14T06:07:13.8386376Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fdde95d1d18bb8cb.xml - 2023-11-14T06:07:13.8387916Z ============================== 1 passed in 6.17s =============================== 2023-11-14T06:07:13.8389412Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2b21dd8ffea6986a.xml 2023-11-14T06:07:13.8390775Z ============================= test session starts ============================== 2023-11-14T06:07:13.8391876Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8392791Z cachedir: .pytest_cache 2023-11-14T06:07:13.8393933Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8395281Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8395814Z configfile: pytest.ini 2023-11-14T06:07:13.8396909Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8397956Z collecting ... collected 1 item 2023-11-14T06:07:13.8399180Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2023-11-14T06:07:13.8400296Z 2023-11-14T06:07:13.8402203Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8404221Z PASSED [6.2425s] [100%] 2023-11-14T06:07:13.8404502Z 2023-11-14T06:07:13.8405777Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2b21dd8ffea6986a.xml - 2023-11-14T06:07:13.8407514Z ============================== 1 passed in 6.28s =============================== 2023-11-14T06:07:13.8409012Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c1e7a9c50c3dd9e.xml 2023-11-14T06:07:13.8410372Z ============================= test session starts ============================== 2023-11-14T06:07:13.8411476Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8412396Z cachedir: .pytest_cache 2023-11-14T06:07:13.8413531Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8414636Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8415139Z configfile: pytest.ini 2023-11-14T06:07:13.8416229Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8417287Z collecting ... collected 1 item 2023-11-14T06:07:13.8418506Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2023-11-14T06:07:13.8419620Z 2023-11-14T06:07:13.8421635Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8423677Z PASSED [8.4405s] [100%] 2023-11-14T06:07:13.8423959Z 2023-11-14T06:07:13.8425235Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c1e7a9c50c3dd9e.xml - 2023-11-14T06:07:13.8426780Z ============================== 1 passed in 8.48s =============================== 2023-11-14T06:07:13.8428294Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8ff421c8b5cee9d0.xml 2023-11-14T06:07:13.8429650Z ============================= test session starts ============================== 2023-11-14T06:07:13.8430785Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8431665Z cachedir: .pytest_cache 2023-11-14T06:07:13.8432804Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8433919Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8434423Z configfile: pytest.ini 2023-11-14T06:07:13.8435773Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8436833Z collecting ... collected 1 item 2023-11-14T06:07:13.8437961Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2023-11-14T06:07:13.8438982Z 2023-11-14T06:07:13.8440768Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8442693Z PASSED [6.0351s] [100%] 2023-11-14T06:07:13.8442982Z 2023-11-14T06:07:13.8444263Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8ff421c8b5cee9d0.xml - 2023-11-14T06:07:13.8445803Z ============================== 1 passed in 6.07s =============================== 2023-11-14T06:07:13.8447290Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7b403652801f7fec.xml 2023-11-14T06:07:13.8448643Z ============================= test session starts ============================== 2023-11-14T06:07:13.8449949Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8450831Z cachedir: .pytest_cache 2023-11-14T06:07:13.8451967Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8453068Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8453565Z configfile: pytest.ini 2023-11-14T06:07:13.8454655Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8455710Z collecting ... collected 1 item 2023-11-14T06:07:13.8456839Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2023-11-14T06:07:13.8457853Z 2023-11-14T06:07:13.8459672Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (no torchvision) [100%] 2023-11-14T06:07:13.8461483Z 2023-11-14T06:07:13.8462728Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7b403652801f7fec.xml - 2023-11-14T06:07:13.8464296Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:13.8465959Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a498ea38a923eb6c.xml 2023-11-14T06:07:13.8467315Z ============================= test session starts ============================== 2023-11-14T06:07:13.8468462Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8469426Z cachedir: .pytest_cache 2023-11-14T06:07:13.8470541Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8471667Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8472195Z configfile: pytest.ini 2023-11-14T06:07:13.8473256Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8474312Z collecting ... collected 1 item 2023-11-14T06:07:13.8475686Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync 2023-11-14T06:07:13.8476681Z 2023-11-14T06:07:13.8478503Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8480392Z PASSED [5.8351s] [100%] 2023-11-14T06:07:13.8480695Z 2023-11-14T06:07:13.8481950Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a498ea38a923eb6c.xml - 2023-11-14T06:07:13.8483518Z ============================== 1 passed in 5.87s =============================== 2023-11-14T06:07:13.8485016Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1eab5e89c0e056b0.xml 2023-11-14T06:07:13.8486348Z ============================= test session starts ============================== 2023-11-14T06:07:13.8487491Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8488402Z cachedir: .pytest_cache 2023-11-14T06:07:13.8489512Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8490622Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8491152Z configfile: pytest.ini 2023-11-14T06:07:13.8492395Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8493427Z collecting ... collected 1 item 2023-11-14T06:07:13.8494571Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2023-11-14T06:07:13.8495553Z 2023-11-14T06:07:13.8497366Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8499771Z STAGE:2023-11-14 05:44:04 5077:5077 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8501130Z STAGE:2023-11-14 05:44:04 5078:5078 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8503237Z STAGE:2023-11-14 05:44:04 5078:5078 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 05:44:04 5077:5077 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8504604Z 2023-11-14T06:07:13.8506072Z STAGE:2023-11-14 05:44:04 5077:5077 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 05:44:04 5078:5078 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8507497Z 2023-11-14T06:07:13.8508167Z STAGE:2023-11-14 05:44:04 5078:5078 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8509639Z STAGE:2023-11-14 05:44:04 5077:5077 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8511066Z STAGE:2023-11-14 05:44:04 5077:5077 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8513230Z STAGE:2023-11-14 05:44:04 5077:5077 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 05:44:04 5078:5078 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8514885Z 2023-11-14T06:07:13.8515640Z STAGE:2023-11-14 05:44:04 5078:5078 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8517046Z STAGE:2023-11-14 05:44:04 5077:5077 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8518422Z STAGE:2023-11-14 05:44:04 5078:5078 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8519830Z STAGE:2023-11-14 05:44:04 5077:5077 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8521231Z STAGE:2023-11-14 05:44:04 5078:5078 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8522690Z STAGE:2023-11-14 05:44:04 5077:5077 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8524176Z STAGE:2023-11-14 05:44:04 5078:5078 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8525128Z PASSED [5.9363s] [100%] 2023-11-14T06:07:13.8525415Z 2023-11-14T06:07:13.8526666Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1eab5e89c0e056b0.xml - 2023-11-14T06:07:13.8528236Z ============================== 1 passed in 5.97s =============================== 2023-11-14T06:07:13.8529736Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fd10e508b72512c1.xml 2023-11-14T06:07:13.8531101Z ============================= test session starts ============================== 2023-11-14T06:07:13.8532211Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8533122Z cachedir: .pytest_cache 2023-11-14T06:07:13.8534265Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8535348Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8536038Z configfile: pytest.ini 2023-11-14T06:07:13.8537126Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8538153Z collecting ... collected 1 item 2023-11-14T06:07:13.8539323Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2023-11-14T06:07:13.8540325Z 2023-11-14T06:07:13.8542171Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8544589Z STAGE:2023-11-14 05:44:16 5099:5099 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8545944Z STAGE:2023-11-14 05:44:16 5100:5100 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8548047Z STAGE:2023-11-14 05:44:16 5099:5099 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 05:44:16 5100:5100 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8549434Z 2023-11-14T06:07:13.8550865Z STAGE:2023-11-14 05:44:16 5099:5099 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 05:44:16 5100:5100 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8552291Z 2023-11-14T06:07:13.8552475Z PASSED [5.9520s] [100%] 2023-11-14T06:07:13.8552756Z 2023-11-14T06:07:13.8554126Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fd10e508b72512c1.xml - 2023-11-14T06:07:13.8555966Z ============================== 1 passed in 5.99s =============================== 2023-11-14T06:07:13.8557477Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-add531748b4dfa2c.xml 2023-11-14T06:07:13.8558851Z ============================= test session starts ============================== 2023-11-14T06:07:13.8559957Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8560866Z cachedir: .pytest_cache 2023-11-14T06:07:13.8562008Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8563088Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8563622Z configfile: pytest.ini 2023-11-14T06:07:13.8564774Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8565830Z collecting ... collected 1 item 2023-11-14T06:07:13.8566895Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2023-11-14T06:07:13.8567846Z 2023-11-14T06:07:13.8569615Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8571965Z STAGE:2023-11-14 05:44:27 5121:5121 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8573340Z STAGE:2023-11-14 05:44:27 5122:5122 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8575417Z STAGE:2023-11-14 05:44:27 5121:5121 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 05:44:27 5122:5122 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8576814Z 2023-11-14T06:07:13.8578252Z STAGE:2023-11-14 05:44:27 5121:5121 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 05:44:27 5122:5122 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8579861Z 2023-11-14T06:07:13.8581185Z STAGE:2023-11-14 05:44:27 5121:5121 ActivityProfilerController.cpp:314] Completed Stage: Warm UpSTAGE:2023-11-14 05:44:27 5122:5122 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8582513Z 2023-11-14T06:07:13.8583204Z STAGE:2023-11-14 05:44:27 5121:5121 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8584592Z STAGE:2023-11-14 05:44:27 5122:5122 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8586058Z STAGE:2023-11-14 05:44:27 5121:5121 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8587549Z STAGE:2023-11-14 05:44:27 5122:5122 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8588500Z PASSED [5.9459s] [100%] 2023-11-14T06:07:13.8588779Z 2023-11-14T06:07:13.8590028Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-add531748b4dfa2c.xml - 2023-11-14T06:07:13.8591596Z ============================== 1 passed in 5.98s =============================== 2023-11-14T06:07:13.8593096Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30bd179b869af655.xml 2023-11-14T06:07:13.8594457Z ============================= test session starts ============================== 2023-11-14T06:07:13.8595815Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8596876Z cachedir: .pytest_cache 2023-11-14T06:07:13.8598053Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8599140Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8599667Z configfile: pytest.ini 2023-11-14T06:07:13.8600743Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8601780Z collecting ... collected 1 item 2023-11-14T06:07:13.8602898Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2023-11-14T06:07:13.8603862Z 2023-11-14T06:07:13.8605816Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only Nccl supports CUDA all gather) [100%] 2023-11-14T06:07:13.8607698Z 2023-11-14T06:07:13.8608974Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30bd179b869af655.xml - 2023-11-14T06:07:13.8610511Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:13.8611997Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a983325d4597d9b5.xml 2023-11-14T06:07:13.8613358Z ============================= test session starts ============================== 2023-11-14T06:07:13.8614492Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8615377Z cachedir: .pytest_cache 2023-11-14T06:07:13.8616510Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8617600Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8618110Z configfile: pytest.ini 2023-11-14T06:07:13.8619196Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8620245Z collecting ... collected 1 item 2023-11-14T06:07:13.8621324Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2023-11-14T06:07:13.8622447Z 2023-11-14T06:07:13.8624188Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8626548Z STAGE:2023-11-14 05:44:44 5147:5147 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8627940Z STAGE:2023-11-14 05:44:44 5146:5146 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8630046Z STAGE:2023-11-14 05:44:44 5146:5146 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 05:44:44 5147:5147 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8631407Z 2023-11-14T06:07:13.8632857Z STAGE:2023-11-14 05:44:44 5146:5146 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 05:44:44 5147:5147 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8634326Z 2023-11-14T06:07:13.8635888Z STAGE:2023-11-14 05:44:44 5146:5146 ActivityProfilerController.cpp:314] Completed Stage: Warm UpSTAGE:2023-11-14 05:44:44 5147:5147 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8637239Z 2023-11-14T06:07:13.8637911Z STAGE:2023-11-14 05:44:44 5146:5146 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8639333Z STAGE:2023-11-14 05:44:44 5147:5147 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8640903Z STAGE:2023-11-14 05:44:44 5146:5146 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8642423Z STAGE:2023-11-14 05:44:44 5147:5147 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8643374Z PASSED [5.9346s] [100%] 2023-11-14T06:07:13.8643660Z 2023-11-14T06:07:13.8644934Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a983325d4597d9b5.xml - 2023-11-14T06:07:13.8646482Z ============================== 1 passed in 5.97s =============================== 2023-11-14T06:07:13.8647984Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8ec7a5fc7170d22e.xml 2023-11-14T06:07:13.8649339Z ============================= test session starts ============================== 2023-11-14T06:07:13.8650442Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8651359Z cachedir: .pytest_cache 2023-11-14T06:07:13.8652500Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8653606Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8654108Z configfile: pytest.ini 2023-11-14T06:07:13.8655189Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8656247Z collecting ... collected 1 item 2023-11-14T06:07:13.8657291Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2023-11-14T06:07:13.8658234Z 2023-11-14T06:07:13.8659945Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8661952Z SKIPPED [5.8420s] (Skipped due to small world size.) [100%] 2023-11-14T06:07:13.8662452Z 2023-11-14T06:07:13.8663724Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8ec7a5fc7170d22e.xml - 2023-11-14T06:07:13.8665265Z ============================== 1 skipped in 5.88s ============================== 2023-11-14T06:07:13.8666915Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4bbfcb68c9759f3.xml 2023-11-14T06:07:13.8668284Z ============================= test session starts ============================== 2023-11-14T06:07:13.8669463Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8670344Z cachedir: .pytest_cache 2023-11-14T06:07:13.8671482Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8672598Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8673103Z configfile: pytest.ini 2023-11-14T06:07:13.8674201Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8675510Z collecting ... collected 1 item 2023-11-14T06:07:13.8676629Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_multigpu_complex 2023-11-14T06:07:13.8677652Z 2023-11-14T06:07:13.8679694Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_multigpu_complex <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Only Nccl backend supports allgather multigpu) [100%] 2023-11-14T06:07:13.8681688Z 2023-11-14T06:07:13.8683069Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4bbfcb68c9759f3.xml - 2023-11-14T06:07:13.8684656Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:13.8686162Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2d2f64650d27a888.xml 2023-11-14T06:07:13.8687496Z ============================= test session starts ============================== 2023-11-14T06:07:13.8688641Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8689551Z cachedir: .pytest_cache 2023-11-14T06:07:13.8690664Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8691774Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8692308Z configfile: pytest.ini 2023-11-14T06:07:13.8693378Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8694444Z collecting ... collected 1 item 2023-11-14T06:07:13.8695588Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup 2023-11-14T06:07:13.8696566Z 2023-11-14T06:07:13.8698370Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8700254Z PASSED [6.0344s] [100%] 2023-11-14T06:07:13.8700561Z 2023-11-14T06:07:13.8701807Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2d2f64650d27a888.xml - 2023-11-14T06:07:13.8703373Z ============================== 1 passed in 6.07s =============================== 2023-11-14T06:07:13.8704877Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51dee0dd4a88aa18.xml 2023-11-14T06:07:13.8706209Z ============================= test session starts ============================== 2023-11-14T06:07:13.8707345Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8708259Z cachedir: .pytest_cache 2023-11-14T06:07:13.8709525Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8710630Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8711160Z configfile: pytest.ini 2023-11-14T06:07:13.8712222Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8713274Z collecting ... collected 1 item 2023-11-14T06:07:13.8714358Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2023-11-14T06:07:13.8715528Z 2023-11-14T06:07:13.8717430Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Only Nccl supports all_gather_v) [100%] 2023-11-14T06:07:13.8719245Z 2023-11-14T06:07:13.8720524Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51dee0dd4a88aa18.xml - 2023-11-14T06:07:13.8722075Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:13.8723574Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c014d9a65f82b711.xml 2023-11-14T06:07:13.8724930Z ============================= test session starts ============================== 2023-11-14T06:07:13.8726157Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8727091Z cachedir: .pytest_cache 2023-11-14T06:07:13.8728232Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8729333Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8729831Z configfile: pytest.ini 2023-11-14T06:07:13.8730937Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8731984Z collecting ... collected 1 item 2023-11-14T06:07:13.8733151Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min 2023-11-14T06:07:13.8734205Z 2023-11-14T06:07:13.8736049Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8738496Z STAGE:2023-11-14 05:45:29 5241:5241 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8739883Z STAGE:2023-11-14 05:45:29 5240:5240 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8741958Z STAGE:2023-11-14 05:45:29 5240:5240 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 05:45:29 5241:5241 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8743354Z 2023-11-14T06:07:13.8744792Z STAGE:2023-11-14 05:45:29 5240:5240 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 05:45:29 5241:5241 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8746253Z 2023-11-14T06:07:13.8746893Z STAGE:2023-11-14 05:45:29 5241:5241 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8748280Z STAGE:2023-11-14 05:45:29 5240:5240 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8749650Z STAGE:2023-11-14 05:45:29 5240:5240 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8751071Z STAGE:2023-11-14 05:45:29 5241:5241 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8752526Z STAGE:2023-11-14 05:45:29 5240:5240 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8754237Z STAGE:2023-11-14 05:45:29 5241:5241 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8755410Z PASSED [5.9340s] [100%] 2023-11-14T06:07:13.8755716Z 2023-11-14T06:07:13.8756968Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c014d9a65f82b711.xml - 2023-11-14T06:07:13.8758532Z ============================== 1 passed in 5.97s =============================== 2023-11-14T06:07:13.8760007Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e5ff12a2809d139.xml 2023-11-14T06:07:13.8761374Z ============================= test session starts ============================== 2023-11-14T06:07:13.8762505Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8763427Z cachedir: .pytest_cache 2023-11-14T06:07:13.8764539Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8765645Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8766169Z configfile: pytest.ini 2023-11-14T06:07:13.8767232Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8768288Z collecting ... collected 1 item 2023-11-14T06:07:13.8769692Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product 2023-11-14T06:07:13.8770768Z 2023-11-14T06:07:13.8772653Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8775100Z STAGE:2023-11-14 05:45:41 5268:5268 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8776477Z STAGE:2023-11-14 05:45:41 5269:5269 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8778577Z STAGE:2023-11-14 05:45:41 5268:5268 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 05:45:41 5269:5269 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8779935Z 2023-11-14T06:07:13.8781407Z STAGE:2023-11-14 05:45:41 5269:5269 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 05:45:41 5268:5268 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8782836Z 2023-11-14T06:07:13.8784149Z STAGE:2023-11-14 05:45:41 5268:5268 ActivityProfilerController.cpp:314] Completed Stage: Warm UpSTAGE:2023-11-14 05:45:41 5269:5269 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8785503Z 2023-11-14T06:07:13.8786865Z STAGE:2023-11-14 05:45:41 5268:5268 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 05:45:41 5269:5269 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8788257Z 2023-11-14T06:07:13.8789692Z STAGE:2023-11-14 05:45:41 5269:5269 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 05:45:41 5268:5268 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8791140Z 2023-11-14T06:07:13.8791303Z PASSED [5.9457s] [100%] 2023-11-14T06:07:13.8791583Z 2023-11-14T06:07:13.8792860Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e5ff12a2809d139.xml - 2023-11-14T06:07:13.8794389Z ============================== 1 passed in 5.98s =============================== 2023-11-14T06:07:13.8796126Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-113b66b7917b0f1d.xml 2023-11-14T06:07:13.8797635Z ============================= test session starts ============================== 2023-11-14T06:07:13.8798775Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8799662Z cachedir: .pytest_cache 2023-11-14T06:07:13.8800806Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8801918Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8802429Z configfile: pytest.ini 2023-11-14T06:07:13.8803522Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8804569Z collecting ... collected 1 item 2023-11-14T06:07:13.8805702Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2023-11-14T06:07:13.8806746Z 2023-11-14T06:07:13.8808553Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8810633Z SKIPPED [5.8440s] (Skipped due to small world size.) [100%] 2023-11-14T06:07:13.8811133Z 2023-11-14T06:07:13.8812532Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-113b66b7917b0f1d.xml - 2023-11-14T06:07:13.8814086Z ============================== 1 skipped in 5.88s ============================== 2023-11-14T06:07:13.8815587Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e8b023e348321575.xml 2023-11-14T06:07:13.8816945Z ============================= test session starts ============================== 2023-11-14T06:07:13.8818089Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8818974Z cachedir: .pytest_cache 2023-11-14T06:07:13.8820112Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8821222Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8821723Z configfile: pytest.ini 2023-11-14T06:07:13.8822806Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8823882Z collecting ... collected 1 item 2023-11-14T06:07:13.8824986Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min 2023-11-14T06:07:13.8825978Z 2023-11-14T06:07:13.8827739Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8830143Z STAGE:2023-11-14 05:46:04 5314:5314 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8831526Z STAGE:2023-11-14 05:46:04 5315:5315 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8833635Z STAGE:2023-11-14 05:46:04 5315:5315 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 05:46:04 5314:5314 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8835256Z 2023-11-14T06:07:13.8836716Z STAGE:2023-11-14 05:46:04 5314:5314 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 05:46:04 5315:5315 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8838178Z 2023-11-14T06:07:13.8838823Z STAGE:2023-11-14 05:46:04 5314:5314 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8840360Z STAGE:2023-11-14 05:46:04 5315:5315 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8841762Z STAGE:2023-11-14 05:46:04 5315:5315 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8843150Z STAGE:2023-11-14 05:46:04 5314:5314 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8844608Z STAGE:2023-11-14 05:46:04 5315:5315 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8846113Z STAGE:2023-11-14 05:46:04 5314:5314 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8847035Z PASSED [5.7400s] [100%] 2023-11-14T06:07:13.8847345Z 2023-11-14T06:07:13.8848590Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e8b023e348321575.xml - 2023-11-14T06:07:13.8850150Z ============================== 1 passed in 5.78s =============================== 2023-11-14T06:07:13.8851658Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f0549cb3b717b331.xml 2023-11-14T06:07:13.8852988Z ============================= test session starts ============================== 2023-11-14T06:07:13.8854119Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8855034Z cachedir: .pytest_cache 2023-11-14T06:07:13.8856272Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8857395Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8857918Z configfile: pytest.ini 2023-11-14T06:07:13.8859022Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8860048Z collecting ... collected 1 item 2023-11-14T06:07:13.8861204Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2023-11-14T06:07:13.8862198Z 2023-11-14T06:07:13.8864020Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8866502Z STAGE:2023-11-14 05:46:15 5336:5336 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8867865Z STAGE:2023-11-14 05:46:15 5337:5337 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8869314Z STAGE:2023-11-14 05:46:15 5337:5337 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8870729Z STAGE:2023-11-14 05:46:15 5336:5336 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8872163Z STAGE:2023-11-14 05:46:15 5337:5337 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8873653Z STAGE:2023-11-14 05:46:15 5336:5336 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8875316Z STAGE:2023-11-14 05:46:15 5337:5337 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8876699Z STAGE:2023-11-14 05:46:15 5336:5336 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8878073Z STAGE:2023-11-14 05:46:15 5336:5336 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8879494Z STAGE:2023-11-14 05:46:15 5337:5337 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8880958Z STAGE:2023-11-14 05:46:15 5336:5336 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8882414Z STAGE:2023-11-14 05:46:15 5337:5337 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8883361Z PASSED [5.9415s] [100%] 2023-11-14T06:07:13.8883796Z 2023-11-14T06:07:13.8885080Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f0549cb3b717b331.xml - 2023-11-14T06:07:13.8886642Z ============================== 1 passed in 5.98s =============================== 2023-11-14T06:07:13.8888107Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bdbde93181036c55.xml 2023-11-14T06:07:13.8889466Z ============================= test session starts ============================== 2023-11-14T06:07:13.8890615Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8891498Z cachedir: .pytest_cache 2023-11-14T06:07:13.8892637Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8893746Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8894266Z configfile: pytest.ini 2023-11-14T06:07:13.8895352Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8896409Z collecting ... collected 1 item 2023-11-14T06:07:13.8897532Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2023-11-14T06:07:13.8898495Z 2023-11-14T06:07:13.8900404Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8902812Z STAGE:2023-11-14 05:46:26 5359:5359 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8904193Z STAGE:2023-11-14 05:46:26 5358:5358 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8906296Z STAGE:2023-11-14 05:46:26 5358:5358 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 05:46:26 5359:5359 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8907665Z 2023-11-14T06:07:13.8909264Z STAGE:2023-11-14 05:46:26 5359:5359 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 05:46:26 5358:5358 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8910725Z 2023-11-14T06:07:13.8911374Z STAGE:2023-11-14 05:46:26 5358:5358 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8912762Z STAGE:2023-11-14 05:46:26 5359:5359 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8915096Z STAGE:2023-11-14 05:46:26 5359:5359 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 05:46:26 5358:5358 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8916474Z 2023-11-14T06:07:13.8917210Z STAGE:2023-11-14 05:46:26 5359:5359 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8918677Z STAGE:2023-11-14 05:46:26 5358:5358 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8919639Z PASSED [5.9381s] [100%] 2023-11-14T06:07:13.8919920Z 2023-11-14T06:07:13.8921189Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bdbde93181036c55.xml - 2023-11-14T06:07:13.8922719Z ============================== 1 passed in 5.98s =============================== 2023-11-14T06:07:13.8924218Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-586c16254ba73641.xml 2023-11-14T06:07:13.8925569Z ============================= test session starts ============================== 2023-11-14T06:07:13.8926699Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8927734Z cachedir: .pytest_cache 2023-11-14T06:07:13.8928875Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8929976Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8930474Z configfile: pytest.ini 2023-11-14T06:07:13.8931557Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8932605Z collecting ... collected 1 item 2023-11-14T06:07:13.8933747Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2023-11-14T06:07:13.8934764Z 2023-11-14T06:07:13.8936559Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8939649Z STAGE:2023-11-14 05:46:38 5380:5380 ActivityProfilerController.cpp:314] Completed Stage: Warm UpSTAGE:2023-11-14 05:46:38 5381:5381 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8940981Z 2023-11-14T06:07:13.8941675Z STAGE:2023-11-14 05:46:38 5380:5380 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8943103Z STAGE:2023-11-14 05:46:38 5381:5381 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8944687Z STAGE:2023-11-14 05:46:38 5380:5380 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8946210Z STAGE:2023-11-14 05:46:38 5381:5381 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8947631Z STAGE:2023-11-14 05:46:38 5381:5381 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8948983Z STAGE:2023-11-14 05:46:38 5380:5380 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.8950394Z STAGE:2023-11-14 05:46:38 5381:5381 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8951807Z STAGE:2023-11-14 05:46:38 5380:5380 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.8953269Z STAGE:2023-11-14 05:46:38 5381:5381 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8954977Z STAGE:2023-11-14 05:46:38 5380:5380 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.8955948Z PASSED [5.9457s] [100%] 2023-11-14T06:07:13.8956227Z 2023-11-14T06:07:13.8957518Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-586c16254ba73641.xml - 2023-11-14T06:07:13.8959041Z ============================== 1 passed in 5.98s =============================== 2023-11-14T06:07:13.8960538Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb3fe0a9e2ff6b06.xml 2023-11-14T06:07:13.8961905Z ============================= test session starts ============================== 2023-11-14T06:07:13.8963041Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8963926Z cachedir: .pytest_cache 2023-11-14T06:07:13.8965066Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8966171Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8966678Z configfile: pytest.ini 2023-11-14T06:07:13.8967770Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8968823Z collecting ... collected 1 item 2023-11-14T06:07:13.8969948Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2023-11-14T06:07:13.8971094Z 2023-11-14T06:07:13.8972834Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8974854Z SKIPPED [5.8340s] (Skipped due to small world size.) [100%] 2023-11-14T06:07:13.8975358Z 2023-11-14T06:07:13.8976638Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb3fe0a9e2ff6b06.xml - 2023-11-14T06:07:13.8978212Z ============================== 1 skipped in 5.87s ============================== 2023-11-14T06:07:13.8979683Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-56091b807efd8598.xml 2023-11-14T06:07:13.8981041Z ============================= test session starts ============================== 2023-11-14T06:07:13.8982191Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.8983074Z cachedir: .pytest_cache 2023-11-14T06:07:13.8984217Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.8985326Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.8985830Z configfile: pytest.ini 2023-11-14T06:07:13.8986922Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.8988111Z collecting ... collected 1 item 2023-11-14T06:07:13.8989233Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2023-11-14T06:07:13.8990174Z 2023-11-14T06:07:13.8991907Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.8993931Z SKIPPED [5.9341s] (Skipped due to small world size.) [100%] 2023-11-14T06:07:13.8994457Z 2023-11-14T06:07:13.8995963Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-56091b807efd8598.xml - 2023-11-14T06:07:13.8997526Z ============================== 1 skipped in 5.97s ============================== 2023-11-14T06:07:13.8999010Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3f2b843826f7d53b.xml 2023-11-14T06:07:13.9000368Z ============================= test session starts ============================== 2023-11-14T06:07:13.9001502Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9002397Z cachedir: .pytest_cache 2023-11-14T06:07:13.9003528Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9004639Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9005166Z configfile: pytest.ini 2023-11-14T06:07:13.9006233Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9007282Z collecting ... collected 1 item 2023-11-14T06:07:13.9008339Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2023-11-14T06:07:13.9009240Z 2023-11-14T06:07:13.9010950Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9013268Z STAGE:2023-11-14 05:47:12 5444:5444 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9014827Z STAGE:2023-11-14 05:47:12 5445:5445 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9016231Z STAGE:2023-11-14 05:47:12 5444:5444 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9017619Z STAGE:2023-11-14 05:47:12 5445:5445 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9019080Z STAGE:2023-11-14 05:47:12 5444:5444 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9020579Z STAGE:2023-11-14 05:47:12 5445:5445 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9021990Z STAGE:2023-11-14 05:47:12 5445:5445 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9023374Z STAGE:2023-11-14 05:47:12 5444:5444 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9024771Z STAGE:2023-11-14 05:47:12 5444:5444 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9026196Z STAGE:2023-11-14 05:47:12 5445:5445 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9027624Z STAGE:2023-11-14 05:47:12 5444:5444 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9029102Z STAGE:2023-11-14 05:47:12 5445:5445 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9030046Z PASSED [5.8341s] [100%] 2023-11-14T06:07:13.9030324Z 2023-11-14T06:07:13.9031713Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3f2b843826f7d53b.xml - 2023-11-14T06:07:13.9033266Z ============================== 1 passed in 5.87s =============================== 2023-11-14T06:07:13.9035005Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5f9f441da3ea77ca.xml 2023-11-14T06:07:13.9036378Z ============================= test session starts ============================== 2023-11-14T06:07:13.9037504Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9038418Z cachedir: .pytest_cache 2023-11-14T06:07:13.9039561Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9040639Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9041175Z configfile: pytest.ini 2023-11-14T06:07:13.9042278Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9043333Z collecting ... collected 1 item 2023-11-14T06:07:13.9044405Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_multigpu 2023-11-14T06:07:13.9045366Z 2023-11-14T06:07:13.9047092Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_multigpu <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9049457Z STAGE:2023-11-14 05:47:24 5466:5466 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9050838Z STAGE:2023-11-14 05:47:24 5467:5467 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9052217Z STAGE:2023-11-14 05:47:24 5467:5467 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9053641Z STAGE:2023-11-14 05:47:24 5466:5466 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9055101Z STAGE:2023-11-14 05:47:24 5467:5467 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9056567Z STAGE:2023-11-14 05:47:24 5466:5466 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9057994Z STAGE:2023-11-14 05:47:24 5466:5466 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9059544Z STAGE:2023-11-14 05:47:24 5467:5467 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9060940Z STAGE:2023-11-14 05:47:24 5466:5466 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9062322Z STAGE:2023-11-14 05:47:24 5467:5467 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9063784Z STAGE:2023-11-14 05:47:24 5466:5466 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9065276Z STAGE:2023-11-14 05:47:24 5467:5467 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9066202Z PASSED [6.1419s] [100%] 2023-11-14T06:07:13.9066515Z 2023-11-14T06:07:13.9067764Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5f9f441da3ea77ca.xml - 2023-11-14T06:07:13.9069376Z ============================== 1 passed in 6.18s =============================== 2023-11-14T06:07:13.9070888Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ac8dc59b1805052.xml 2023-11-14T06:07:13.9072219Z ============================= test session starts ============================== 2023-11-14T06:07:13.9073353Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9074259Z cachedir: .pytest_cache 2023-11-14T06:07:13.9075798Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9076924Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9077460Z configfile: pytest.ini 2023-11-14T06:07:13.9078532Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9079581Z collecting ... collected 1 item 2023-11-14T06:07:13.9080733Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_multigpu_complex 2023-11-14T06:07:13.9081711Z 2023-11-14T06:07:13.9083517Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_multigpu_complex <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9085895Z STAGE:2023-11-14 05:47:35 5491:5491 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9087289Z STAGE:2023-11-14 05:47:35 5490:5490 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9088694Z STAGE:2023-11-14 05:47:35 5490:5490 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9090114Z STAGE:2023-11-14 05:47:35 5491:5491 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9091546Z STAGE:2023-11-14 05:47:35 5490:5490 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9093054Z STAGE:2023-11-14 05:47:35 5491:5491 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9094479Z STAGE:2023-11-14 05:47:35 5491:5491 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9095829Z STAGE:2023-11-14 05:47:35 5490:5490 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9097234Z STAGE:2023-11-14 05:47:35 5490:5490 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9098658Z STAGE:2023-11-14 05:47:35 5491:5491 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9100117Z STAGE:2023-11-14 05:47:35 5490:5490 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9101579Z STAGE:2023-11-14 05:47:35 5491:5491 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9102664Z PASSED [6.0347s] [100%] 2023-11-14T06:07:13.9102941Z 2023-11-14T06:07:13.9104226Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ac8dc59b1805052.xml - 2023-11-14T06:07:13.9105786Z ============================== 1 passed in 6.07s =============================== 2023-11-14T06:07:13.9107260Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b3fa2a0cf405b83a.xml 2023-11-14T06:07:13.9108624Z ============================= test session starts ============================== 2023-11-14T06:07:13.9109755Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9110639Z cachedir: .pytest_cache 2023-11-14T06:07:13.9111776Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9112876Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9113388Z configfile: pytest.ini 2023-11-14T06:07:13.9114482Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9115782Z collecting ... collected 1 item 2023-11-14T06:07:13.9116878Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2023-11-14T06:07:13.9117804Z 2023-11-14T06:07:13.9119653Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9122710Z STAGE:2023-11-14 05:47:47 5514:5514 ActivityProfilerController.cpp:314] Completed Stage: Warm UpSTAGE:2023-11-14 05:47:47 5515:5515 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9124061Z 2023-11-14T06:07:13.9125440Z STAGE:2023-11-14 05:47:47 5515:5515 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 05:47:47 5514:5514 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9126800Z 2023-11-14T06:07:13.9128269Z STAGE:2023-11-14 05:47:47 5515:5515 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 05:47:47 5514:5514 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9129698Z 2023-11-14T06:07:13.9130377Z STAGE:2023-11-14 05:47:47 5514:5514 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9131730Z STAGE:2023-11-14 05:47:47 5515:5515 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9133131Z STAGE:2023-11-14 05:47:47 5514:5514 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9134546Z STAGE:2023-11-14 05:47:47 5515:5515 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9136711Z STAGE:2023-11-14 05:47:47 5515:5515 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 05:47:47 5514:5514 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9138168Z 2023-11-14T06:07:13.9138327Z PASSED [5.8333s] [100%] 2023-11-14T06:07:13.9138633Z 2023-11-14T06:07:13.9139890Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b3fa2a0cf405b83a.xml - 2023-11-14T06:07:13.9141455Z ============================== 1 passed in 5.87s =============================== 2023-11-14T06:07:13.9142927Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-72bfca61089b0504.xml 2023-11-14T06:07:13.9144284Z ============================= test session starts ============================== 2023-11-14T06:07:13.9145414Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9146472Z cachedir: .pytest_cache 2023-11-14T06:07:13.9147590Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9148700Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9149227Z configfile: pytest.ini 2023-11-14T06:07:13.9150289Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9151347Z collecting ... collected 1 item 2023-11-14T06:07:13.9152469Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_result_cuda 2023-11-14T06:07:13.9153424Z 2023-11-14T06:07:13.9155439Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_result_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9157316Z PASSED [5.9420s] [100%] 2023-11-14T06:07:13.9157627Z 2023-11-14T06:07:13.9158877Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-72bfca61089b0504.xml - 2023-11-14T06:07:13.9160435Z ============================== 1 passed in 5.98s =============================== 2023-11-14T06:07:13.9161932Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-584714f9fc9a72c5.xml 2023-11-14T06:07:13.9163386Z ============================= test session starts ============================== 2023-11-14T06:07:13.9164604Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9165524Z cachedir: .pytest_cache 2023-11-14T06:07:13.9166633Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9167751Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9168278Z configfile: pytest.ini 2023-11-14T06:07:13.9169378Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9170442Z collecting ... collected 1 item 2023-11-14T06:07:13.9171506Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum 2023-11-14T06:07:13.9172410Z 2023-11-14T06:07:13.9174141Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9176440Z STAGE:2023-11-14 05:48:09 5557:5557 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9177826Z STAGE:2023-11-14 05:48:09 5556:5556 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9179938Z STAGE:2023-11-14 05:48:09 5557:5557 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 05:48:09 5556:5556 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9181299Z 2023-11-14T06:07:13.9182765Z STAGE:2023-11-14 05:48:09 5557:5557 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 05:48:09 5556:5556 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9184202Z 2023-11-14T06:07:13.9184849Z STAGE:2023-11-14 05:48:09 5556:5556 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9186231Z STAGE:2023-11-14 05:48:09 5557:5557 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9187632Z STAGE:2023-11-14 05:48:09 5556:5556 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9189055Z STAGE:2023-11-14 05:48:09 5557:5557 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9190635Z STAGE:2023-11-14 05:48:09 5556:5556 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9192118Z STAGE:2023-11-14 05:48:09 5557:5557 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9193068Z PASSED [5.9414s] [100%] 2023-11-14T06:07:13.9193348Z 2023-11-14T06:07:13.9194818Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-584714f9fc9a72c5.xml - 2023-11-14T06:07:13.9196402Z ============================== 1 passed in 5.98s =============================== 2023-11-14T06:07:13.9197904Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0adf1aab22feb1c2.xml 2023-11-14T06:07:13.9199268Z ============================= test session starts ============================== 2023-11-14T06:07:13.9200371Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9201299Z cachedir: .pytest_cache 2023-11-14T06:07:13.9202444Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9203532Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9204065Z configfile: pytest.ini 2023-11-14T06:07:13.9205156Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9206210Z collecting ... collected 1 item 2023-11-14T06:07:13.9207431Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2023-11-14T06:07:13.9208415Z 2023-11-14T06:07:13.9210151Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9212511Z STAGE:2023-11-14 05:48:21 5578:5578 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9213893Z STAGE:2023-11-14 05:48:21 5579:5579 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9215266Z STAGE:2023-11-14 05:48:21 5578:5578 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9216683Z STAGE:2023-11-14 05:48:21 5579:5579 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9218881Z STAGE:2023-11-14 05:48:21 5578:5578 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 05:48:21 5579:5579 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9220307Z 2023-11-14T06:07:13.9221656Z STAGE:2023-11-14 05:48:21 5578:5578 ActivityProfilerController.cpp:314] Completed Stage: Warm UpSTAGE:2023-11-14 05:48:21 5579:5579 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9222986Z 2023-11-14T06:07:13.9224344Z STAGE:2023-11-14 05:48:21 5579:5579 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 05:48:21 5578:5578 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9225734Z 2023-11-14T06:07:13.9227170Z STAGE:2023-11-14 05:48:21 5578:5578 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 05:48:21 5579:5579 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9228626Z 2023-11-14T06:07:13.9229950Z STAGE:2023-11-14 05:48:21 5578:5578 ActivityProfilerController.cpp:314] Completed Stage: Warm UpSTAGE:2023-11-14 05:48:21 5579:5579 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9231272Z 2023-11-14T06:07:13.9231974Z STAGE:2023-11-14 05:48:21 5578:5578 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9233387Z STAGE:2023-11-14 05:48:21 5579:5579 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9235154Z STAGE:2023-11-14 05:48:21 5578:5578 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9236657Z STAGE:2023-11-14 05:48:21 5579:5579 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9237604Z PASSED [6.1447s] [100%] 2023-11-14T06:07:13.9237885Z 2023-11-14T06:07:13.9239151Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0adf1aab22feb1c2.xml - 2023-11-14T06:07:13.9240733Z ============================== 1 passed in 6.18s =============================== 2023-11-14T06:07:13.9242225Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-42842533df8b252f.xml 2023-11-14T06:07:13.9243575Z ============================= test session starts ============================== 2023-11-14T06:07:13.9244690Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9245600Z cachedir: .pytest_cache 2023-11-14T06:07:13.9246744Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9247829Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9248354Z configfile: pytest.ini 2023-11-14T06:07:13.9249441Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9250593Z collecting ... collected 1 item 2023-11-14T06:07:13.9251646Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2023-11-14T06:07:13.9252546Z 2023-11-14T06:07:13.9254341Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only MPI supports all_to_all) [100%] 2023-11-14T06:07:13.9256111Z 2023-11-14T06:07:13.9257389Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-42842533df8b252f.xml - 2023-11-14T06:07:13.9258943Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:13.9260408Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c4120e466f4e713.xml 2023-11-14T06:07:13.9261780Z ============================= test session starts ============================== 2023-11-14T06:07:13.9262911Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9263791Z cachedir: .pytest_cache 2023-11-14T06:07:13.9264936Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9266054Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9266556Z configfile: pytest.ini 2023-11-14T06:07:13.9267642Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9268692Z collecting ... collected 1 item 2023-11-14T06:07:13.9269844Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group 2023-11-14T06:07:13.9270792Z 2023-11-14T06:07:13.9272662Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Only MPI supports all_to_all) [100%] 2023-11-14T06:07:13.9274504Z 2023-11-14T06:07:13.9276145Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c4120e466f4e713.xml - 2023-11-14T06:07:13.9277871Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:13.9279366Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b365cf4e8961d93.xml 2023-11-14T06:07:13.9280689Z ============================= test session starts ============================== 2023-11-14T06:07:13.9281821Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9282750Z cachedir: .pytest_cache 2023-11-14T06:07:13.9283867Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9284975Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9285508Z configfile: pytest.ini 2023-11-14T06:07:13.9286594Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9287632Z collecting ... collected 1 item 2023-11-14T06:07:13.9288705Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2023-11-14T06:07:13.9289616Z 2023-11-14T06:07:13.9291477Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only MPI supports all_to_all) [100%] 2023-11-14T06:07:13.9293264Z 2023-11-14T06:07:13.9294658Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b365cf4e8961d93.xml - 2023-11-14T06:07:13.9296205Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:13.9297716Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b3f3afa0f570c718.xml 2023-11-14T06:07:13.9299079Z ============================= test session starts ============================== 2023-11-14T06:07:13.9300213Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9301101Z cachedir: .pytest_cache 2023-11-14T06:07:13.9302234Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9303344Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9303856Z configfile: pytest.ini 2023-11-14T06:07:13.9304948Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9306004Z collecting ... collected 1 item 2023-11-14T06:07:13.9307196Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group 2023-11-14T06:07:13.9308288Z 2023-11-14T06:07:13.9310344Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only MPI supports CPU all_to_all_single) [100%] 2023-11-14T06:07:13.9312352Z 2023-11-14T06:07:13.9313606Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b3f3afa0f570c718.xml - 2023-11-14T06:07:13.9315447Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:13.9316966Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-773af5119130b2a2.xml 2023-11-14T06:07:13.9318297Z ============================= test session starts ============================== 2023-11-14T06:07:13.9319430Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9320516Z cachedir: .pytest_cache 2023-11-14T06:07:13.9321628Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9322742Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9323272Z configfile: pytest.ini 2023-11-14T06:07:13.9324334Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9325386Z collecting ... collected 1 item 2023-11-14T06:07:13.9326563Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2023-11-14T06:07:13.9327569Z 2023-11-14T06:07:13.9329589Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only MPI supports CPU all_to_all_single) [100%] 2023-11-14T06:07:13.9331530Z 2023-11-14T06:07:13.9332772Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-773af5119130b2a2.xml - 2023-11-14T06:07:13.9334330Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:13.9335826Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f3aded032445a5a.xml 2023-11-14T06:07:13.9337313Z ============================= test session starts ============================== 2023-11-14T06:07:13.9338444Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9339360Z cachedir: .pytest_cache 2023-11-14T06:07:13.9340511Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9341605Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9342133Z configfile: pytest.ini 2023-11-14T06:07:13.9343220Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9344272Z collecting ... collected 1 item 2023-11-14T06:07:13.9345462Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2023-11-14T06:07:13.9346536Z 2023-11-14T06:07:13.9348602Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Only MPI supports CPU all_to_all_single) [100%] 2023-11-14T06:07:13.9350607Z 2023-11-14T06:07:13.9351851Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f3aded032445a5a.xml - 2023-11-14T06:07:13.9353417Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:13.9355130Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3710ed8320dbe006.xml 2023-11-14T06:07:13.9356491Z ============================= test session starts ============================== 2023-11-14T06:07:13.9357647Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9358572Z cachedir: .pytest_cache 2023-11-14T06:07:13.9359681Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9360786Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9361308Z configfile: pytest.ini 2023-11-14T06:07:13.9362376Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9363612Z collecting ... collected 1 item 2023-11-14T06:07:13.9364867Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2023-11-14T06:07:13.9365961Z 2023-11-14T06:07:13.9368113Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only Nccl supports CUDA all_to_all_single) [100%] 2023-11-14T06:07:13.9370197Z 2023-11-14T06:07:13.9371445Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3710ed8320dbe006.xml - 2023-11-14T06:07:13.9373012Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:13.9374520Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-64de7f16b808587e.xml 2023-11-14T06:07:13.9375879Z ============================= test session starts ============================== 2023-11-14T06:07:13.9376986Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9377905Z cachedir: .pytest_cache 2023-11-14T06:07:13.9379163Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9380264Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9380794Z configfile: pytest.ini 2023-11-14T06:07:13.9381896Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9382927Z collecting ... collected 1 item 2023-11-14T06:07:13.9384131Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2023-11-14T06:07:13.9385207Z 2023-11-14T06:07:13.9387255Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Only MPI supports CPU all_to_all_single) [100%] 2023-11-14T06:07:13.9389218Z 2023-11-14T06:07:13.9390504Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-64de7f16b808587e.xml - 2023-11-14T06:07:13.9392070Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:13.9393537Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13fa052b8a05419a.xml 2023-11-14T06:07:13.9395157Z ============================= test session starts ============================== 2023-11-14T06:07:13.9396317Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9397204Z cachedir: .pytest_cache 2023-11-14T06:07:13.9398339Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9399451Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9399953Z configfile: pytest.ini 2023-11-14T06:07:13.9401046Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9402098Z collecting ... collected 1 item 2023-11-14T06:07:13.9403327Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda 2023-11-14T06:07:13.9404393Z 2023-11-14T06:07:13.9406478Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only Nccl supports CUDA all_to_all_single) [100%] 2023-11-14T06:07:13.9408663Z 2023-11-14T06:07:13.9409911Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13fa052b8a05419a.xml - 2023-11-14T06:07:13.9411473Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:13.9412979Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c3ee94082f625a8c.xml 2023-11-14T06:07:13.9414315Z ============================= test session starts ============================== 2023-11-14T06:07:13.9415444Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9416365Z cachedir: .pytest_cache 2023-11-14T06:07:13.9417506Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9418589Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9419120Z configfile: pytest.ini 2023-11-14T06:07:13.9420208Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9421240Z collecting ... collected 1 item 2023-11-14T06:07:13.9422497Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group 2023-11-14T06:07:13.9423445Z 2023-11-14T06:07:13.9425206Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_backend_full_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9427064Z PASSED [6.7405s] [100%] 2023-11-14T06:07:13.9427357Z 2023-11-14T06:07:13.9428607Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c3ee94082f625a8c.xml - 2023-11-14T06:07:13.9430166Z ============================== 1 passed in 6.78s =============================== 2023-11-14T06:07:13.9431658Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-78718b57f1dc0d99.xml 2023-11-14T06:07:13.9433016Z ============================= test session starts ============================== 2023-11-14T06:07:13.9434130Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9435290Z cachedir: .pytest_cache 2023-11-14T06:07:13.9436436Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9437520Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9438062Z configfile: pytest.ini 2023-11-14T06:07:13.9439152Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9440178Z collecting ... collected 1 item 2023-11-14T06:07:13.9441197Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier 2023-11-14T06:07:13.9442057Z 2023-11-14T06:07:13.9443748Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9445544Z PASSED [6.7452s] [100%] 2023-11-14T06:07:13.9445826Z 2023-11-14T06:07:13.9447071Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-78718b57f1dc0d99.xml - 2023-11-14T06:07:13.9448776Z ============================== 1 passed in 6.78s =============================== 2023-11-14T06:07:13.9450270Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ef633029817cc5b.xml 2023-11-14T06:07:13.9451619Z ============================= test session starts ============================== 2023-11-14T06:07:13.9452716Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9453626Z cachedir: .pytest_cache 2023-11-14T06:07:13.9454774Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9455851Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9456374Z configfile: pytest.ini 2023-11-14T06:07:13.9457468Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9458490Z collecting ... collected 1 item 2023-11-14T06:07:13.9459623Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda 2023-11-14T06:07:13.9460602Z 2023-11-14T06:07:13.9462358Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9464384Z SKIPPED [5.9433s] (Skipped due to small world size.) [100%] 2023-11-14T06:07:13.9464948Z 2023-11-14T06:07:13.9466312Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ef633029817cc5b.xml - 2023-11-14T06:07:13.9467889Z ============================== 1 skipped in 5.98s ============================== 2023-11-14T06:07:13.9469426Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-69699f97e0448663.xml 2023-11-14T06:07:13.9470787Z ============================= test session starts ============================== 2023-11-14T06:07:13.9471890Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9472800Z cachedir: .pytest_cache 2023-11-14T06:07:13.9473937Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9475326Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9475846Z configfile: pytest.ini 2023-11-14T06:07:13.9476943Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9477970Z collecting ... collected 1 item 2023-11-14T06:07:13.9479012Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2023-11-14T06:07:13.9479930Z 2023-11-14T06:07:13.9481634Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9483606Z SKIPPED [5.8344s] (Skipped due to small world size.) [100%] 2023-11-14T06:07:13.9484101Z 2023-11-14T06:07:13.9485371Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-69699f97e0448663.xml - 2023-11-14T06:07:13.9485651Z ============================== 1 skipped in 5.87s ============================== 2023-11-14T06:07:13.9486682Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a8dc7bb5c9ccf400.xml 2023-11-14T06:07:13.9486985Z ============================= test session starts ============================== 2023-11-14T06:07:13.9487638Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9487955Z cachedir: .pytest_cache 2023-11-14T06:07:13.9488835Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9489018Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9489186Z configfile: pytest.ini 2023-11-14T06:07:13.9490009Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9490184Z collecting ... collected 1 item 2023-11-14T06:07:13.9490964Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2023-11-14T06:07:13.9490985Z 2023-11-14T06:07:13.9492705Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9493025Z SKIPPED [5.8433s] (Skipped due to small world size.) [100%] 2023-11-14T06:07:13.9493067Z 2023-11-14T06:07:13.9494313Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a8dc7bb5c9ccf400.xml - 2023-11-14T06:07:13.9494584Z ============================== 1 skipped in 5.88s ============================== 2023-11-14T06:07:13.9495776Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-57967a7a01c6de2f.xml 2023-11-14T06:07:13.9496078Z ============================= test session starts ============================== 2023-11-14T06:07:13.9496736Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9496925Z cachedir: .pytest_cache 2023-11-14T06:07:13.9497775Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9497968Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9498149Z configfile: pytest.ini 2023-11-14T06:07:13.9498951Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9499145Z collecting ... collected 1 item 2023-11-14T06:07:13.9499907Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group 2023-11-14T06:07:13.9499929Z 2023-11-14T06:07:13.9501664Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9501993Z SKIPPED [5.9470s] (Skipped due to small world size.) [100%] 2023-11-14T06:07:13.9502014Z 2023-11-14T06:07:13.9503255Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-57967a7a01c6de2f.xml - 2023-11-14T06:07:13.9503555Z ============================== 1 skipped in 5.99s ============================== 2023-11-14T06:07:13.9504567Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8af2b6da1bf0e86a.xml 2023-11-14T06:07:13.9504838Z ============================= test session starts ============================== 2023-11-14T06:07:13.9505521Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9505694Z cachedir: .pytest_cache 2023-11-14T06:07:13.9506542Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9506754Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9506914Z configfile: pytest.ini 2023-11-14T06:07:13.9507867Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9508041Z collecting ... collected 1 item 2023-11-14T06:07:13.9508810Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2023-11-14T06:07:13.9508831Z 2023-11-14T06:07:13.9510604Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9510766Z PASSED [5.9356s] [100%] 2023-11-14T06:07:13.9510785Z 2023-11-14T06:07:13.9512051Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8af2b6da1bf0e86a.xml - 2023-11-14T06:07:13.9512322Z ============================== 1 passed in 5.97s =============================== 2023-11-14T06:07:13.9513356Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15efff3a1d0272f1.xml 2023-11-14T06:07:13.9513656Z ============================= test session starts ============================== 2023-11-14T06:07:13.9514306Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9514468Z cachedir: .pytest_cache 2023-11-14T06:07:13.9515723Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9515932Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9516114Z configfile: pytest.ini 2023-11-14T06:07:13.9516920Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9517093Z collecting ... collected 1 item 2023-11-14T06:07:13.9517917Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2023-11-14T06:07:13.9517945Z 2023-11-14T06:07:13.9519723Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9519903Z PASSED [5.7362s] [100%] 2023-11-14T06:07:13.9519923Z 2023-11-14T06:07:13.9521171Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15efff3a1d0272f1.xml - 2023-11-14T06:07:13.9521444Z ============================== 1 passed in 5.77s =============================== 2023-11-14T06:07:13.9522483Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ea0c5f0b2bfdb0c.xml 2023-11-14T06:07:13.9522768Z ============================= test session starts ============================== 2023-11-14T06:07:13.9523414Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9523601Z cachedir: .pytest_cache 2023-11-14T06:07:13.9524448Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9524651Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9524810Z configfile: pytest.ini 2023-11-14T06:07:13.9525625Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9525832Z collecting ... collected 1 item 2023-11-14T06:07:13.9526674Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err 2023-11-14T06:07:13.9526695Z 2023-11-14T06:07:13.9528655Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (NCCL Batch Send Recv Only) [100%] 2023-11-14T06:07:13.9528795Z 2023-11-14T06:07:13.9530065Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ea0c5f0b2bfdb0c.xml - 2023-11-14T06:07:13.9530337Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:13.9531380Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1fd204ab22b27a7f.xml 2023-11-14T06:07:13.9531656Z ============================= test session starts ============================== 2023-11-14T06:07:13.9532326Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9532497Z cachedir: .pytest_cache 2023-11-14T06:07:13.9533343Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9533553Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9533714Z configfile: pytest.ini 2023-11-14T06:07:13.9534507Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9534708Z collecting ... collected 1 item 2023-11-14T06:07:13.9535571Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2023-11-14T06:07:13.9535595Z 2023-11-14T06:07:13.9537489Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (NCCL Batch Send Recv Only) [100%] 2023-11-14T06:07:13.9537517Z 2023-11-14T06:07:13.9538765Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1fd204ab22b27a7f.xml - 2023-11-14T06:07:13.9539045Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:13.9540083Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9199eda87445e0b1.xml 2023-11-14T06:07:13.9540360Z ============================= test session starts ============================== 2023-11-14T06:07:13.9541030Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9541192Z cachedir: .pytest_cache 2023-11-14T06:07:13.9542034Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9542242Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9542416Z configfile: pytest.ini 2023-11-14T06:07:13.9543214Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9543408Z collecting ... collected 1 item 2023-11-14T06:07:13.9544251Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2023-11-14T06:07:13.9544272Z 2023-11-14T06:07:13.9546216Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (NCCL Batch Send Recv Only) [100%] 2023-11-14T06:07:13.9546238Z 2023-11-14T06:07:13.9547477Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9199eda87445e0b1.xml - 2023-11-14T06:07:13.9547779Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:13.9548919Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c56409f0b30ce821.xml 2023-11-14T06:07:13.9549192Z ============================= test session starts ============================== 2023-11-14T06:07:13.9549865Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9550028Z cachedir: .pytest_cache 2023-11-14T06:07:13.9550875Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9551084Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9551244Z configfile: pytest.ini 2023-11-14T06:07:13.9552039Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9552243Z collecting ... collected 1 item 2023-11-14T06:07:13.9553054Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2023-11-14T06:07:13.9553075Z 2023-11-14T06:07:13.9555221Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (NCCL Batch Send Recv Only) [100%] 2023-11-14T06:07:13.9555245Z 2023-11-14T06:07:13.9556633Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c56409f0b30ce821.xml - 2023-11-14T06:07:13.9556953Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:13.9557978Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e8f5add6ec72509.xml 2023-11-14T06:07:13.9558268Z ============================= test session starts ============================== 2023-11-14T06:07:13.9558942Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9559107Z cachedir: .pytest_cache 2023-11-14T06:07:13.9559949Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9560154Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9560312Z configfile: pytest.ini 2023-11-14T06:07:13.9561137Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9561316Z collecting ... collected 1 item 2023-11-14T06:07:13.9562036Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2023-11-14T06:07:13.9562057Z 2023-11-14T06:07:13.9563773Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9565106Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:314] Completed Stage: Warm UpSTAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9565128Z 2023-11-14T06:07:13.9565814Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9566473Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9567173Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9567892Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9568526Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9569375Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9570056Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9570699Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9571420Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9572115Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9572745Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9573397Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9574052Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9574740Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9575435Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9576119Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9576776Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9577500Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9578182Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9578852Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9579541Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9580270Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9580900Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9581537Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9582219Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9582883Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9583577Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9584286Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9584931Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9585584Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9586236Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9586880Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9587592Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9588288Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9588944Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9589577Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9590358Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9591028Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9591717Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9592408Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9593065Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9593698Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9594378Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9595263Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9595983Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9596694Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9597326Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9597955Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9598770Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9599456Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9600174Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9600868Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9601509Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9602164Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9602817Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9603471Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9604190Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9604882Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9605535Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9606166Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9606837Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9607518Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9608213Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9608929Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9609563Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9610199Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9610873Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9611525Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9612442Z STAGE:2023-11-14 05:51:16 5798:5798 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9613163Z STAGE:2023-11-14 05:51:16 5797:5797 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9613795Z STAGE:2023-11-14 05:51:17 5798:5798 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9614452Z STAGE:2023-11-14 05:51:17 5797:5797 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9615116Z STAGE:2023-11-14 05:51:17 5797:5797 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9615775Z STAGE:2023-11-14 05:51:17 5798:5798 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9616493Z STAGE:2023-11-14 05:51:17 5797:5797 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9617182Z STAGE:2023-11-14 05:51:17 5798:5798 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9617822Z STAGE:2023-11-14 05:51:17 5797:5797 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9618477Z STAGE:2023-11-14 05:51:17 5798:5798 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:13.9619130Z STAGE:2023-11-14 05:51:17 5798:5798 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9619797Z STAGE:2023-11-14 05:51:17 5797:5797 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:13.9620584Z STAGE:2023-11-14 05:51:17 5798:5798 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9621297Z STAGE:2023-11-14 05:51:17 5797:5797 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:13.9621481Z PASSED [6.1464s] [100%] 2023-11-14T06:07:13.9621503Z 2023-11-14T06:07:13.9622751Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e8f5add6ec72509.xml - 2023-11-14T06:07:13.9623058Z ============================== 1 passed in 6.18s =============================== 2023-11-14T06:07:13.9624075Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d760b56f723bba08.xml 2023-11-14T06:07:13.9624345Z ============================= test session starts ============================== 2023-11-14T06:07:13.9625021Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9625193Z cachedir: .pytest_cache 2023-11-14T06:07:13.9626043Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9626255Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9626414Z configfile: pytest.ini 2023-11-14T06:07:13.9627242Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9627430Z collecting ... collected 1 item 2023-11-14T06:07:13.9628432Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2023-11-14T06:07:13.9628454Z 2023-11-14T06:07:13.9630470Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9630635Z PASSED [6.1372s] [100%] 2023-11-14T06:07:13.9630657Z 2023-11-14T06:07:13.9631920Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d760b56f723bba08.xml - 2023-11-14T06:07:13.9632189Z ============================== 1 passed in 6.17s =============================== 2023-11-14T06:07:13.9633323Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a67403e21a0e67d2.xml 2023-11-14T06:07:13.9633625Z ============================= test session starts ============================== 2023-11-14T06:07:13.9634274Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9634435Z cachedir: .pytest_cache 2023-11-14T06:07:13.9635506Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9635692Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9635876Z configfile: pytest.ini 2023-11-14T06:07:13.9636680Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9636854Z collecting ... collected 1 item 2023-11-14T06:07:13.9637691Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2023-11-14T06:07:13.9637713Z 2023-11-14T06:07:13.9639486Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9639674Z PASSED [6.4353s] [100%] 2023-11-14T06:07:13.9639695Z 2023-11-14T06:07:13.9641077Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a67403e21a0e67d2.xml - 2023-11-14T06:07:13.9641375Z ============================== 1 passed in 6.47s =============================== 2023-11-14T06:07:13.9642425Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cae12df9d44ec4fe.xml 2023-11-14T06:07:13.9642711Z ============================= test session starts ============================== 2023-11-14T06:07:13.9643385Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9643548Z cachedir: .pytest_cache 2023-11-14T06:07:13.9644390Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9644599Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9644757Z configfile: pytest.ini 2023-11-14T06:07:13.9645563Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9645771Z collecting ... collected 1 item 2023-11-14T06:07:13.9646641Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2023-11-14T06:07:13.9646662Z 2023-11-14T06:07:13.9648538Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9648705Z PASSED [6.1341s] [100%] 2023-11-14T06:07:13.9648726Z 2023-11-14T06:07:13.9649969Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cae12df9d44ec4fe.xml - 2023-11-14T06:07:13.9650271Z ============================== 1 passed in 6.17s =============================== 2023-11-14T06:07:13.9651290Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e1b2c0e952cc14c.xml 2023-11-14T06:07:13.9651591Z ============================= test session starts ============================== 2023-11-14T06:07:13.9652237Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9652540Z cachedir: .pytest_cache 2023-11-14T06:07:13.9653419Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9653602Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9653764Z configfile: pytest.ini 2023-11-14T06:07:13.9654587Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9654762Z collecting ... collected 1 item 2023-11-14T06:07:13.9655625Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping 2023-11-14T06:07:13.9655670Z 2023-11-14T06:07:13.9657502Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9657781Z 2023-11-14T06:07:13.9657961Z PASSED [5.8421s] [100%] 2023-11-14T06:07:13.9657981Z 2023-11-14T06:07:13.9659223Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e1b2c0e952cc14c.xml - 2023-11-14T06:07:13.9659520Z ============================== 1 passed in 5.88s =============================== 2023-11-14T06:07:13.9660627Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-008a112b0c5906b8.xml 2023-11-14T06:07:13.9660920Z ============================= test session starts ============================== 2023-11-14T06:07:13.9661601Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9661761Z cachedir: .pytest_cache 2023-11-14T06:07:13.9662609Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9662827Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9662987Z configfile: pytest.ini 2023-11-14T06:07:13.9663788Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9663990Z collecting ... collected 1 item 2023-11-14T06:07:13.9664753Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2023-11-14T06:07:13.9664773Z 2023-11-14T06:07:13.9666539Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9666702Z PASSED [6.0353s] [100%] 2023-11-14T06:07:13.9666722Z 2023-11-14T06:07:13.9667962Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-008a112b0c5906b8.xml - 2023-11-14T06:07:13.9668265Z ============================== 1 passed in 6.07s =============================== 2023-11-14T06:07:13.9669327Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2563a37ec75fac00.xml 2023-11-14T06:07:13.9669632Z ============================= test session starts ============================== 2023-11-14T06:07:13.9670286Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9670452Z cachedir: .pytest_cache 2023-11-14T06:07:13.9671321Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9671502Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9671659Z configfile: pytest.ini 2023-11-14T06:07:13.9672609Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9672788Z collecting ... collected 1 item 2023-11-14T06:07:13.9673681Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2023-11-14T06:07:13.9673703Z 2023-11-14T06:07:13.9675753Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9679073Z [rank1]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:13.9679244Z PASSED [6.1381s] [100%] 2023-11-14T06:07:13.9679264Z 2023-11-14T06:07:13.9680515Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2563a37ec75fac00.xml - 2023-11-14T06:07:13.9680935Z ============================== 1 passed in 6.18s =============================== 2023-11-14T06:07:13.9681987Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bc15b85b75a4b4d5.xml 2023-11-14T06:07:13.9682286Z ============================= test session starts ============================== 2023-11-14T06:07:13.9682932Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9683105Z cachedir: .pytest_cache 2023-11-14T06:07:13.9683980Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9684164Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9684325Z configfile: pytest.ini 2023-11-14T06:07:13.9685152Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9685326Z collecting ... collected 1 item 2023-11-14T06:07:13.9686200Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2023-11-14T06:07:13.9686221Z 2023-11-14T06:07:13.9688045Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9691368Z [rank1]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:13.9694653Z [rank0]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:13.9694977Z PASSED [6.0436s] [100%] 2023-11-14T06:07:13.9694997Z 2023-11-14T06:07:13.9696251Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bc15b85b75a4b4d5.xml - 2023-11-14T06:07:13.9696524Z ============================== 1 passed in 6.08s =============================== 2023-11-14T06:07:13.9697567Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4702bdc83785d1e8.xml 2023-11-14T06:07:13.9697843Z ============================= test session starts ============================== 2023-11-14T06:07:13.9698516Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9698677Z cachedir: .pytest_cache 2023-11-14T06:07:13.9699526Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9699748Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9699908Z configfile: pytest.ini 2023-11-14T06:07:13.9700715Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9700916Z collecting ... collected 1 item 2023-11-14T06:07:13.9701645Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2023-11-14T06:07:13.9701762Z 2023-11-14T06:07:13.9703516Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9706132Z [rank1]:[W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-11-14T06:07:13.9708754Z [rank0]:[W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-11-14T06:07:13.9712607Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py:266: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1175.) 2023-11-14T06:07:13.9713168Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-11-14T06:07:13.9717255Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py:266: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1175.) 2023-11-14T06:07:13.9717816Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-11-14T06:07:13.9720573Z [rank0]:[W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-11-14T06:07:13.9723180Z [rank1]:[W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-11-14T06:07:13.9725734Z [rank0]:[W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-11-14T06:07:13.9728446Z [rank1]:[W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-11-14T06:07:13.9731017Z [rank0]:[W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-11-14T06:07:13.9733619Z [rank1]:[W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-11-14T06:07:13.9736174Z [rank0]:[W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-11-14T06:07:13.9738759Z [rank1]:[W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-11-14T06:07:13.9741318Z [rank0]:[W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-11-14T06:07:13.9743906Z [rank1]:[W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2023-11-14T06:07:13.9744191Z PASSED [5.8425s] [100%] 2023-11-14T06:07:13.9744212Z 2023-11-14T06:07:13.9745484Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4702bdc83785d1e8.xml - 2023-11-14T06:07:13.9745757Z ============================== 1 passed in 5.88s =============================== 2023-11-14T06:07:13.9746766Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5440508529da0bf2.xml 2023-11-14T06:07:13.9747083Z ============================= test session starts ============================== 2023-11-14T06:07:13.9747740Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9747906Z cachedir: .pytest_cache 2023-11-14T06:07:13.9748779Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9748964Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9749166Z configfile: pytest.ini 2023-11-14T06:07:13.9749971Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9750150Z collecting ... collected 1 item 2023-11-14T06:07:13.9750870Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2023-11-14T06:07:13.9750892Z 2023-11-14T06:07:13.9752661Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9752863Z PASSED [6.0367s] [100%] 2023-11-14T06:07:13.9752883Z 2023-11-14T06:07:13.9754132Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5440508529da0bf2.xml - 2023-11-14T06:07:13.9754417Z ============================== 1 passed in 6.08s =============================== 2023-11-14T06:07:13.9771050Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3f5581195dc4f129.xml 2023-11-14T06:07:13.9771423Z ============================= test session starts ============================== 2023-11-14T06:07:13.9772171Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9772337Z cachedir: .pytest_cache 2023-11-14T06:07:13.9773214Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9773429Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9773590Z configfile: pytest.ini 2023-11-14T06:07:13.9774397Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9774616Z collecting ... collected 1 item 2023-11-14T06:07:13.9775417Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook 2023-11-14T06:07:13.9775442Z 2023-11-14T06:07:13.9777235Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9779710Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py:1351: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2023-11-14T06:07:13.9780401Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2023-11-14T06:07:13.9783108Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py:1351: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2023-11-14T06:07:13.9783768Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2023-11-14T06:07:13.9783952Z PASSED [5.9341s] [100%] 2023-11-14T06:07:13.9783973Z 2023-11-14T06:07:13.9785225Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3f5581195dc4f129.xml - 2023-11-14T06:07:13.9785537Z ============================== 1 passed in 5.97s =============================== 2023-11-14T06:07:13.9786553Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-700392a528786283.xml 2023-11-14T06:07:13.9786868Z ============================= test session starts ============================== 2023-11-14T06:07:13.9787517Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9787682Z cachedir: .pytest_cache 2023-11-14T06:07:13.9788531Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9788745Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9788936Z configfile: pytest.ini 2023-11-14T06:07:13.9789876Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9790078Z collecting ... collected 1 item 2023-11-14T06:07:13.9790822Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2023-11-14T06:07:13.9790843Z 2023-11-14T06:07:13.9792611Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9796238Z [rank1]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:13.9799542Z [rank0]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:13.9800251Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:07:13.9801062Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:07:13.9802534Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:07:13.9803301Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:07:13.9804969Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:07:13.9805563Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:07:13.9807005Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:07:13.9807724Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:07:13.9809222Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:07:13.9809967Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:07:13.9811617Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9738, in test_ddp_has_finalized 2023-11-14T06:07:13.9812438Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] (out1.sum() + out2.sum()).backward() 2023-11-14T06:07:13.9813804Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 506, in backward 2023-11-14T06:07:13.9814594Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] torch.autograd.backward( 2023-11-14T06:07:13.9815907Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 266, in backward 2023-11-14T06:07:13.9817117Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-11-14T06:07:13.9818800Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] RuntimeError: [/var/lib/jenkins/workspace/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 5000ms for recv operation to complete 2023-11-14T06:07:13.9819343Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:07:13.9820334Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:07:13.9821402Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_ddp_has_finalized 2023-11-14T06:07:13.9821975Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:07:13.9823025Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:07:13.9823818Z [rank1]:[2023-11-14 05:53:29,388] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 2023-11-14T06:07:13.9824121Z ('RERUN', {'yellow': True}) [10.8513s] [100%] 2023-11-14T06:07:13.9829289Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py [rank0]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:13.9832722Z [rank1]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:13.9833417Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:07:13.9834230Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:07:13.9836234Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:07:13.9837032Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:07:13.9838648Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:07:13.9839281Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:07:13.9840718Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:07:13.9841452Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:07:13.9842930Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:07:13.9843686Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:07:13.9845363Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9738, in test_ddp_has_finalized 2023-11-14T06:07:13.9846165Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] (out1.sum() + out2.sum()).backward() 2023-11-14T06:07:13.9847412Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 506, in backward 2023-11-14T06:07:13.9848174Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] torch.autograd.backward( 2023-11-14T06:07:13.9849495Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 266, in backward 2023-11-14T06:07:13.9850690Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-11-14T06:07:13.9852368Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] RuntimeError: [/var/lib/jenkins/workspace/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 5000ms for recv operation to complete 2023-11-14T06:07:13.9853051Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:07:13.9854043Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:07:13.9855116Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_ddp_has_finalized 2023-11-14T06:07:13.9855682Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:07:13.9856727Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:07:13.9857519Z [rank1]:[2023-11-14 05:53:40,046] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 2023-11-14T06:07:13.9857838Z ('RERUN', {'yellow': True}) [10.7242s] [100%] 2023-11-14T06:07:13.9863088Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py [rank0]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:13.9866422Z [rank1]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:13.9867128Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:07:13.9867938Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:07:13.9869464Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:07:13.9870242Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:07:13.9871684Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:07:13.9872273Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:07:13.9873716Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:07:13.9874442Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:07:13.9876228Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:07:13.9877143Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:07:13.9878815Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9738, in test_ddp_has_finalized 2023-11-14T06:07:13.9879650Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] (out1.sum() + out2.sum()).backward() 2023-11-14T06:07:13.9880866Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 506, in backward 2023-11-14T06:07:13.9881632Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] torch.autograd.backward( 2023-11-14T06:07:13.9882962Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 266, in backward 2023-11-14T06:07:13.9884159Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-11-14T06:07:13.9885950Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] RuntimeError: [/var/lib/jenkins/workspace/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 5000ms for recv operation to complete 2023-11-14T06:07:13.9886514Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:07:13.9887503Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:07:13.9888584Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_ddp_has_finalized 2023-11-14T06:07:13.9889146Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:07:13.9890190Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:07:13.9890989Z [rank1]:[2023-11-14 05:53:50,778] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 2023-11-14T06:07:13.9891179Z FAILED [10.7230s] [100%] 2023-11-14T06:07:13.9891202Z 2023-11-14T06:07:13.9891478Z ==================================== RERUNS ==================================== 2023-11-14T06:07:13.9891891Z _______________ TestDistBackendWithSpawn.test_ddp_has_finalized ________________ 2023-11-14T06:07:13.9892078Z Traceback (most recent call last): 2023-11-14T06:07:13.9893004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 542, in wrapper 2023-11-14T06:07:13.9893199Z self._join_processes(fn) 2023-11-14T06:07:13.9894162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 759, in _join_processes 2023-11-14T06:07:13.9894380Z self._check_no_test_errors(elapsed_time) 2023-11-14T06:07:13.9895419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 773, in _check_no_test_errors 2023-11-14T06:07:13.9895586Z raise RuntimeError( 2023-11-14T06:07:13.9895955Z RuntimeError: Process 0 timed out after 10.518654108047485 seconds 2023-11-14T06:07:13.9896356Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:07:13.9896756Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:07:13.9897173Z ------------------------------ Captured log call ------------------------------- 2023-11-14T06:07:13.9897592Z INFO numba.cuda.cudadrv.driver:driver.py:245 init 2023-11-14T06:07:13.9898002Z _______________ TestDistBackendWithSpawn.test_ddp_has_finalized ________________ 2023-11-14T06:07:13.9898209Z Traceback (most recent call last): 2023-11-14T06:07:13.9899120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 542, in wrapper 2023-11-14T06:07:13.9899293Z self._join_processes(fn) 2023-11-14T06:07:13.9900291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 759, in _join_processes 2023-11-14T06:07:13.9900509Z self._check_no_test_errors(elapsed_time) 2023-11-14T06:07:13.9901537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 773, in _check_no_test_errors 2023-11-14T06:07:13.9901687Z raise RuntimeError( 2023-11-14T06:07:13.9902025Z RuntimeError: Process 0 timed out after 10.719219446182251 seconds 2023-11-14T06:07:13.9902423Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:07:13.9902821Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:07:13.9903206Z ------------------------------ Captured log call ------------------------------- 2023-11-14T06:07:13.9903501Z INFO numba.cuda.cudadrv.driver:driver.py:245 init 2023-11-14T06:07:13.9903898Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:07:13.9904424Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:07:13.9904695Z =================================== FAILURES =================================== 2023-11-14T06:07:13.9905116Z _______________ TestDistBackendWithSpawn.test_ddp_has_finalized ________________ 2023-11-14T06:07:13.9905312Z Traceback (most recent call last): 2023-11-14T06:07:13.9906228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 542, in wrapper 2023-11-14T06:07:13.9906409Z self._join_processes(fn) 2023-11-14T06:07:13.9907392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 759, in _join_processes 2023-11-14T06:07:13.9907620Z self._check_no_test_errors(elapsed_time) 2023-11-14T06:07:13.9908616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 773, in _check_no_test_errors 2023-11-14T06:07:13.9908811Z raise RuntimeError( 2023-11-14T06:07:13.9909153Z RuntimeError: Process 0 timed out after 10.71896243095398 seconds 2023-11-14T06:07:13.9909581Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:07:13.9909979Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:07:13.9910368Z ------------------------------ Captured log call ------------------------------- 2023-11-14T06:07:13.9910676Z INFO numba.cuda.cudadrv.driver:driver.py:245 init 2023-11-14T06:07:13.9911069Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:07:13.9911464Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:07:13.9911878Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:07:13.9912267Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:07:13.9913535Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-700392a528786283.xml - 2023-11-14T06:07:13.9913824Z =========================== short test summary info ============================ 2023-11-14T06:07:13.9915258Z FAILED [10.7230s] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized - RuntimeError: Process 0 timed out after 10.71896243095398 seconds 2023-11-14T06:07:13.9915749Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2023-11-14T06:07:13.9916039Z ========================= 1 failed, 2 rerun in 32.35s ========================== 2023-11-14T06:07:13.9916250Z Got exit code 1, retrying (retries left=1) 2023-11-14T06:07:13.9917315Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e91ba7ddccec34c8.xml 2023-11-14T06:07:13.9917592Z ============================= test session starts ============================== 2023-11-14T06:07:13.9918275Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:13.9918443Z cachedir: .pytest_cache 2023-11-14T06:07:13.9919290Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:13.9919502Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:13.9919673Z configfile: pytest.ini 2023-11-14T06:07:13.9920481Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:13.9920692Z collecting ... collected 1 item 2023-11-14T06:07:13.9921430Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2023-11-14T06:07:13.9921453Z 2023-11-14T06:07:13.9923337Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:13.9926653Z [rank1]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:13.9929960Z [rank0]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:13.9930654Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:07:13.9931459Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:07:13.9932932Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:07:13.9933673Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:07:13.9935150Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:07:13.9935749Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:07:13.9937177Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:07:13.9938026Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:07:13.9939471Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:07:13.9940242Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:07:13.9941892Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9738, in test_ddp_has_finalized 2023-11-14T06:07:13.9942717Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] (out1.sum() + out2.sum()).backward() 2023-11-14T06:07:13.9943957Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 506, in backward 2023-11-14T06:07:13.9944722Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] torch.autograd.backward( 2023-11-14T06:07:13.9946032Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 266, in backward 2023-11-14T06:07:13.9947296Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-11-14T06:07:13.9949025Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] RuntimeError: [/var/lib/jenkins/workspace/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 5000ms for recv operation to complete 2023-11-14T06:07:13.9949573Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:07:13.9950557Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:07:13.9951629Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_ddp_has_finalized 2023-11-14T06:07:13.9952171Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:07:13.9953243Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:07:13.9954035Z [rank1]:[2023-11-14 05:54:07,405] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 2023-11-14T06:07:13.9954355Z ('RERUN', {'yellow': True}) [11.0515s] [100%] 2023-11-14T06:07:13.9959769Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py [rank0]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:13.9963084Z [rank1]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:13.9963924Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:07:13.9964730Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:07:13.9966196Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:07:13.9966932Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:07:13.9968414Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:07:13.9969004Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:07:13.9970675Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:07:13.9971429Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:07:13.9972903Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:07:13.9973663Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:07:13.9975306Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9738, in test_ddp_has_finalized 2023-11-14T06:07:13.9976127Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] (out1.sum() + out2.sum()).backward() 2023-11-14T06:07:13.9977364Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 506, in backward 2023-11-14T06:07:13.9978129Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] torch.autograd.backward( 2023-11-14T06:07:13.9979436Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 266, in backward 2023-11-14T06:07:13.9980617Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-11-14T06:07:13.9982324Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] RuntimeError: [/var/lib/jenkins/workspace/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 5000ms for recv operation to complete 2023-11-14T06:07:13.9982866Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:07:13.9983852Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:07:13.9984918Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_ddp_has_finalized 2023-11-14T06:07:13.9985574Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:07:13.9986650Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:07:13.9987442Z [rank1]:[2023-11-14 05:54:18,059] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 2023-11-14T06:07:13.9987750Z ('RERUN', {'yellow': True}) [10.6226s] [100%] 2023-11-14T06:07:13.9992909Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py [rank0]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:13.9996555Z [rank1]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:13.9997276Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] Caught exception: 2023-11-14T06:07:13.9998089Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] Traceback (most recent call last): 2023-11-14T06:07:13.9999552Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test 2023-11-14T06:07:14.0000318Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] getattr(self, test_name)() 2023-11-14T06:07:14.0001779Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 544, in wrapper 2023-11-14T06:07:14.0002376Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] fn() 2023-11-14T06:07:14.0003813Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2536, in wrapper 2023-11-14T06:07:14.0004540Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] method(*args, **kwargs) 2023-11-14T06:07:14.0006018Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 2023-11-14T06:07:14.0006778Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] return func(*args, **kwargs) 2023-11-14T06:07:14.0008427Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 9738, in test_ddp_has_finalized 2023-11-14T06:07:14.0009250Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] (out1.sum() + out2.sum()).backward() 2023-11-14T06:07:14.0010630Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 506, in backward 2023-11-14T06:07:14.0011400Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] torch.autograd.backward( 2023-11-14T06:07:14.0012718Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 266, in backward 2023-11-14T06:07:14.0013894Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-11-14T06:07:14.0015597Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] RuntimeError: [/var/lib/jenkins/workspace/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 5000ms for recv operation to complete 2023-11-14T06:07:14.0016155Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:07:14.0017139Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] To execute this test, run the following from the base repo dir: 2023-11-14T06:07:14.0018297Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] python test/distributed/test_distributed_spawn.py -k test_ddp_has_finalized 2023-11-14T06:07:14.0018855Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] 2023-11-14T06:07:14.0019926Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2023-11-14T06:07:14.0020717Z [rank1]:[2023-11-14 05:54:28,747] torch.testing._internal.common_distributed: [ERROR] exiting process 1 with exit code: 10 2023-11-14T06:07:14.0020912Z FAILED [10.7227s] [100%] 2023-11-14T06:07:14.0020935Z 2023-11-14T06:07:14.0021190Z ==================================== RERUNS ==================================== 2023-11-14T06:07:14.0021597Z _______________ TestDistBackendWithSpawn.test_ddp_has_finalized ________________ 2023-11-14T06:07:14.0021808Z Traceback (most recent call last): 2023-11-14T06:07:14.0022726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 542, in wrapper 2023-11-14T06:07:14.0022928Z self._join_processes(fn) 2023-11-14T06:07:14.0023891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 759, in _join_processes 2023-11-14T06:07:14.0024105Z self._check_no_test_errors(elapsed_time) 2023-11-14T06:07:14.0025136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 773, in _check_no_test_errors 2023-11-14T06:07:14.0025306Z raise RuntimeError( 2023-11-14T06:07:14.0025651Z RuntimeError: Process 0 timed out after 10.718669414520264 seconds 2023-11-14T06:07:14.0026073Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:07:14.0026475Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:07:14.0026867Z ------------------------------ Captured log call ------------------------------- 2023-11-14T06:07:14.0027170Z INFO numba.cuda.cudadrv.driver:driver.py:245 init 2023-11-14T06:07:14.0027578Z _______________ TestDistBackendWithSpawn.test_ddp_has_finalized ________________ 2023-11-14T06:07:14.0027792Z Traceback (most recent call last): 2023-11-14T06:07:14.0028697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 542, in wrapper 2023-11-14T06:07:14.0028870Z self._join_processes(fn) 2023-11-14T06:07:14.0029977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 759, in _join_processes 2023-11-14T06:07:14.0030189Z self._check_no_test_errors(elapsed_time) 2023-11-14T06:07:14.0031194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 773, in _check_no_test_errors 2023-11-14T06:07:14.0031381Z raise RuntimeError( 2023-11-14T06:07:14.0031723Z RuntimeError: Process 0 timed out after 10.618561029434204 seconds 2023-11-14T06:07:14.0032153Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:07:14.0032561Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:07:14.0032953Z ------------------------------ Captured log call ------------------------------- 2023-11-14T06:07:14.0033256Z INFO numba.cuda.cudadrv.driver:driver.py:245 init 2023-11-14T06:07:14.0033652Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:07:14.0034061Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:07:14.0034343Z =================================== FAILURES =================================== 2023-11-14T06:07:14.0034927Z _______________ TestDistBackendWithSpawn.test_ddp_has_finalized ________________ 2023-11-14T06:07:14.0035122Z Traceback (most recent call last): 2023-11-14T06:07:14.0036064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 542, in wrapper 2023-11-14T06:07:14.0036362Z self._join_processes(fn) 2023-11-14T06:07:14.0037381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 759, in _join_processes 2023-11-14T06:07:14.0037593Z self._check_no_test_errors(elapsed_time) 2023-11-14T06:07:14.0038596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 773, in _check_no_test_errors 2023-11-14T06:07:14.0038798Z raise RuntimeError( 2023-11-14T06:07:14.0039143Z RuntimeError: Process 0 timed out after 10.718602418899536 seconds 2023-11-14T06:07:14.0039541Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:07:14.0039968Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:07:14.0040359Z ------------------------------ Captured log call ------------------------------- 2023-11-14T06:07:14.0040664Z INFO numba.cuda.cudadrv.driver:driver.py:245 init 2023-11-14T06:07:14.0041068Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:07:14.0041469Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:07:14.0041878Z ----------------------------- Captured stdout call ----------------------------- 2023-11-14T06:07:14.0042271Z Process 1 terminated with exit code 10, terminating remaining processes. 2023-11-14T06:07:14.0043531Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e91ba7ddccec34c8.xml - 2023-11-14T06:07:14.0043836Z =========================== short test summary info ============================ 2023-11-14T06:07:14.0045017Z FAILED [10.7227s] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized - RuntimeError: Process 0 timed out after 10.718602418899536 seconds 2023-11-14T06:07:14.0045335Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2023-11-14T06:07:14.0045625Z ========================= 1 failed, 2 rerun in 32.44s ========================== 2023-11-14T06:07:14.0048544Z Test exited with non-zero exitcode 1. Command to reproduce: BACKEND=gloo WORLD_SIZE=2 /opt/conda/envs/py_3.10/bin/python distributed/test_distributed_spawn.py --import-disabled-tests --import-slow-tests --use-pytest --shard-id=1 --num-shards=2 -v -vv -rfEX -p no:xdist -x --reruns=2 --pytest-single-test distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2023-11-14T06:07:14.0049749Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9613591347de16a.xml 2023-11-14T06:07:14.0050026Z ============================= test session starts ============================== 2023-11-14T06:07:14.0050702Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0050866Z cachedir: .pytest_cache 2023-11-14T06:07:14.0051723Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0051937Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0052099Z configfile: pytest.ini 2023-11-14T06:07:14.0052905Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0053119Z collecting ... collected 1 item 2023-11-14T06:07:14.0053906Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2023-11-14T06:07:14.0053928Z 2023-11-14T06:07:14.0055723Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0055886Z PASSED [6.5460s] [100%] 2023-11-14T06:07:14.0055907Z 2023-11-14T06:07:14.0057255Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9613591347de16a.xml - 2023-11-14T06:07:14.0057577Z ============================== 1 passed in 6.58s =============================== 2023-11-14T06:07:14.0058606Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-52defb573393a1f1.xml 2023-11-14T06:07:14.0058923Z ============================= test session starts ============================== 2023-11-14T06:07:14.0059574Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0059741Z cachedir: .pytest_cache 2023-11-14T06:07:14.0060612Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0060799Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0060968Z configfile: pytest.ini 2023-11-14T06:07:14.0061799Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0061979Z collecting ... collected 1 item 2023-11-14T06:07:14.0062791Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2023-11-14T06:07:14.0062818Z 2023-11-14T06:07:14.0064643Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0064809Z PASSED [6.8450s] [100%] 2023-11-14T06:07:14.0064830Z 2023-11-14T06:07:14.0066105Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-52defb573393a1f1.xml - 2023-11-14T06:07:14.0066389Z ============================== 1 passed in 6.88s =============================== 2023-11-14T06:07:14.0067434Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d3d5430b0a950244.xml 2023-11-14T06:07:14.0067712Z ============================= test session starts ============================== 2023-11-14T06:07:14.0068365Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0068676Z cachedir: .pytest_cache 2023-11-14T06:07:14.0069584Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0069768Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0069958Z configfile: pytest.ini 2023-11-14T06:07:14.0070764Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0070964Z collecting ... collected 1 item 2023-11-14T06:07:14.0071778Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2023-11-14T06:07:14.0071800Z 2023-11-14T06:07:14.0073581Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0073782Z PASSED [6.3433s] [100%] 2023-11-14T06:07:14.0073803Z 2023-11-14T06:07:14.0075276Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d3d5430b0a950244.xml - 2023-11-14T06:07:14.0075586Z ============================== 1 passed in 6.38s =============================== 2023-11-14T06:07:14.0076609Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9ceeb35cb9e28942.xml 2023-11-14T06:07:14.0077016Z ============================= test session starts ============================== 2023-11-14T06:07:14.0077723Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0077893Z cachedir: .pytest_cache 2023-11-14T06:07:14.0078739Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0078966Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0079130Z configfile: pytest.ini 2023-11-14T06:07:14.0079963Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0080140Z collecting ... collected 1 item 2023-11-14T06:07:14.0080907Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg 2023-11-14T06:07:14.0080928Z 2023-11-14T06:07:14.0082698Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0085997Z [rank1]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:14.0089300Z [rank0]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:14.0092132Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:2217: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-11-14T06:07:14.0092464Z warnings.warn( 2023-11-14T06:07:14.0095305Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:2217: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-11-14T06:07:14.0095487Z warnings.warn( 2023-11-14T06:07:14.0095645Z PASSED [6.2483s] [100%] 2023-11-14T06:07:14.0095665Z 2023-11-14T06:07:14.0096948Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9ceeb35cb9e28942.xml - 2023-11-14T06:07:14.0097239Z ============================== 1 passed in 6.29s =============================== 2023-11-14T06:07:14.0098264Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9cca654cac4f9f35.xml 2023-11-14T06:07:14.0098565Z ============================= test session starts ============================== 2023-11-14T06:07:14.0099314Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0099494Z cachedir: .pytest_cache 2023-11-14T06:07:14.0100378Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0100566Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0100755Z configfile: pytest.ini 2023-11-14T06:07:14.0101582Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0101752Z collecting ... collected 1 item 2023-11-14T06:07:14.0102646Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2023-11-14T06:07:14.0102668Z 2023-11-14T06:07:14.0104517Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0104682Z PASSED [6.1570s] [100%] 2023-11-14T06:07:14.0104728Z 2023-11-14T06:07:14.0105982Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9cca654cac4f9f35.xml - 2023-11-14T06:07:14.0106258Z ============================== 1 passed in 6.20s =============================== 2023-11-14T06:07:14.0107310Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c8cb5e62554ad8e.xml 2023-11-14T06:07:14.0107588Z ============================= test session starts ============================== 2023-11-14T06:07:14.0108241Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0108431Z cachedir: .pytest_cache 2023-11-14T06:07:14.0109295Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0109511Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0109672Z configfile: pytest.ini 2023-11-14T06:07:14.0110474Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0110673Z collecting ... collected 1 item 2023-11-14T06:07:14.0111546Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2023-11-14T06:07:14.0111665Z 2023-11-14T06:07:14.0113538Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0113732Z PASSED [6.1372s] [100%] 2023-11-14T06:07:14.0113752Z 2023-11-14T06:07:14.0115260Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c8cb5e62554ad8e.xml - 2023-11-14T06:07:14.0115571Z ============================== 1 passed in 6.17s =============================== 2023-11-14T06:07:14.0116601Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc19f69085cd5d60.xml 2023-11-14T06:07:14.0116893Z ============================= test session starts ============================== 2023-11-14T06:07:14.0117571Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0117736Z cachedir: .pytest_cache 2023-11-14T06:07:14.0118607Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0118791Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0118954Z configfile: pytest.ini 2023-11-14T06:07:14.0119932Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0120124Z collecting ... collected 1 item 2023-11-14T06:07:14.0120852Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2023-11-14T06:07:14.0120873Z 2023-11-14T06:07:14.0122602Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0122773Z PASSED [6.1362s] [100%] 2023-11-14T06:07:14.0122793Z 2023-11-14T06:07:14.0124072Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc19f69085cd5d60.xml - 2023-11-14T06:07:14.0124345Z ============================== 1 passed in 6.17s =============================== 2023-11-14T06:07:14.0125373Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-58fefe7a75eb8d83.xml 2023-11-14T06:07:14.0125680Z ============================= test session starts ============================== 2023-11-14T06:07:14.0126331Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0126531Z cachedir: .pytest_cache 2023-11-14T06:07:14.0127381Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0127566Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0127749Z configfile: pytest.ini 2023-11-14T06:07:14.0128554Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0128730Z collecting ... collected 1 item 2023-11-14T06:07:14.0129769Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2023-11-14T06:07:14.0129791Z 2023-11-14T06:07:14.0131792Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0133357Z [rank0]:[2023-11-14 05:56:03,094] torch.nn.parallel.distributed: [WARNING] Received mixed precision config _MixedPrecision(param_dtype=torch.float16, reduce_dtype=torch.float16, buffer_dtype=torch.float16) 2023-11-14T06:07:14.0134745Z [rank1]:[2023-11-14 05:56:03,191] torch.nn.parallel.distributed: [WARNING] Received mixed precision config _MixedPrecision(param_dtype=torch.float16, reduce_dtype=torch.float16, buffer_dtype=torch.float16) 2023-11-14T06:07:14.0134930Z PASSED [6.1454s] [100%] 2023-11-14T06:07:14.0134950Z 2023-11-14T06:07:14.0136202Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-58fefe7a75eb8d83.xml - 2023-11-14T06:07:14.0136478Z ============================== 1 passed in 6.18s =============================== 2023-11-14T06:07:14.0137518Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b23db701a815e67f.xml 2023-11-14T06:07:14.0137808Z ============================= test session starts ============================== 2023-11-14T06:07:14.0138487Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0138650Z cachedir: .pytest_cache 2023-11-14T06:07:14.0139496Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0139704Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0140027Z configfile: pytest.ini 2023-11-14T06:07:14.0140851Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0141054Z collecting ... collected 1 item 2023-11-14T06:07:14.0141938Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params 2023-11-14T06:07:14.0141966Z 2023-11-14T06:07:14.0143859Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0145251Z [rank1]:[2023-11-14 05:56:14,902] torch.nn.parallel.distributed: [WARNING] Received mixed precision config _MixedPrecision(param_dtype=torch.float16, reduce_dtype=torch.float16, buffer_dtype=torch.float16) 2023-11-14T06:07:14.0146639Z [rank0]:[2023-11-14 05:56:15,002] torch.nn.parallel.distributed: [WARNING] Received mixed precision config _MixedPrecision(param_dtype=torch.float16, reduce_dtype=torch.float16, buffer_dtype=torch.float16) 2023-11-14T06:07:14.0146826Z PASSED [6.1503s] [100%] 2023-11-14T06:07:14.0146847Z 2023-11-14T06:07:14.0148086Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b23db701a815e67f.xml - 2023-11-14T06:07:14.0148393Z ============================== 1 passed in 6.19s =============================== 2023-11-14T06:07:14.0149407Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6052f85bf4f36bd9.xml 2023-11-14T06:07:14.0149687Z ============================= test session starts ============================== 2023-11-14T06:07:14.0150367Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0150539Z cachedir: .pytest_cache 2023-11-14T06:07:14.0151388Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0151598Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0151761Z configfile: pytest.ini 2023-11-14T06:07:14.0152587Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0152883Z collecting ... collected 1 item 2023-11-14T06:07:14.0153730Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2023-11-14T06:07:14.0153751Z 2023-11-14T06:07:14.0155894Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0158752Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:2217: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-11-14T06:07:14.0158947Z warnings.warn( 2023-11-14T06:07:14.0161792Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:2217: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-11-14T06:07:14.0161973Z warnings.warn( 2023-11-14T06:07:14.0162133Z PASSED [6.1384s] [100%] 2023-11-14T06:07:14.0162154Z 2023-11-14T06:07:14.0163532Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6052f85bf4f36bd9.xml - 2023-11-14T06:07:14.0163858Z ============================== 1 passed in 6.18s =============================== 2023-11-14T06:07:14.0164893Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dfef39b05cf903c6.xml 2023-11-14T06:07:14.0165211Z ============================= test session starts ============================== 2023-11-14T06:07:14.0165869Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0166035Z cachedir: .pytest_cache 2023-11-14T06:07:14.0166913Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0167097Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0167264Z configfile: pytest.ini 2023-11-14T06:07:14.0168094Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0168274Z collecting ... collected 1 item 2023-11-14T06:07:14.0169104Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2023-11-14T06:07:14.0169155Z 2023-11-14T06:07:14.0171008Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0173267Z SKIPPED [0.3248s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [100%] 2023-11-14T06:07:14.0173316Z 2023-11-14T06:07:14.0174569Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dfef39b05cf903c6.xml - 2023-11-14T06:07:14.0174846Z ============================== 1 skipped in 0.36s ============================== 2023-11-14T06:07:14.0175884Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-392bb019e04a525b.xml 2023-11-14T06:07:14.0176304Z ============================= test session starts ============================== 2023-11-14T06:07:14.0176962Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0177157Z cachedir: .pytest_cache 2023-11-14T06:07:14.0178006Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0178222Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0178388Z configfile: pytest.ini 2023-11-14T06:07:14.0179193Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0179393Z collecting ... collected 1 item 2023-11-14T06:07:14.0180175Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged 2023-11-14T06:07:14.0180204Z 2023-11-14T06:07:14.0181961Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0182149Z PASSED [6.0329s] [100%] 2023-11-14T06:07:14.0182169Z 2023-11-14T06:07:14.0183504Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-392bb019e04a525b.xml - 2023-11-14T06:07:14.0183820Z ============================== 1 passed in 6.07s =============================== 2023-11-14T06:07:14.0184845Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a92d5866ed468410.xml 2023-11-14T06:07:14.0185120Z ============================= test session starts ============================== 2023-11-14T06:07:14.0185811Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0185974Z cachedir: .pytest_cache 2023-11-14T06:07:14.0186844Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0187028Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0187190Z configfile: pytest.ini 2023-11-14T06:07:14.0188014Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0188199Z collecting ... collected 1 item 2023-11-14T06:07:14.0188989Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2023-11-14T06:07:14.0189011Z 2023-11-14T06:07:14.0190827Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0190978Z PASSED [6.1353s] [100%] 2023-11-14T06:07:14.0190997Z 2023-11-14T06:07:14.0192265Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a92d5866ed468410.xml - 2023-11-14T06:07:14.0192542Z ============================== 1 passed in 6.17s =============================== 2023-11-14T06:07:14.0193566Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89c8eb122c442fd2.xml 2023-11-14T06:07:14.0193870Z ============================= test session starts ============================== 2023-11-14T06:07:14.0194527Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0194969Z cachedir: .pytest_cache 2023-11-14T06:07:14.0195833Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0196184Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0196370Z configfile: pytest.ini 2023-11-14T06:07:14.0197179Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0197356Z collecting ... collected 1 item 2023-11-14T06:07:14.0198208Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2023-11-14T06:07:14.0198236Z 2023-11-14T06:07:14.0200038Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0202880Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:2217: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-11-14T06:07:14.0203048Z warnings.warn( 2023-11-14T06:07:14.0206005Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:2217: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2023-11-14T06:07:14.0206180Z warnings.warn( 2023-11-14T06:07:14.0209464Z [rank1]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:14.0212769Z [rank0]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2023-11-14T06:07:14.0212963Z PASSED [6.0488s] [100%] 2023-11-14T06:07:14.0212989Z 2023-11-14T06:07:14.0214249Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89c8eb122c442fd2.xml - 2023-11-14T06:07:14.0214523Z ============================== 1 passed in 6.09s =============================== 2023-11-14T06:07:14.0215570Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a29961c4787a509.xml 2023-11-14T06:07:14.0215858Z ============================= test session starts ============================== 2023-11-14T06:07:14.0216516Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0216702Z cachedir: .pytest_cache 2023-11-14T06:07:14.0217556Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0217764Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0218039Z configfile: pytest.ini 2023-11-14T06:07:14.0218846Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0219045Z collecting ... collected 1 item 2023-11-14T06:07:14.0219893Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2023-11-14T06:07:14.0219910Z 2023-11-14T06:07:14.0221708Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0221876Z PASSED [6.1464s] [100%] 2023-11-14T06:07:14.0221896Z 2023-11-14T06:07:14.0223137Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a29961c4787a509.xml - 2023-11-14T06:07:14.0223443Z ============================== 1 passed in 6.18s =============================== 2023-11-14T06:07:14.0224459Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2284649f6ce1fd6b.xml 2023-11-14T06:07:14.0224765Z ============================= test session starts ============================== 2023-11-14T06:07:14.0225413Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0225577Z cachedir: .pytest_cache 2023-11-14T06:07:14.0226543Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0226747Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0226908Z configfile: pytest.ini 2023-11-14T06:07:14.0227738Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0227927Z collecting ... collected 1 item 2023-11-14T06:07:14.0228700Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states 2023-11-14T06:07:14.0228747Z 2023-11-14T06:07:14.0230497Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0230657Z PASSED [6.1383s] [100%] 2023-11-14T06:07:14.0230677Z 2023-11-14T06:07:14.0231956Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2284649f6ce1fd6b.xml - 2023-11-14T06:07:14.0232234Z ============================== 1 passed in 6.18s =============================== 2023-11-14T06:07:14.0233273Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0216a5c1e9673a99.xml 2023-11-14T06:07:14.0233563Z ============================= test session starts ============================== 2023-11-14T06:07:14.0234212Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0234410Z cachedir: .pytest_cache 2023-11-14T06:07:14.0235510Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0235696Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0235882Z configfile: pytest.ini 2023-11-14T06:07:14.0236693Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0236878Z collecting ... collected 1 item 2023-11-14T06:07:14.0237716Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable 2023-11-14T06:07:14.0237871Z 2023-11-14T06:07:14.0239684Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0239871Z PASSED [6.1357s] [100%] 2023-11-14T06:07:14.0239890Z 2023-11-14T06:07:14.0241136Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0216a5c1e9673a99.xml - 2023-11-14T06:07:14.0241443Z ============================== 1 passed in 6.17s =============================== 2023-11-14T06:07:14.0242463Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e690c97262b9f1a.xml 2023-11-14T06:07:14.0242742Z ============================= test session starts ============================== 2023-11-14T06:07:14.0243410Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0243584Z cachedir: .pytest_cache 2023-11-14T06:07:14.0244435Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0244643Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0244806Z configfile: pytest.ini 2023-11-14T06:07:14.0245605Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0245813Z collecting ... collected 1 item 2023-11-14T06:07:14.0246801Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn 2023-11-14T06:07:14.0246827Z 2023-11-14T06:07:14.0248724Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0254605Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py:266: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) 2023-11-14T06:07:14.0255172Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-11-14T06:07:14.0261010Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py:266: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) 2023-11-14T06:07:14.0261568Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2023-11-14T06:07:14.0262677Z /opt/conda/envs/py_3.10/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up 2023-11-14T06:07:14.0262941Z _warnings.warn(warn_message, ResourceWarning) 2023-11-14T06:07:14.0263098Z PASSED [6.2345s] [100%] 2023-11-14T06:07:14.0263119Z 2023-11-14T06:07:14.0264355Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e690c97262b9f1a.xml - 2023-11-14T06:07:14.0264660Z ============================== 1 passed in 6.27s =============================== 2023-11-14T06:07:14.0265694Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d1cd7d2afbcc0beb.xml 2023-11-14T06:07:14.0265992Z ============================= test session starts ============================== 2023-11-14T06:07:14.0266645Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0266817Z cachedir: .pytest_cache 2023-11-14T06:07:14.0267689Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0267873Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0268032Z configfile: pytest.ini 2023-11-14T06:07:14.0268857Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0269037Z collecting ... collected 1 item 2023-11-14T06:07:14.0269936Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2023-11-14T06:07:14.0269960Z 2023-11-14T06:07:14.0271679Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0271850Z PASSED [5.9509s] [100%] 2023-11-14T06:07:14.0271870Z 2023-11-14T06:07:14.0273154Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d1cd7d2afbcc0beb.xml - 2023-11-14T06:07:14.0273427Z ============================== 1 passed in 5.99s =============================== 2023-11-14T06:07:14.0274466Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7d11243343d2612e.xml 2023-11-14T06:07:14.0274948Z ============================= test session starts ============================== 2023-11-14T06:07:14.0275614Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0275802Z cachedir: .pytest_cache 2023-11-14T06:07:14.0276648Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0276869Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0277032Z configfile: pytest.ini 2023-11-14T06:07:14.0277839Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0278041Z collecting ... collected 1 item 2023-11-14T06:07:14.0278707Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2023-11-14T06:07:14.0278729Z 2023-11-14T06:07:14.0280381Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0281062Z STAGE:2023-11-14 05:58:16 6648:6648 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0281702Z STAGE:2023-11-14 05:58:16 6649:6649 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0282388Z STAGE:2023-11-14 05:58:16 6649:6649 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0283245Z STAGE:2023-11-14 05:58:16 6649:6649 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0283905Z STAGE:2023-11-14 05:58:16 6648:6648 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0284623Z STAGE:2023-11-14 05:58:16 6648:6648 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0285263Z STAGE:2023-11-14 05:58:16 6649:6649 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0285910Z STAGE:2023-11-14 05:58:16 6648:6648 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0286600Z STAGE:2023-11-14 05:58:16 6648:6648 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0287298Z STAGE:2023-11-14 05:58:16 6648:6648 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0287992Z STAGE:2023-11-14 05:58:16 6649:6649 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0288685Z STAGE:2023-11-14 05:58:16 6649:6649 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0288844Z PASSED [6.0343s] [100%] 2023-11-14T06:07:14.0288864Z 2023-11-14T06:07:14.0290128Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7d11243343d2612e.xml - 2023-11-14T06:07:14.0290517Z ============================== 1 passed in 6.07s =============================== 2023-11-14T06:07:14.0291589Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b13d84c867fe99ab.xml 2023-11-14T06:07:14.0291867Z ============================= test session starts ============================== 2023-11-14T06:07:14.0292518Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0292714Z cachedir: .pytest_cache 2023-11-14T06:07:14.0293567Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0293752Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0293938Z configfile: pytest.ini 2023-11-14T06:07:14.0294744Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0294961Z collecting ... collected 1 item 2023-11-14T06:07:14.0295666Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2023-11-14T06:07:14.0295689Z 2023-11-14T06:07:14.0297511Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Only Nccl supports CUDA gather) [100%] 2023-11-14T06:07:14.0297541Z 2023-11-14T06:07:14.0298810Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b13d84c867fe99ab.xml - 2023-11-14T06:07:14.0299087Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:14.0300129Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5da298f07d8641f6.xml 2023-11-14T06:07:14.0300414Z ============================= test session starts ============================== 2023-11-14T06:07:14.0301071Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0301257Z cachedir: .pytest_cache 2023-11-14T06:07:14.0302109Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0302294Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0302643Z configfile: pytest.ini 2023-11-14T06:07:14.0303450Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0303654Z collecting ... collected 1 item 2023-11-14T06:07:14.0304394Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group 2023-11-14T06:07:14.0304416Z 2023-11-14T06:07:14.0306136Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0306818Z STAGE:2023-11-14 05:58:33 6673:6673 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0307458Z STAGE:2023-11-14 05:58:33 6674:6674 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0308145Z STAGE:2023-11-14 05:58:33 6674:6674 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0308855Z STAGE:2023-11-14 05:58:33 6674:6674 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0309522Z STAGE:2023-11-14 05:58:33 6673:6673 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0310248Z STAGE:2023-11-14 05:58:33 6673:6673 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0310880Z STAGE:2023-11-14 05:58:33 6674:6674 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0311640Z STAGE:2023-11-14 05:58:33 6673:6673 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0312328Z STAGE:2023-11-14 05:58:33 6673:6673 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0313024Z STAGE:2023-11-14 05:58:33 6673:6673 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0313721Z STAGE:2023-11-14 05:58:33 6674:6674 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0314416Z STAGE:2023-11-14 05:58:33 6674:6674 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0314808Z PASSED [6.0470s] [100%] 2023-11-14T06:07:14.0314830Z 2023-11-14T06:07:14.0316124Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5da298f07d8641f6.xml - 2023-11-14T06:07:14.0316407Z ============================== 1 passed in 6.09s =============================== 2023-11-14T06:07:14.0317457Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32ba4a8b8a071bce.xml 2023-11-14T06:07:14.0317736Z ============================= test session starts ============================== 2023-11-14T06:07:14.0318387Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0318583Z cachedir: .pytest_cache 2023-11-14T06:07:14.0319434Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0319625Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0319808Z configfile: pytest.ini 2023-11-14T06:07:14.0320610Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0320785Z collecting ... collected 1 item 2023-11-14T06:07:14.0321589Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2023-11-14T06:07:14.0321610Z 2023-11-14T06:07:14.0323362Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0323703Z PASSED [6.0344s] [100%] 2023-11-14T06:07:14.0323722Z 2023-11-14T06:07:14.0324982Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32ba4a8b8a071bce.xml - 2023-11-14T06:07:14.0325280Z ============================== 1 passed in 6.07s =============================== 2023-11-14T06:07:14.0326302Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70f6fcfa34947147.xml 2023-11-14T06:07:14.0326587Z ============================= test session starts ============================== 2023-11-14T06:07:14.0327264Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0327427Z cachedir: .pytest_cache 2023-11-14T06:07:14.0328281Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0328500Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0328661Z configfile: pytest.ini 2023-11-14T06:07:14.0329459Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0329665Z collecting ... collected 1 item 2023-11-14T06:07:14.0330446Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params 2023-11-14T06:07:14.0330468Z 2023-11-14T06:07:14.0332360Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0332541Z PASSED [6.0359s] [100%] 2023-11-14T06:07:14.0332561Z 2023-11-14T06:07:14.0333836Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70f6fcfa34947147.xml - 2023-11-14T06:07:14.0334124Z ============================== 1 passed in 6.07s =============================== 2023-11-14T06:07:14.0335146Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-695df6a87d1ca124.xml 2023-11-14T06:07:14.0335449Z ============================= test session starts ============================== 2023-11-14T06:07:14.0336098Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0336268Z cachedir: .pytest_cache 2023-11-14T06:07:14.0337150Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0337337Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0337526Z configfile: pytest.ini 2023-11-14T06:07:14.0338325Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0338513Z collecting ... collected 1 item 2023-11-14T06:07:14.0339226Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future 2023-11-14T06:07:14.0339247Z 2023-11-14T06:07:14.0340914Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0341071Z PASSED [5.9360s] [100%] 2023-11-14T06:07:14.0341118Z 2023-11-14T06:07:14.0342369Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-695df6a87d1ca124.xml - 2023-11-14T06:07:14.0342647Z ============================== 1 passed in 5.97s =============================== 2023-11-14T06:07:14.0343691Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1b7f35dfb2ff66f.xml 2023-11-14T06:07:14.0344102Z ============================= test session starts ============================== 2023-11-14T06:07:14.0344759Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0344953Z cachedir: .pytest_cache 2023-11-14T06:07:14.0345800Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0346014Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0346185Z configfile: pytest.ini 2023-11-14T06:07:14.0346989Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0347192Z collecting ... collected 1 item 2023-11-14T06:07:14.0347873Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank 2023-11-14T06:07:14.0347901Z 2023-11-14T06:07:14.0349562Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0349753Z PASSED [5.8422s] [100%] 2023-11-14T06:07:14.0349773Z 2023-11-14T06:07:14.0351017Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1b7f35dfb2ff66f.xml - 2023-11-14T06:07:14.0351406Z ============================== 1 passed in 5.88s =============================== 2023-11-14T06:07:14.0352446Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34bc5819fb526dcc.xml 2023-11-14T06:07:14.0352722Z ============================= test session starts ============================== 2023-11-14T06:07:14.0353393Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0353566Z cachedir: .pytest_cache 2023-11-14T06:07:14.0354445Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0354865Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0355040Z configfile: pytest.ini 2023-11-14T06:07:14.0355875Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0356052Z collecting ... collected 1 item 2023-11-14T06:07:14.0356842Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2023-11-14T06:07:14.0356864Z 2023-11-14T06:07:14.0358642Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0358813Z PASSED [5.9535s] [100%] 2023-11-14T06:07:14.0358833Z 2023-11-14T06:07:14.0360099Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34bc5819fb526dcc.xml - 2023-11-14T06:07:14.0360371Z ============================== 1 passed in 5.99s =============================== 2023-11-14T06:07:14.0361394Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e9170ff2b6efaeae.xml 2023-11-14T06:07:14.0361703Z ============================= test session starts ============================== 2023-11-14T06:07:14.0362360Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0362553Z cachedir: .pytest_cache 2023-11-14T06:07:14.0363405Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0363741Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0363928Z configfile: pytest.ini 2023-11-14T06:07:14.0364795Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0364976Z collecting ... collected 1 item 2023-11-14T06:07:14.0365755Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2023-11-14T06:07:14.0365776Z 2023-11-14T06:07:14.0367503Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0367691Z PASSED [6.0355s] [100%] 2023-11-14T06:07:14.0367710Z 2023-11-14T06:07:14.0368959Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e9170ff2b6efaeae.xml - 2023-11-14T06:07:14.0369242Z ============================== 1 passed in 6.07s =============================== 2023-11-14T06:07:14.0370327Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2bc13a1ae54b17e4.xml 2023-11-14T06:07:14.0370606Z ============================= test session starts ============================== 2023-11-14T06:07:14.0371287Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0371570Z cachedir: .pytest_cache 2023-11-14T06:07:14.0372446Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0372660Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0372820Z configfile: pytest.ini 2023-11-14T06:07:14.0373627Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0373839Z collecting ... collected 1 item 2023-11-14T06:07:14.0374501Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv 2023-11-14T06:07:14.0374522Z 2023-11-14T06:07:14.0376187Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0376348Z PASSED [6.0333s] [100%] 2023-11-14T06:07:14.0376375Z 2023-11-14T06:07:14.0377622Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2bc13a1ae54b17e4.xml - 2023-11-14T06:07:14.0377916Z ============================== 1 passed in 6.07s =============================== 2023-11-14T06:07:14.0378934Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4d54ca8c3597dd27.xml 2023-11-14T06:07:14.0379242Z ============================= test session starts ============================== 2023-11-14T06:07:14.0379894Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0380057Z cachedir: .pytest_cache 2023-11-14T06:07:14.0380926Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0381107Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0381275Z configfile: pytest.ini 2023-11-14T06:07:14.0382106Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0382283Z collecting ... collected 1 item 2023-11-14T06:07:14.0382950Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2023-11-14T06:07:14.0383098Z 2023-11-14T06:07:14.0384750Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0384915Z PASSED [6.0377s] [100%] 2023-11-14T06:07:14.0384934Z 2023-11-14T06:07:14.0386213Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4d54ca8c3597dd27.xml - 2023-11-14T06:07:14.0386495Z ============================== 1 passed in 6.07s =============================== 2023-11-14T06:07:14.0387544Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ea9e7533e641e6f.xml 2023-11-14T06:07:14.0387824Z ============================= test session starts ============================== 2023-11-14T06:07:14.0388476Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0388673Z cachedir: .pytest_cache 2023-11-14T06:07:14.0389520Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0389702Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0389887Z configfile: pytest.ini 2023-11-14T06:07:14.0390683Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0390860Z collecting ... collected 1 item 2023-11-14T06:07:14.0391766Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler 2023-11-14T06:07:14.0391789Z 2023-11-14T06:07:14.0393559Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0394251Z STAGE:2023-11-14 06:00:17 6883:6883 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0395086Z STAGE:2023-11-14 06:00:17 6884:6884 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0395759Z STAGE:2023-11-14 06:00:17 6884:6884 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0396490Z STAGE:2023-11-14 06:00:17 6884:6884 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0397159Z STAGE:2023-11-14 06:00:17 6883:6883 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0397881Z STAGE:2023-11-14 06:00:17 6883:6883 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0398040Z PASSED [6.0372s] [100%] 2023-11-14T06:07:14.0398061Z 2023-11-14T06:07:14.0399308Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ea9e7533e641e6f.xml - 2023-11-14T06:07:14.0399622Z ============================== 1 passed in 6.07s =============================== 2023-11-14T06:07:14.0400640Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c8e49f57f9738e72.xml 2023-11-14T06:07:14.0400943Z ============================= test session starts ============================== 2023-11-14T06:07:14.0401595Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0401770Z cachedir: .pytest_cache 2023-11-14T06:07:14.0402653Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0402842Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0403005Z configfile: pytest.ini 2023-11-14T06:07:14.0403827Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0404165Z collecting ... collected 1 item 2023-11-14T06:07:14.0405004Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2023-11-14T06:07:14.0405049Z 2023-11-14T06:07:14.0406859Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0408130Z [rank1]:[E ProcessGroupNCCL.cpp:754] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=2, OpType=ALLREDUCE, NumelIn=10, NumelOut=10, Timeout(ms)=15000) ran for 101 milliseconds before timing out. 2023-11-14T06:07:14.0409191Z [rank1]:[E ProcessGroupNCCL.cpp:820] [Rank 1] Work WorkNCCL(SeqNum=2, OpType=ALLREDUCE, NumelIn=10, NumelOut=10, Timeout(ms)=15000) timed out in blocking wait (NCCL_BLOCKING_WAIT=1). 2023-11-14T06:07:14.0409735Z [rank0]:[E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2023-11-14T06:07:14.0410935Z [rank1]:[E ProcessGroupNCCL.cpp:768] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. 2023-11-14T06:07:14.0411524Z [rank1]:[E ProcessGroupNCCL.cpp:774] To avoid data inconsistency, we are taking the entire process down. 2023-11-14T06:07:14.0412809Z [rank1]:[E ProcessGroupNCCL.cpp:768] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. 2023-11-14T06:07:14.0414017Z [rank1]:[E ProcessGroupNCCL.cpp:768] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. 2023-11-14T06:07:14.0414178Z PASSED [7.9450s] [100%] 2023-11-14T06:07:14.0414206Z 2023-11-14T06:07:14.0415485Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c8e49f57f9738e72.xml - 2023-11-14T06:07:14.0415760Z ============================== 1 passed in 7.98s =============================== 2023-11-14T06:07:14.0416779Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b4029a89c195b80c.xml 2023-11-14T06:07:14.0417092Z ============================= test session starts ============================== 2023-11-14T06:07:14.0417744Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0417933Z cachedir: .pytest_cache 2023-11-14T06:07:14.0418778Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0418964Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0419162Z configfile: pytest.ini 2023-11-14T06:07:14.0419971Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0420148Z collecting ... collected 1 item 2023-11-14T06:07:14.0421093Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2023-11-14T06:07:14.0421114Z 2023-11-14T06:07:14.0423021Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0424312Z [rank1]:[E ProcessGroupNCCL.cpp:754] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=2, OpType=ALLREDUCE, NumelIn=10, NumelOut=10, Timeout(ms)=15000) ran for 101 milliseconds before timing out. 2023-11-14T06:07:14.0424963Z [rank0]:[E ProcessGroupGloo.cpp:2950] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2023-11-14T06:07:14.0425513Z [rank0]:[E ProcessGroupGloo.cpp:138] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms 2023-11-14T06:07:14.0426540Z [rank1]:[E ProcessGroupNCCL.cpp:820] [Rank 1] Work WorkNCCL(SeqNum=2, OpType=ALLREDUCE, NumelIn=10, NumelOut=10, Timeout(ms)=15000) timed out in blocking wait (NCCL_BLOCKING_WAIT=1). 2023-11-14T06:07:14.0427718Z [rank1]:[E ProcessGroupNCCL.cpp:768] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. 2023-11-14T06:07:14.0428345Z [rank1]:[E ProcessGroupNCCL.cpp:774] To avoid data inconsistency, we are taking the entire process down. 2023-11-14T06:07:14.0429511Z [rank1]:[E ProcessGroupNCCL.cpp:768] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. 2023-11-14T06:07:14.0430699Z [rank1]:[E ProcessGroupNCCL.cpp:768] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. 2023-11-14T06:07:14.0430861Z PASSED [8.1403s] [100%] 2023-11-14T06:07:14.0430880Z 2023-11-14T06:07:14.0432129Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b4029a89c195b80c.xml - 2023-11-14T06:07:14.0432518Z ============================== 1 passed in 8.18s =============================== 2023-11-14T06:07:14.0433555Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dce97195ad54772d.xml 2023-11-14T06:07:14.0433858Z ============================= test session starts ============================== 2023-11-14T06:07:14.0434511Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0434928Z cachedir: .pytest_cache 2023-11-14T06:07:14.0435819Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0436004Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0436167Z configfile: pytest.ini 2023-11-14T06:07:14.0436991Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0437173Z collecting ... collected 1 item 2023-11-14T06:07:14.0437952Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo 2023-11-14T06:07:14.0437998Z 2023-11-14T06:07:14.0439754Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0440301Z [rank0]:[E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2023-11-14T06:07:14.0440487Z PASSED [8.0543s] [100%] 2023-11-14T06:07:14.0440506Z 2023-11-14T06:07:14.0441753Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dce97195ad54772d.xml - 2023-11-14T06:07:14.0442052Z ============================== 1 passed in 8.09s =============================== 2023-11-14T06:07:14.0443074Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-016c4a267886d17b.xml 2023-11-14T06:07:14.0443354Z ============================= test session starts ============================== 2023-11-14T06:07:14.0444027Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0444192Z cachedir: .pytest_cache 2023-11-14T06:07:14.0445198Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0445411Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0445577Z configfile: pytest.ini 2023-11-14T06:07:14.0446376Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0446580Z collecting ... collected 1 item 2023-11-14T06:07:14.0447415Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup 2023-11-14T06:07:14.0447437Z 2023-11-14T06:07:14.0449269Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0449797Z [rank0]:[E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2023-11-14T06:07:14.0449966Z PASSED [6.1543s] [100%] 2023-11-14T06:07:14.0450011Z 2023-11-14T06:07:14.0451256Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-016c4a267886d17b.xml - 2023-11-14T06:07:14.0451525Z ============================== 1 passed in 6.19s =============================== 2023-11-14T06:07:14.0452684Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3408f4e0db3157c1.xml 2023-11-14T06:07:14.0452986Z ============================= test session starts ============================== 2023-11-14T06:07:14.0453641Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0453827Z cachedir: .pytest_cache 2023-11-14T06:07:14.0454677Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0454894Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0455054Z configfile: pytest.ini 2023-11-14T06:07:14.0455858Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0456059Z collecting ... collected 1 item 2023-11-14T06:07:14.0456858Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce 2023-11-14T06:07:14.0456879Z 2023-11-14T06:07:14.0458926Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Test requires backend gloo to be one of {'nccl'}) [100%] 2023-11-14T06:07:14.0458976Z 2023-11-14T06:07:14.0460223Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3408f4e0db3157c1.xml - 2023-11-14T06:07:14.0460512Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:14.0461564Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6a68b2b4eecc0c38.xml 2023-11-14T06:07:14.0461840Z ============================= test session starts ============================== 2023-11-14T06:07:14.0462523Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0462691Z cachedir: .pytest_cache 2023-11-14T06:07:14.0463537Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0463752Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0463917Z configfile: pytest.ini 2023-11-14T06:07:14.0464716Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0465113Z collecting ... collected 1 item 2023-11-14T06:07:14.0465897Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2023-11-14T06:07:14.0465919Z 2023-11-14T06:07:14.0467978Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Test requires backend gloo to be one of {'nccl'}) [100%] 2023-11-14T06:07:14.0468001Z 2023-11-14T06:07:14.0469261Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6a68b2b4eecc0c38.xml - 2023-11-14T06:07:14.0469536Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:14.0470638Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-691c3f0b0bf33453.xml 2023-11-14T06:07:14.0470927Z ============================= test session starts ============================== 2023-11-14T06:07:14.0471600Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0471770Z cachedir: .pytest_cache 2023-11-14T06:07:14.0472616Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0472930Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0473110Z configfile: pytest.ini 2023-11-14T06:07:14.0473920Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0474122Z collecting ... collected 1 item 2023-11-14T06:07:14.0475109Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2023-11-14T06:07:14.0475141Z 2023-11-14T06:07:14.0477195Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only NCCL backend supports high priority stream) [100%] 2023-11-14T06:07:14.0477217Z 2023-11-14T06:07:14.0478469Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-691c3f0b0bf33453.xml - 2023-11-14T06:07:14.0478746Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:14.0479786Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3257467972824c7.xml 2023-11-14T06:07:14.0480063Z ============================= test session starts ============================== 2023-11-14T06:07:14.0480744Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0480906Z cachedir: .pytest_cache 2023-11-14T06:07:14.0481751Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0481962Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0482124Z configfile: pytest.ini 2023-11-14T06:07:14.0482926Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0483139Z collecting ... collected 1 item 2023-11-14T06:07:14.0483861Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort 2023-11-14T06:07:14.0483882Z 2023-11-14T06:07:14.0485896Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_init_abort <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (TORCH_NCCL_USE_COMM_NONBLOCKING only applies to NCCL) [100%] 2023-11-14T06:07:14.0486059Z 2023-11-14T06:07:14.0487323Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3257467972824c7.xml - 2023-11-14T06:07:14.0487622Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:14.0488638Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-75143866348aa99a.xml 2023-11-14T06:07:14.0488924Z ============================= test session starts ============================== 2023-11-14T06:07:14.0489601Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0489764Z cachedir: .pytest_cache 2023-11-14T06:07:14.0490608Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0490831Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0490991Z configfile: pytest.ini 2023-11-14T06:07:14.0491789Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0491997Z collecting ... collected 1 item 2023-11-14T06:07:14.0492988Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2023-11-14T06:07:14.0493128Z 2023-11-14T06:07:14.0495294Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Test requires world size of 4) [100%] 2023-11-14T06:07:14.0495316Z 2023-11-14T06:07:14.0496567Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-75143866348aa99a.xml - 2023-11-14T06:07:14.0496869Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:14.0497886Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6712135cd5b769b4.xml 2023-11-14T06:07:14.0498161Z ============================= test session starts ============================== 2023-11-14T06:07:14.0498843Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0499014Z cachedir: .pytest_cache 2023-11-14T06:07:14.0499862Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0500076Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0500238Z configfile: pytest.ini 2023-11-14T06:07:14.0501075Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0501256Z collecting ... collected 1 item 2023-11-14T06:07:14.0502094Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2023-11-14T06:07:14.0502115Z 2023-11-14T06:07:14.0504096Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Test requires world size of 4) [100%] 2023-11-14T06:07:14.0504119Z 2023-11-14T06:07:14.0505359Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6712135cd5b769b4.xml - 2023-11-14T06:07:14.0505655Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:14.0506790Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f4bea4e91a1a7d84.xml 2023-11-14T06:07:14.0507066Z ============================= test session starts ============================== 2023-11-14T06:07:14.0507740Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0507907Z cachedir: .pytest_cache 2023-11-14T06:07:14.0508781Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0508968Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0509130Z configfile: pytest.ini 2023-11-14T06:07:14.0509956Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0510132Z collecting ... collected 1 item 2023-11-14T06:07:14.0511087Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size 2023-11-14T06:07:14.0511108Z 2023-11-14T06:07:14.0513201Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Test requires world size of 4) [100%] 2023-11-14T06:07:14.0513222Z 2023-11-14T06:07:14.0514791Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f4bea4e91a1a7d84.xml - 2023-11-14T06:07:14.0515137Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:14.0516169Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-066b58d1b312a291.xml 2023-11-14T06:07:14.0516485Z ============================= test session starts ============================== 2023-11-14T06:07:14.0517136Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0517302Z cachedir: .pytest_cache 2023-11-14T06:07:14.0518181Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0518372Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0518530Z configfile: pytest.ini 2023-11-14T06:07:14.0519358Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0519542Z collecting ... collected 1 item 2023-11-14T06:07:14.0520385Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2023-11-14T06:07:14.0520435Z 2023-11-14T06:07:14.0522250Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0522426Z PASSED [6.2353s] [100%] 2023-11-14T06:07:14.0522445Z 2023-11-14T06:07:14.0523718Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-066b58d1b312a291.xml - 2023-11-14T06:07:14.0523996Z ============================== 1 passed in 6.27s =============================== 2023-11-14T06:07:14.0525045Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af55d825bec77c78.xml 2023-11-14T06:07:14.0525321Z ============================= test session starts ============================== 2023-11-14T06:07:14.0525972Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0526311Z cachedir: .pytest_cache 2023-11-14T06:07:14.0527169Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0527355Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0527535Z configfile: pytest.ini 2023-11-14T06:07:14.0528335Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0528513Z collecting ... collected 1 item 2023-11-14T06:07:14.0529322Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2023-11-14T06:07:14.0529343Z 2023-11-14T06:07:14.0531102Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0531305Z PASSED [6.1428s] [100%] 2023-11-14T06:07:14.0531324Z 2023-11-14T06:07:14.0532567Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af55d825bec77c78.xml - 2023-11-14T06:07:14.0532863Z ============================== 1 passed in 6.18s =============================== 2023-11-14T06:07:14.0533887Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d31389e48a81cae1.xml 2023-11-14T06:07:14.0534287Z ============================= test session starts ============================== 2023-11-14T06:07:14.0534989Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0535153Z cachedir: .pytest_cache 2023-11-14T06:07:14.0536033Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0536221Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0536381Z configfile: pytest.ini 2023-11-14T06:07:14.0537184Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0537385Z collecting ... collected 1 item 2023-11-14T06:07:14.0538216Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2023-11-14T06:07:14.0538238Z 2023-11-14T06:07:14.0540082Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0540229Z PASSED [6.2430s] [100%] 2023-11-14T06:07:14.0540248Z 2023-11-14T06:07:14.0541513Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d31389e48a81cae1.xml - 2023-11-14T06:07:14.0541797Z ============================== 1 passed in 6.28s =============================== 2023-11-14T06:07:14.0542817Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-79b628499dec5c4a.xml 2023-11-14T06:07:14.0543121Z ============================= test session starts ============================== 2023-11-14T06:07:14.0543770Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0543939Z cachedir: .pytest_cache 2023-11-14T06:07:14.0544826Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0545009Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0545195Z configfile: pytest.ini 2023-11-14T06:07:14.0545999Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0546296Z collecting ... collected 1 item 2023-11-14T06:07:14.0547285Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2023-11-14T06:07:14.0547306Z 2023-11-14T06:07:14.0549254Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0549567Z SKIPPED [5.8477s] (Need at least 4 CUDA devices) [100%] 2023-11-14T06:07:14.0549587Z 2023-11-14T06:07:14.0550832Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-79b628499dec5c4a.xml - 2023-11-14T06:07:14.0551108Z ============================== 1 skipped in 5.89s ============================== 2023-11-14T06:07:14.0552163Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-14e6c982cc5fbc72.xml 2023-11-14T06:07:14.0552438Z ============================= test session starts ============================== 2023-11-14T06:07:14.0553086Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0553275Z cachedir: .pytest_cache 2023-11-14T06:07:14.0554233Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0554461Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0554976Z configfile: pytest.ini 2023-11-14T06:07:14.0555815Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0556021Z collecting ... collected 1 item 2023-11-14T06:07:14.0557054Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2023-11-14T06:07:14.0557085Z 2023-11-14T06:07:14.0559130Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0559413Z SKIPPED [5.8468s] (Need at least 4 CUDA devices) [100%] 2023-11-14T06:07:14.0559441Z 2023-11-14T06:07:14.0560688Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-14e6c982cc5fbc72.xml - 2023-11-14T06:07:14.0560989Z ============================== 1 skipped in 5.89s ============================== 2023-11-14T06:07:14.0562008Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-054a7e4687e3a111.xml 2023-11-14T06:07:14.0562325Z ============================= test session starts ============================== 2023-11-14T06:07:14.0562977Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0563142Z cachedir: .pytest_cache 2023-11-14T06:07:14.0564018Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0564205Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0564373Z configfile: pytest.ini 2023-11-14T06:07:14.0565203Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0565384Z collecting ... collected 1 item 2023-11-14T06:07:14.0566233Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2023-11-14T06:07:14.0566425Z 2023-11-14T06:07:14.0568275Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0568443Z PASSED [6.3352s] [100%] 2023-11-14T06:07:14.0568463Z 2023-11-14T06:07:14.0569794Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-054a7e4687e3a111.xml - 2023-11-14T06:07:14.0570075Z ============================== 1 passed in 6.37s =============================== 2023-11-14T06:07:14.0571124Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-263aa259b77e97a5.xml 2023-11-14T06:07:14.0571407Z ============================= test session starts ============================== 2023-11-14T06:07:14.0572077Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0572268Z cachedir: .pytest_cache 2023-11-14T06:07:14.0573118Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0573303Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0573489Z configfile: pytest.ini 2023-11-14T06:07:14.0574287Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0574585Z collecting ... collected 1 item 2023-11-14T06:07:14.0575402Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2023-11-14T06:07:14.0575422Z 2023-11-14T06:07:14.0577164Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0577861Z STAGE:2023-11-14 06:03:09 7169:7169 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0578506Z STAGE:2023-11-14 06:03:09 7170:7170 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0579197Z STAGE:2023-11-14 06:03:09 7170:7170 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0579857Z STAGE:2023-11-14 06:03:09 7169:7169 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0580561Z STAGE:2023-11-14 06:03:09 7170:7170 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0581293Z STAGE:2023-11-14 06:03:09 7169:7169 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0581929Z STAGE:2023-11-14 06:03:09 7170:7170 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0582578Z STAGE:2023-11-14 06:03:09 7169:7169 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0583264Z STAGE:2023-11-14 06:03:09 7169:7169 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0583917Z STAGE:2023-11-14 06:03:09 7170:7170 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0584631Z STAGE:2023-11-14 06:03:09 7169:7169 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0585333Z STAGE:2023-11-14 06:03:09 7170:7170 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0585493Z PASSED [6.0374s] [100%] 2023-11-14T06:07:14.0585513Z 2023-11-14T06:07:14.0586788Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-263aa259b77e97a5.xml - 2023-11-14T06:07:14.0587063Z ============================== 1 passed in 6.07s =============================== 2023-11-14T06:07:14.0588195Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32cb87c54d935331.xml 2023-11-14T06:07:14.0588495Z ============================= test session starts ============================== 2023-11-14T06:07:14.0589148Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0589338Z cachedir: .pytest_cache 2023-11-14T06:07:14.0590214Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0590379Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0590565Z configfile: pytest.ini 2023-11-14T06:07:14.0591368Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0591556Z collecting ... collected 1 item 2023-11-14T06:07:14.0592353Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2023-11-14T06:07:14.0592382Z 2023-11-14T06:07:14.0594122Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0595041Z STAGE:2023-11-14 06:03:20 7198:7198 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0595846Z STAGE:2023-11-14 06:03:20 7197:7197 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0597239Z STAGE:2023-11-14 06:03:20 7198:7198 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 06:03:20 7197:7197 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0597288Z 2023-11-14T06:07:14.0598726Z STAGE:2023-11-14 06:03:20 7197:7197 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 06:03:20 7198:7198 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0598755Z 2023-11-14T06:07:14.0599400Z STAGE:2023-11-14 06:03:20 7198:7198 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0600061Z STAGE:2023-11-14 06:03:20 7197:7197 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0600722Z STAGE:2023-11-14 06:03:20 7197:7197 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0601403Z STAGE:2023-11-14 06:03:20 7198:7198 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0602106Z STAGE:2023-11-14 06:03:20 7197:7197 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0602799Z STAGE:2023-11-14 06:03:20 7198:7198 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0602981Z PASSED [5.8345s] [100%] 2023-11-14T06:07:14.0603000Z 2023-11-14T06:07:14.0604256Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32cb87c54d935331.xml - 2023-11-14T06:07:14.0604534Z ============================== 1 passed in 5.87s =============================== 2023-11-14T06:07:14.0605581Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c0793b30ce0e2fd.xml 2023-11-14T06:07:14.0605855Z ============================= test session starts ============================== 2023-11-14T06:07:14.0606538Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0606706Z cachedir: .pytest_cache 2023-11-14T06:07:14.0607558Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0607766Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0607926Z configfile: pytest.ini 2023-11-14T06:07:14.0608875Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0609082Z collecting ... collected 1 item 2023-11-14T06:07:14.0609814Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2023-11-14T06:07:14.0609836Z 2023-11-14T06:07:14.0611572Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0611883Z SKIPPED [5.8469s] (Skipped due to small world size.) [100%] 2023-11-14T06:07:14.0611904Z 2023-11-14T06:07:14.0613142Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c0793b30ce0e2fd.xml - 2023-11-14T06:07:14.0613456Z ============================== 1 skipped in 5.89s ============================== 2023-11-14T06:07:14.0614479Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2e55e0043dcf126f.xml 2023-11-14T06:07:14.0614780Z ============================= test session starts ============================== 2023-11-14T06:07:14.0615425Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0615590Z cachedir: .pytest_cache 2023-11-14T06:07:14.0616620Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0616823Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0616983Z configfile: pytest.ini 2023-11-14T06:07:14.0617815Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0617991Z collecting ... collected 1 item 2023-11-14T06:07:14.0618755Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min 2023-11-14T06:07:14.0618776Z 2023-11-14T06:07:14.0620479Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0620793Z SKIPPED [5.8482s] (Skipped due to small world size.) [100%] 2023-11-14T06:07:14.0620813Z 2023-11-14T06:07:14.0622115Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2e55e0043dcf126f.xml - 2023-11-14T06:07:14.0622377Z ============================== 1 skipped in 5.89s ============================== 2023-11-14T06:07:14.0623418Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-560ff1bcd8020f97.xml 2023-11-14T06:07:14.0623702Z ============================= test session starts ============================== 2023-11-14T06:07:14.0624356Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0624544Z cachedir: .pytest_cache 2023-11-14T06:07:14.0625395Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0625604Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0625763Z configfile: pytest.ini 2023-11-14T06:07:14.0626569Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0626774Z collecting ... collected 1 item 2023-11-14T06:07:14.0627503Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2023-11-14T06:07:14.0627524Z 2023-11-14T06:07:14.0629366Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0629699Z SKIPPED [5.9336s] (Skipped due to small world size.) [100%] 2023-11-14T06:07:14.0629719Z 2023-11-14T06:07:14.0630969Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-560ff1bcd8020f97.xml - 2023-11-14T06:07:14.0631282Z ============================== 1 skipped in 5.97s ============================== 2023-11-14T06:07:14.0632302Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-884269dd150ded62.xml 2023-11-14T06:07:14.0632577Z ============================= test session starts ============================== 2023-11-14T06:07:14.0633253Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0633429Z cachedir: .pytest_cache 2023-11-14T06:07:14.0634308Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0634497Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0634887Z configfile: pytest.ini 2023-11-14T06:07:14.0635730Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0635910Z collecting ... collected 1 item 2023-11-14T06:07:14.0636734Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2023-11-14T06:07:14.0636759Z 2023-11-14T06:07:14.0638482Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0639155Z STAGE:2023-11-14 06:04:05 7280:7280 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0639817Z STAGE:2023-11-14 06:04:06 7279:7279 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0641180Z STAGE:2023-11-14 06:04:06 7279:7279 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 06:04:06 7280:7280 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0641201Z 2023-11-14T06:07:14.0642642Z STAGE:2023-11-14 06:04:06 7280:7280 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 06:04:06 7279:7279 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0642689Z 2023-11-14T06:07:14.0643333Z STAGE:2023-11-14 06:04:06 7280:7280 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0643975Z STAGE:2023-11-14 06:04:06 7279:7279 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0644673Z STAGE:2023-11-14 06:04:06 7280:7280 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0645329Z STAGE:2023-11-14 06:04:06 7279:7279 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0646025Z STAGE:2023-11-14 06:04:06 7280:7280 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0646740Z STAGE:2023-11-14 06:04:06 7279:7279 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0646900Z PASSED [5.9386s] [100%] 2023-11-14T06:07:14.0646926Z 2023-11-14T06:07:14.0648200Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-884269dd150ded62.xml - 2023-11-14T06:07:14.0648476Z ============================== 1 passed in 5.98s =============================== 2023-11-14T06:07:14.0649492Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c63d4dd5c837ed8f.xml 2023-11-14T06:07:14.0649937Z ============================= test session starts ============================== 2023-11-14T06:07:14.0650597Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0650788Z cachedir: .pytest_cache 2023-11-14T06:07:14.0651636Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0651824Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0652020Z configfile: pytest.ini 2023-11-14T06:07:14.0652828Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0653010Z collecting ... collected 1 item 2023-11-14T06:07:14.0653759Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_multigpu 2023-11-14T06:07:14.0653787Z 2023-11-14T06:07:14.0655732Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_multigpu <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only Nccl backend supports reduce multigpu) [100%] 2023-11-14T06:07:14.0655753Z 2023-11-14T06:07:14.0657027Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c63d4dd5c837ed8f.xml - 2023-11-14T06:07:14.0657411Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:14.0658449Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-997d52848e6f94f4.xml 2023-11-14T06:07:14.0658754Z ============================= test session starts ============================== 2023-11-14T06:07:14.0659409Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0659611Z cachedir: .pytest_cache 2023-11-14T06:07:14.0660462Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0660648Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0660838Z configfile: pytest.ini 2023-11-14T06:07:14.0661639Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0661816Z collecting ... collected 1 item 2023-11-14T06:07:14.0662575Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product 2023-11-14T06:07:14.0662597Z 2023-11-14T06:07:14.0664299Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0665053Z STAGE:2023-11-14 06:04:22 7305:7305 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0665700Z STAGE:2023-11-14 06:04:22 7304:7304 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0666361Z STAGE:2023-11-14 06:04:22 7304:7304 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0667051Z STAGE:2023-11-14 06:04:22 7305:7305 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0667759Z STAGE:2023-11-14 06:04:22 7304:7304 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0668494Z STAGE:2023-11-14 06:04:22 7305:7305 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0669131Z STAGE:2023-11-14 06:04:22 7305:7305 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0669799Z STAGE:2023-11-14 06:04:22 7304:7304 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0670605Z STAGE:2023-11-14 06:04:22 7305:7305 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0671264Z STAGE:2023-11-14 06:04:22 7304:7304 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0671960Z STAGE:2023-11-14 06:04:22 7305:7305 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0672679Z STAGE:2023-11-14 06:04:22 7304:7304 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0672845Z PASSED [5.8371s] [100%] 2023-11-14T06:07:14.0672867Z 2023-11-14T06:07:14.0674145Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-997d52848e6f94f4.xml - 2023-11-14T06:07:14.0674420Z ============================== 1 passed in 5.88s =============================== 2023-11-14T06:07:14.0675667Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-64a9352591e67483.xml 2023-11-14T06:07:14.0675986Z ============================= test session starts ============================== 2023-11-14T06:07:14.0676650Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0676817Z cachedir: .pytest_cache 2023-11-14T06:07:14.0677684Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0678001Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0678208Z configfile: pytest.ini 2023-11-14T06:07:14.0679018Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0679197Z collecting ... collected 1 item 2023-11-14T06:07:14.0679991Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2023-11-14T06:07:14.0680019Z 2023-11-14T06:07:14.0681936Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only Nccl supports reduce_scatter_v) [100%] 2023-11-14T06:07:14.0681958Z 2023-11-14T06:07:14.0683225Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-64a9352591e67483.xml - 2023-11-14T06:07:14.0683508Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:14.0684526Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-04a0409456687347.xml 2023-11-14T06:07:14.0684829Z ============================= test session starts ============================== 2023-11-14T06:07:14.0685478Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0685675Z cachedir: .pytest_cache 2023-11-14T06:07:14.0686523Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0686710Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0686897Z configfile: pytest.ini 2023-11-14T06:07:14.0687696Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0687883Z collecting ... collected 1 item 2023-11-14T06:07:14.0688604Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2023-11-14T06:07:14.0688625Z 2023-11-14T06:07:14.0690296Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0691797Z STAGE:2023-11-14 06:04:39 7329:7329 ActivityProfilerController.cpp:314] Completed Stage: Warm UpSTAGE:2023-11-14 06:04:39 7330:7330 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0691817Z 2023-11-14T06:07:14.0692484Z STAGE:2023-11-14 06:04:39 7330:7330 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0693148Z STAGE:2023-11-14 06:04:39 7329:7329 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0693875Z STAGE:2023-11-14 06:04:39 7330:7330 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0694575Z STAGE:2023-11-14 06:04:39 7329:7329 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0695241Z STAGE:2023-11-14 06:04:39 7330:7330 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0695878Z STAGE:2023-11-14 06:04:39 7329:7329 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0696557Z STAGE:2023-11-14 06:04:39 7330:7330 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0697233Z STAGE:2023-11-14 06:04:39 7329:7329 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0697929Z STAGE:2023-11-14 06:04:39 7330:7330 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0698622Z STAGE:2023-11-14 06:04:39 7329:7329 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0698911Z PASSED [5.8357s] [100%] 2023-11-14T06:07:14.0698934Z 2023-11-14T06:07:14.0700193Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-04a0409456687347.xml - 2023-11-14T06:07:14.0700494Z ============================== 1 passed in 5.87s =============================== 2023-11-14T06:07:14.0701516Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f34ff544b4c04c0c.xml 2023-11-14T06:07:14.0701802Z ============================= test session starts ============================== 2023-11-14T06:07:14.0702481Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0702643Z cachedir: .pytest_cache 2023-11-14T06:07:14.0703519Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0703711Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0703874Z configfile: pytest.ini 2023-11-14T06:07:14.0704700Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0704877Z collecting ... collected 1 item 2023-11-14T06:07:14.0705643Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2023-11-14T06:07:14.0705673Z 2023-11-14T06:07:14.0707589Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Only Nccl supports CUDA reduce) [100%] 2023-11-14T06:07:14.0707610Z 2023-11-14T06:07:14.0708855Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f34ff544b4c04c0c.xml - 2023-11-14T06:07:14.0709157Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:14.0710177Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-877844c626dccdad.xml 2023-11-14T06:07:14.0710457Z ============================= test session starts ============================== 2023-11-14T06:07:14.0711135Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0711421Z cachedir: .pytest_cache 2023-11-14T06:07:14.0712315Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0712499Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0712663Z configfile: pytest.ini 2023-11-14T06:07:14.0713495Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0713683Z collecting ... collected 1 item 2023-11-14T06:07:14.0714404Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2023-11-14T06:07:14.0714425Z 2023-11-14T06:07:14.0716393Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0716572Z PASSED [5.7418s] [100%] 2023-11-14T06:07:14.0716591Z 2023-11-14T06:07:14.0717871Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-877844c626dccdad.xml - 2023-11-14T06:07:14.0718146Z ============================== 1 passed in 5.78s =============================== 2023-11-14T06:07:14.0719163Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-27b712d4ae912e86.xml 2023-11-14T06:07:14.0719587Z ============================= test session starts ============================== 2023-11-14T06:07:14.0720267Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0720456Z cachedir: .pytest_cache 2023-11-14T06:07:14.0721301Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0721498Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0721685Z configfile: pytest.ini 2023-11-14T06:07:14.0722490Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0722669Z collecting ... collected 1 item 2023-11-14T06:07:14.0723420Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2023-11-14T06:07:14.0723441Z 2023-11-14T06:07:14.0725307Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0725999Z STAGE:2023-11-14 06:05:07 7372:7372 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0726640Z STAGE:2023-11-14 06:05:07 7373:7373 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0727390Z STAGE:2023-11-14 06:05:07 7373:7373 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0728277Z STAGE:2023-11-14 06:05:07 7373:7373 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0728941Z STAGE:2023-11-14 06:05:07 7372:7372 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0729667Z STAGE:2023-11-14 06:05:07 7372:7372 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0730307Z STAGE:2023-11-14 06:05:07 7373:7373 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0730950Z STAGE:2023-11-14 06:05:07 7372:7372 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0731632Z STAGE:2023-11-14 06:05:07 7373:7373 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0732281Z STAGE:2023-11-14 06:05:07 7372:7372 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0733142Z STAGE:2023-11-14 06:05:07 7373:7373 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0733866Z STAGE:2023-11-14 06:05:07 7372:7372 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0734026Z PASSED [5.9389s] [100%] 2023-11-14T06:07:14.0734047Z 2023-11-14T06:07:14.0735314Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-27b712d4ae912e86.xml - 2023-11-14T06:07:14.0735595Z ============================== 1 passed in 5.98s =============================== 2023-11-14T06:07:14.0736618Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f2ec2b9bb9299300.xml 2023-11-14T06:07:14.0736919Z ============================= test session starts ============================== 2023-11-14T06:07:14.0737581Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0737770Z cachedir: .pytest_cache 2023-11-14T06:07:14.0738619Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0738805Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0738992Z configfile: pytest.ini 2023-11-14T06:07:14.0739803Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0740079Z collecting ... collected 1 item 2023-11-14T06:07:14.0740829Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda 2023-11-14T06:07:14.0740850Z 2023-11-14T06:07:14.0742683Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (Only Nccl supports CUDA gather) [100%] 2023-11-14T06:07:14.0742714Z 2023-11-14T06:07:14.0743981Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f2ec2b9bb9299300.xml - 2023-11-14T06:07:14.0744261Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:14.0745277Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f7a9233114a957f.xml 2023-11-14T06:07:14.0745587Z ============================= test session starts ============================== 2023-11-14T06:07:14.0746238Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0746429Z cachedir: .pytest_cache 2023-11-14T06:07:14.0747276Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0747472Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0747665Z configfile: pytest.ini 2023-11-14T06:07:14.0748473Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0748652Z collecting ... collected 1 item 2023-11-14T06:07:14.0749436Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2023-11-14T06:07:14.0749456Z 2023-11-14T06:07:14.0751345Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (Only Nccl supports CUDA gather) [100%] 2023-11-14T06:07:14.0751367Z 2023-11-14T06:07:14.0752636Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f7a9233114a957f.xml - 2023-11-14T06:07:14.0753032Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:14.0754055Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-010ac736bd74e081.xml 2023-11-14T06:07:14.0754362Z ============================= test session starts ============================== 2023-11-14T06:07:14.0755214Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0755418Z cachedir: .pytest_cache 2023-11-14T06:07:14.0756281Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0756467Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0756654Z configfile: pytest.ini 2023-11-14T06:07:14.0757453Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0757641Z collecting ... collected 1 item 2023-11-14T06:07:14.0758414Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2023-11-14T06:07:14.0758434Z 2023-11-14T06:07:14.0760145Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0761627Z STAGE:2023-11-14 06:05:29 7401:7401 ActivityProfilerController.cpp:314] Completed Stage: Warm UpSTAGE:2023-11-14 06:05:29 7400:7400 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0761652Z 2023-11-14T06:07:14.0763033Z STAGE:2023-11-14 06:05:30 7400:7400 ActivityProfilerController.cpp:320] Completed Stage: CollectionSTAGE:2023-11-14 06:05:30 7401:7401 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0763060Z 2023-11-14T06:07:14.0764525Z STAGE:2023-11-14 06:05:30 7401:7401 ActivityProfilerController.cpp:324] Completed Stage: Post ProcessingSTAGE:2023-11-14 06:05:30 7400:7400 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0764547Z 2023-11-14T06:07:14.0765181Z STAGE:2023-11-14 06:05:30 7400:7400 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0765817Z STAGE:2023-11-14 06:05:30 7401:7401 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0766513Z STAGE:2023-11-14 06:05:30 7401:7401 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0767175Z STAGE:2023-11-14 06:05:30 7400:7400 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0767893Z STAGE:2023-11-14 06:05:30 7401:7401 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0768587Z STAGE:2023-11-14 06:05:30 7400:7400 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0768756Z PASSED [5.7335s] [100%] 2023-11-14T06:07:14.0768777Z 2023-11-14T06:07:14.0770099Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-010ac736bd74e081.xml - 2023-11-14T06:07:14.0770379Z ============================== 1 passed in 5.77s =============================== 2023-11-14T06:07:14.0771437Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e398410b713cf020.xml 2023-11-14T06:07:14.0771720Z ============================= test session starts ============================== 2023-11-14T06:07:14.0772372Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0772559Z cachedir: .pytest_cache 2023-11-14T06:07:14.0773408Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0773752Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0773943Z configfile: pytest.ini 2023-11-14T06:07:14.0774750Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0774926Z collecting ... collected 1 item 2023-11-14T06:07:14.0775666Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2023-11-14T06:07:14.0775688Z 2023-11-14T06:07:14.0777389Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0777726Z SKIPPED [5.8460s] (Skipped due to small world size.) [100%] 2023-11-14T06:07:14.0777748Z 2023-11-14T06:07:14.0778994Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e398410b713cf020.xml - 2023-11-14T06:07:14.0779303Z ============================== 1 skipped in 5.88s ============================== 2023-11-14T06:07:14.0780310Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a5303858a1646c9f.xml 2023-11-14T06:07:14.0780584Z ============================= test session starts ============================== 2023-11-14T06:07:14.0781427Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0781613Z cachedir: .pytest_cache 2023-11-14T06:07:14.0782471Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0782682Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0782845Z configfile: pytest.ini 2023-11-14T06:07:14.0783669Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0783857Z collecting ... collected 1 item 2023-11-14T06:07:14.0784619Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2023-11-14T06:07:14.0784640Z 2023-11-14T06:07:14.0786394Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0786557Z PASSED [5.7384s] [100%] 2023-11-14T06:07:14.0786578Z 2023-11-14T06:07:14.0787831Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a5303858a1646c9f.xml - 2023-11-14T06:07:14.0788103Z ============================== 1 passed in 5.78s =============================== 2023-11-14T06:07:14.0789132Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f3f026ed1bc50aaa.xml 2023-11-14T06:07:14.0789430Z ============================= test session starts ============================== 2023-11-14T06:07:14.0790077Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0790241Z cachedir: .pytest_cache 2023-11-14T06:07:14.0791112Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0791311Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0791499Z configfile: pytest.ini 2023-11-14T06:07:14.0792302Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0792481Z collecting ... collected 1 item 2023-11-14T06:07:14.0793358Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2023-11-14T06:07:14.0793479Z 2023-11-14T06:07:14.0795522Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0796215Z STAGE:2023-11-14 06:06:03 7464:7464 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0796865Z STAGE:2023-11-14 06:06:03 7465:7465 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0797533Z STAGE:2023-11-14 06:06:03 7464:7464 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0798209Z STAGE:2023-11-14 06:06:03 7465:7465 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0798905Z STAGE:2023-11-14 06:06:03 7464:7464 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0799607Z STAGE:2023-11-14 06:06:03 7465:7465 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0799791Z PASSED [5.8348s] [100%] 2023-11-14T06:07:14.0799811Z 2023-11-14T06:07:14.0801059Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f3f026ed1bc50aaa.xml - 2023-11-14T06:07:14.0801356Z ============================== 1 passed in 5.87s =============================== 2023-11-14T06:07:14.0802504Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-020d0ed4d835eaa5.xml 2023-11-14T06:07:14.0802808Z ============================= test session starts ============================== 2023-11-14T06:07:14.0803493Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0803655Z cachedir: .pytest_cache 2023-11-14T06:07:14.0804518Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0804729Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0804891Z configfile: pytest.ini 2023-11-14T06:07:14.0805711Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0805886Z collecting ... collected 1 item 2023-11-14T06:07:14.0806723Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler 2023-11-14T06:07:14.0806744Z 2023-11-14T06:07:14.0808652Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0002s] (NCCL Send Recv Only) [100%] 2023-11-14T06:07:14.0808674Z 2023-11-14T06:07:14.0809927Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-020d0ed4d835eaa5.xml - 2023-11-14T06:07:14.0810221Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:14.0811237Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-42e62a6fa656f400.xml 2023-11-14T06:07:14.0811516Z ============================= test session starts ============================== 2023-11-14T06:07:14.0812196Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0812365Z cachedir: .pytest_cache 2023-11-14T06:07:14.0813239Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0813427Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0813590Z configfile: pytest.ini 2023-11-14T06:07:14.0814569Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0814749Z collecting ... collected 1 item 2023-11-14T06:07:14.0815559Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2023-11-14T06:07:14.0815580Z 2023-11-14T06:07:14.0817461Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0003s] (NCCL Send Recv Only) [100%] 2023-11-14T06:07:14.0817483Z 2023-11-14T06:07:14.0818730Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-42e62a6fa656f400.xml - 2023-11-14T06:07:14.0819030Z ============================== 1 skipped in 0.04s ============================== 2023-11-14T06:07:14.0820053Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-54819b4865d23f05.xml 2023-11-14T06:07:14.0820332Z ============================= test session starts ============================== 2023-11-14T06:07:14.0821008Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0821171Z cachedir: .pytest_cache 2023-11-14T06:07:14.0822140Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0822339Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0822502Z configfile: pytest.ini 2023-11-14T06:07:14.0823333Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0823507Z collecting ... collected 1 item 2023-11-14T06:07:14.0824362Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2023-11-14T06:07:14.0824391Z 2023-11-14T06:07:14.0826252Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0826906Z STAGE:2023-11-14 06:06:26 7493:7493 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0827571Z STAGE:2023-11-14 06:06:26 7492:7492 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0828237Z STAGE:2023-11-14 06:06:26 7492:7492 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0828936Z STAGE:2023-11-14 06:06:26 7492:7492 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0829610Z STAGE:2023-11-14 06:06:26 7493:7493 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0830315Z STAGE:2023-11-14 06:06:26 7493:7493 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0830498Z PASSED [5.9372s] [100%] 2023-11-14T06:07:14.0830518Z 2023-11-14T06:07:14.0831757Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-54819b4865d23f05.xml - 2023-11-14T06:07:14.0832033Z ============================== 1 passed in 5.97s =============================== 2023-11-14T06:07:14.0833087Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8d505b21c2e6332c.xml 2023-11-14T06:07:14.0833365Z ============================= test session starts ============================== 2023-11-14T06:07:14.0834041Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0834325Z cachedir: .pytest_cache 2023-11-14T06:07:14.0835369Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0835588Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0835751Z configfile: pytest.ini 2023-11-14T06:07:14.0836562Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0836765Z collecting ... collected 1 item 2023-11-14T06:07:14.0837611Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2023-11-14T06:07:14.0837632Z 2023-11-14T06:07:14.0839467Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0840125Z STAGE:2023-11-14 06:06:37 7515:7515 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0840764Z STAGE:2023-11-14 06:06:37 7514:7514 ActivityProfilerController.cpp:314] Completed Stage: Warm Up 2023-11-14T06:07:14.0841450Z STAGE:2023-11-14 06:06:37 7514:7514 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0842154Z STAGE:2023-11-14 06:06:37 7514:7514 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0842951Z STAGE:2023-11-14 06:06:37 7515:7515 ActivityProfilerController.cpp:320] Completed Stage: Collection 2023-11-14T06:07:14.0843702Z STAGE:2023-11-14 06:06:37 7515:7515 ActivityProfilerController.cpp:324] Completed Stage: Post Processing 2023-11-14T06:07:14.0843862Z PASSED [5.9457s] [100%] 2023-11-14T06:07:14.0843883Z 2023-11-14T06:07:14.0845154Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8d505b21c2e6332c.xml - 2023-11-14T06:07:14.0845438Z ============================== 1 passed in 5.98s =============================== 2023-11-14T06:07:14.0846458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b55fee1eeb8885cc.xml 2023-11-14T06:07:14.0846757Z ============================= test session starts ============================== 2023-11-14T06:07:14.0847411Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0847597Z cachedir: .pytest_cache 2023-11-14T06:07:14.0848451Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0848640Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0848832Z configfile: pytest.ini 2023-11-14T06:07:14.0849631Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0849816Z collecting ... collected 1 item 2023-11-14T06:07:14.0850606Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2023-11-14T06:07:14.0850626Z 2023-11-14T06:07:14.0852364Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0852548Z PASSED [5.8523s] [100%] 2023-11-14T06:07:14.0852569Z 2023-11-14T06:07:14.0853827Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b55fee1eeb8885cc.xml - 2023-11-14T06:07:14.0854104Z ============================== 1 passed in 5.89s =============================== 2023-11-14T06:07:14.0855148Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f635cdfd203c24d6.xml 2023-11-14T06:07:14.0855567Z ============================= test session starts ============================== 2023-11-14T06:07:14.0856250Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0856417Z cachedir: .pytest_cache 2023-11-14T06:07:14.0857262Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0857471Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0857641Z configfile: pytest.ini 2023-11-14T06:07:14.0858448Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0858651Z collecting ... collected 1 item 2023-11-14T06:07:14.0859442Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2023-11-14T06:07:14.0859470Z 2023-11-14T06:07:14.0861267Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0861429Z PASSED [6.0447s] [100%] 2023-11-14T06:07:14.0861450Z 2023-11-14T06:07:14.0862693Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f635cdfd203c24d6.xml - 2023-11-14T06:07:14.0863099Z ============================== 1 passed in 6.08s =============================== 2023-11-14T06:07:14.0864141Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b3811cd57ad63e8.xml 2023-11-14T06:07:14.0864447Z ============================= test session starts ============================== 2023-11-14T06:07:14.0865095Z platform linux -- Python 3.10.13, pytest-7.3.2, pluggy-1.3.0 -- /opt/conda/envs/py_3.10/bin/python 2023-11-14T06:07:14.0865270Z cachedir: .pytest_cache 2023-11-14T06:07:14.0866144Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2023-11-14T06:07:14.0866330Z rootdir: /var/lib/jenkins/workspace 2023-11-14T06:07:14.0866497Z configfile: pytest.ini 2023-11-14T06:07:14.0867326Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-12.0, shard-0.1.2, xdist-3.3.1, xdoctest-1.1.0 2023-11-14T06:07:14.0867511Z collecting ... collected 1 item 2023-11-14T06:07:14.0868308Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2023-11-14T06:07:14.0868329Z 2023-11-14T06:07:14.0870130Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp <- ../../../../opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py INFO:numba.cuda.cudadrv.driver:init 2023-11-14T06:07:14.0870306Z PASSED [6.1370s] [100%] 2023-11-14T06:07:14.0870330Z 2023-11-14T06:07:14.0871615Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b3811cd57ad63e8.xml - 2023-11-14T06:07:14.0871895Z ============================== 1 passed in 6.17s =============================== 2023-11-14T06:07:14.0872106Z Traceback (most recent call last): 2023-11-14T06:07:14.0872733Z File "/var/lib/jenkins/workspace/test/distributed/test_distributed_spawn.py", line 59, in 2023-11-14T06:07:14.0872882Z run_tests() 2023-11-14T06:07:14.0873789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1011, in run_tests 2023-11-14T06:07:14.0874151Z assert len(failed_tests) == 0, "{} unit test(s) failed:\n\t{}".format( 2023-11-14T06:07:14.0874354Z AssertionError: 1 unit test(s) failed: 2023-11-14T06:07:14.0875194Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2023-11-14T06:07:14.0875217Z 2023-11-14T06:07:14.0876496Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn 2/2 (/var/lib/jenkins/workspace/test/test-reports/distributed.test_distributed_spawn_2.2_ddlahcw4_toprint.log) 2023-11-14T06:07:14.0876517Z 2023-11-14T06:07:14.0876784Z distributed/test_distributed_spawn 2/2 failed! 2023-11-14T06:07:14.0877373Z Starting test batch 'probable_relevance' 4306.259344577789 seconds after initiating testing 2023-11-14T06:07:14.0877601Z With sharding, this batch will run 20 tests 2023-11-14T06:07:16.0257328Z Running distributed/fsdp/test_fsdp_core 1/2 ... [2023-11-14 06:07:16.024991] 2023-11-14T06:07:16.0269865Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '--shard-id=0', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:07:16.026527] 2023-11-14T06:14:03.0862263Z 2023-11-14T06:14:03.0865121Z distributed/fsdp/test_fsdp_core 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_core_1.2_b2f383fff00019b6_.log 2023-11-14T06:14:03.0895534Z Running 22 items in this shard: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParamInit::test_param_change_after_init_mixed_precision_False, test/distributed/fsdp/test_fsdp_core.py::TestParamInit::test_param_change_after_init_mixed_precision_True, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_pre_backward_hook_registration_cuda_first_False, test/distributed/fsdp/test_fsdp_core.py::TestNoGrad::test_transformer_no_grad_mixed_precision_False, test/distributed/fsdp/test_fsdp_core.py::TestNoGrad::test_transformer_no_grad_mixed_precision_True, test/distributed/fsdp/test_fsdp_core.py::TestAutograd::test_unshard_params_as_tensors 2023-11-14T06:14:03.0912054Z 2023-11-14T06:14:03.0912777Z Running distributed/tensor/parallel/test_fsdp_2d_parallel 1/1 ... [2023-11-14 06:14:03.089879] 2023-11-14T06:14:03.0915627Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_fsdp_2d_parallel.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:14:03.090183] 2023-11-14T06:14:54.4682137Z 2023-11-14T06:14:54.4684527Z distributed/tensor/parallel/test_fsdp_2d_parallel 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_fsdp_2d_parallel_1.1_dcae9ad6d8b7f765_.log 2023-11-14T06:14:54.4696519Z Running 12 items in this shard: test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_2d_e2e_training_default, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_2d_e2e_training_not_use_orig_params, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_2d_e2e_training_use_orig_params, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_2d_fsdp_state_enable_extension, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_raise_invalid_tp_composition, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_2d_load_state_dict_is_even_sharded_model_False, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_2d_load_state_dict_is_even_sharded_model_True, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_2d_optim_state_dict_is_even_sharded_model_False, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_2d_optim_state_dict_is_even_sharded_model_True, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_2d_state_dict_is_even_sharded_model_False, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_2d_state_dict_is_even_sharded_model_True, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_fsdp_2d_extension 2023-11-14T06:14:54.4706997Z 2023-11-14T06:14:54.4707473Z Running distributed/test_c10d_ucc 1/1 ... [2023-11-14 06:14:54.468381] 2023-11-14T06:14:54.4709804Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_ucc.py', '--shard-id=0', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:14:54.468687] 2023-11-14T06:22:07.8794197Z 2023-11-14T06:22:07.8796459Z distributed/test_c10d_ucc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_ucc_1.1_43a9d5815ac9ab5f_.log 2023-11-14T06:22:07.8803150Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::RendezvousEnvTest::test_logging_init 2023-11-14T06:22:07.8805116Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::TimeoutTest::test_default_store_timeout_ucc 2023-11-14T06:22:07.8806910Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_allgather_basics 2023-11-14T06:22:07.8808729Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_allreduce_basics 2023-11-14T06:22:07.8810803Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_barrier_implies_wait 2023-11-14T06:22:07.8812592Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_broadcast_basics 2023-11-14T06:22:07.8814330Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_empty_tensors 2023-11-14T06:22:07.8816235Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_reduce_basics 2023-11-14T06:22:07.8818177Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::ProcessGroupUCCTest::test_send_recv_all_to_all 2023-11-14T06:22:07.8820027Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_dataclass_output 2023-11-14T06:22:07.8822610Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_dataclass_output_unused_param 2023-11-14T06:22:07.8824976Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2023-11-14T06:22:07.8827099Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2023-11-14T06:22:07.8829607Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2023-11-14T06:22:07.8832076Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2023-11-14T06:22:07.8834871Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False 2023-11-14T06:22:07.8837444Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2023-11-14T06:22:07.8839727Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2023-11-14T06:22:07.8842185Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2023-11-14T06:22:07.8844943Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing 2023-11-14T06:22:07.8847535Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False 2023-11-14T06:22:07.8850125Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2023-11-14T06:22:07.8852538Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False 2023-11-14T06:22:07.8854954Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2023-11-14T06:22:07.8857457Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu 2023-11-14T06:22:07.8859834Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_ucc 2023-11-14T06:22:07.8861894Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once 2023-11-14T06:22:07.8863971Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients 2023-11-14T06:22:07.8865993Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init 2023-11-14T06:22:07.8867743Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type 2023-11-14T06:22:07.8869922Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty 2023-11-14T06:22:07.8871830Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_global_local_unused_params_grad 2023-11-14T06:22:07.8873693Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view 2023-11-14T06:22:07.8876017Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph 2023-11-14T06:22:07.8878318Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ignored_output 2023-11-14T06:22:07.8880018Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters 2023-11-14T06:22:07.8881757Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_invalid_powerSGD_state 2023-11-14T06:22:07.8883400Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_save_load_checkpoint 2023-11-14T06:22:07.8884972Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sparse_gradients 2023-11-14T06:22:07.8886623Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view 2023-11-14T06:22:07.8888352Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2023-11-14T06:22:07.8890171Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input 2023-11-14T06:22:07.8892226Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_1gpu_module_device_ids_integer_list 2023-11-14T06:22:07.8894350Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_1gpu_module_device_ids_torch_device_list 2023-11-14T06:22:07.8896210Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_2gpu_module 2023-11-14T06:22:07.8897894Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_4gpu_module 2023-11-14T06:22:07.8899547Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_cpu_module 2023-11-14T06:22:07.8901277Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_ucc_backend_cpu_module_grad_is_view 2023-11-14T06:22:07.8902943Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_incremented_ucc_default 2023-11-14T06:22:07.8904458Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_incremented_ucc_subgroup 2023-11-14T06:22:07.8905984Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_set_default_pg_ucc 2023-11-14T06:22:07.8907455Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_sequence_num_set_ucc_new_group 2023-11-14T06:22:07.8908857Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_tensor_dtype_complex 2023-11-14T06:22:07.8910177Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_tensor_dtype_mismatch 2023-11-14T06:22:07.8911549Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_ucc_barrier_device_ids 2023-11-14T06:22:07.8912905Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_ucc_rank_membership 2023-11-14T06:22:07.8914223Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CommTest::test_ucc_warn_not_in_group 2023-11-14T06:22:07.8915893Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_allgather_work_wait_cpu 2023-11-14T06:22:07.8917632Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_allgather_work_wait_gpu 2023-11-14T06:22:07.8919082Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_allreduce_work_wait_cpu 2023-11-14T06:22:07.8920486Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_allreduce_work_wait_gpu 2023-11-14T06:22:07.8922090Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_broadcast_work_wait_cpu 2023-11-14T06:22:07.8923527Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_broadcast_work_wait_gpu 2023-11-14T06:22:07.8925003Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_consecutive_comm_work_wait_cpu 2023-11-14T06:22:07.8926486Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_consecutive_comm_work_wait_gpu 2023-11-14T06:22:07.8928011Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_nested_comm_tensor_wrapping_cpu 2023-11-14T06:22:07.8929544Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::CompilerTest::test_nested_comm_tensor_wrapping_gpu 2023-11-14T06:22:07.8931214Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_allgather_base 2023-11-14T06:22:07.8933065Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_collectives 2023-11-14T06:22:07.8935021Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2023-11-14T06:22:07.8937119Z Running 1 items in this shard: test/distributed/test_c10d_ucc.py::UccProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2023-11-14T06:22:07.8938260Z 2023-11-14T06:22:07.8939052Z Running distributed/fsdp/test_fsdp_unshard_params 1/1 ... [2023-11-14 06:22:07.881623] 2023-11-14T06:22:07.8941849Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_unshard_params.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:22:07.882013] 2023-11-14T06:23:30.5675356Z 2023-11-14T06:23:30.5677776Z distributed/fsdp/test_fsdp_unshard_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_unshard_params_1.1_dfdb81cd23df9d6f_.log 2023-11-14T06:23:30.5689222Z Running 14 items in this shard: test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_named_parameters_and_buffers, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_param_data, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_recurse, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_respects_reshard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_writeback, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_singleton_param_writeback, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_with_grads_core, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_with_grads_none_grads, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsNoShard::test_unshard_params_param_data_no_shard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsNoShard::test_unshard_params_writeback_no_shard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_offload_to_cpu_no_shard_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_rank0_only_with_writeback_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_unshard_params_from_backward_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_unshard_params_from_forward_raises 2023-11-14T06:23:30.5699245Z 2023-11-14T06:23:30.5699710Z Running distributed/test_launcher 1/1 ... [2023-11-14 06:23:30.567903] 2023-11-14T06:23:30.5702039Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_launcher.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:23:30.568215] 2023-11-14T06:23:34.1406205Z 2023-11-14T06:23:34.1408156Z distributed/test_launcher 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_launcher_1.1_2c56c5528fbe9ba1_.log 2023-11-14T06:23:34.1410450Z Running 1 items in this shard: test/distributed/test_launcher.py::TestDistributedLaunch::test_launch_user_script 2023-11-14T06:23:34.1411375Z 2023-11-14T06:23:34.1412269Z Running distributed/test_inductor_collectives 1/1 ... [2023-11-14 06:23:34.140787] 2023-11-14T06:23:34.1415090Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_inductor_collectives.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:23:34.141095] 2023-11-14T06:23:45.1290067Z 2023-11-14T06:23:45.1292055Z distributed/test_inductor_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_inductor_collectives_1.1_dc424b0b5fd53bbd_.log 2023-11-14T06:23:45.1313952Z Running 30 items in this shard: test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_input_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_output_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_contiguous_input, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_into_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_output_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_input_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_broadcast_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_c10d_functional_tagged_pt2_compliant, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_inductor_allreduce_eager_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_reduce_scatter_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_backwards, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_graphbreaks_unsupported_async_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_pg_var, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_reduce_scatter, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_pg, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allgather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allreduce, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_reduce_scatter_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_all_gather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_doesnt_mutate_shared, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_reduce_scatter_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_single_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_steal_buffer, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_meta 2023-11-14T06:23:45.1334910Z 2023-11-14T06:23:45.1335617Z Running distributed/_shard/sharded_optim/test_sharded_optim 1/1 ... [2023-11-14 06:23:45.129220] 2023-11-14T06:23:45.1338236Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:23:45.129531] 2023-11-14T06:23:54.7159225Z 2023-11-14T06:23:54.7161757Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_optim.test_sharded_optim_1.1_e4069bc29a1e9457_.log 2023-11-14T06:23:54.7164955Z Running 2 items in this shard: test/distributed/_shard/sharded_optim/test_sharded_optim.py::TestShardedOptimizer::test_named_params_with_sharded_tensor, test/distributed/_shard/sharded_optim/test_sharded_optim.py::TestShardedOptimizer::test_sharded_optim 2023-11-14T06:23:54.7166762Z 2023-11-14T06:23:54.7167637Z Running distributed/_tensor/test_dtensor_ops 1/1 ... [2023-11-14 06:23:54.716058] 2023-11-14T06:23:54.7170763Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_ops.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:23:54.716355] 2023-11-14T06:24:00.4818602Z 2023-11-14T06:24:00.4820636Z distributed/_tensor/test_dtensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_ops_1.1_d3ae1cbf6d08bb8d_.log 2023-11-14T06:24:00.4822232Z Running 0 items in this shard: 2023-11-14T06:24:00.4822573Z 2023-11-14T06:24:00.4823827Z Running distributed/rpc/cuda/test_tensorpipe_agent 1/2 ... [2023-11-14 06:24:00.482072] 2023-11-14T06:24:00.4828169Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '--shard-id=0', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:24:00.482375] 2023-11-14T06:35:40.1665975Z 2023-11-14T06:35:40.1670805Z distributed/rpc/cuda/test_tensorpipe_agent 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.cuda.test_tensorpipe_agent_1.2_52f2d26eb667a09e_.log 2023-11-14T06:35:40.1673299Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRpcTest::test_profiler_remote_cuda 2023-11-14T06:35:40.1675489Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_simple 2023-11-14T06:35:40.1677373Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_to_cpu_continuation 2023-11-14T06:35:40.1679340Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDistAutogradTest::test_gpu_to_cpu_continuation_gpu_root 2023-11-14T06:35:40.1681394Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_invalid_devices 2023-11-14T06:35:40.1683195Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_valid_device 2023-11-14T06:35:40.1685308Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaDdpComparisonTest::test_ddp_dist_autograd_local_vs_remote_gpu 2023-11-14T06:35:40.1687798Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_gloo_ckpt_never_find_unused 2023-11-14T06:35:40.1691146Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_except_last 2023-11-14T06:35:40.1694403Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_never 2023-11-14T06:35:40.1696366Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipePipeWithDDPTest::test_basic_nccl_ckpt_never_find_unused 2023-11-14T06:35:40.1698454Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_async_execution_with_cuda_future 2023-11-14T06:35:40.1700657Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_cuda_tensor 2023-11-14T06:35:40.1702957Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_list_with_cuda_sparse_tensor 2023-11-14T06:35:40.1705291Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_list_with_cuda_tensor 2023-11-14T06:35:40.1707775Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_not_cuda 2023-11-14T06:35:40.1709882Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_nested 2023-11-14T06:35:40.1711886Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu 2023-11-14T06:35:40.1713872Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_default 2023-11-14T06:35:40.1716585Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_default_to_non_default 2023-11-14T06:35:40.1718737Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_1 2023-11-14T06:35:40.1720834Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_2 2023-11-14T06:35:40.1722861Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_3 2023-11-14T06:35:40.1724968Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_4 2023-11-14T06:35:40.1727035Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_5 2023-11-14T06:35:40.1729078Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_6 2023-11-14T06:35:40.1731130Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_2 2023-11-14T06:35:40.1733234Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_3 2023-11-14T06:35:40.1735333Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_4 2023-11-14T06:35:40.1737645Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_5 2023-11-14T06:35:40.1739723Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_6 2023-11-14T06:35:40.1741831Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_8 2023-11-14T06:35:40.1744001Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_non_default_to_default 2023-11-14T06:35:40.1746206Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_to_cpu_non_default 2023-11-14T06:35:40.1748320Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_in_options 2023-11-14T06:35:40.1750398Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_min_device 2023-11-14T06:35:40.1752559Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_remote 2023-11-14T06:35:40.1755319Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_response_loop 2023-11-14T06:35:40.1757535Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_one_to_many 2023-11-14T06:35:40.1759594Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_return_to_gpu 2023-11-14T06:35:40.1761725Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_return_to_gpu_self 2023-11-14T06:35:40.1763865Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_wrong_worker_name 2023-11-14T06:35:40.1765921Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_mismatch 2023-11-14T06:35:40.1767966Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_devices_option_mismatch_reverse 2023-11-14T06:35:40.1770150Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization1 2023-11-14T06:35:40.1772363Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization3 2023-11-14T06:35:40.1774543Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization1 2023-11-14T06:35:40.1776651Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization2 2023-11-14T06:35:40.1778788Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization3 2023-11-14T06:35:40.1780939Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_dist_autograd_sync_streams 2023-11-14T06:35:40.1783109Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_gradients_synchronizations 2023-11-14T06:35:40.1784438Z 2023-11-14T06:35:40.1785183Z Running distributed/_composable/fully_shard/test_fully_shard_runtime 1/1 ... [2023-11-14 06:35:40.167983] 2023-11-14T06:35:40.1787884Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_runtime.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:35:40.168294] 2023-11-14T06:35:58.3701140Z 2023-11-14T06:35:58.3703736Z distributed/_composable/fully_shard/test_fully_shard_runtime 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_runtime_1.1_1706520139d64dec_.log 2023-11-14T06:35:58.3706896Z Running 2 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_runtime.py::TestRuntime::test_training, test/distributed/_composable/fully_shard/test_fully_shard_runtime.py::TestRuntime::test_unshard_reshard_order 2023-11-14T06:35:58.3708558Z 2023-11-14T06:35:58.3709111Z Running distributed/fsdp/test_fsdp_state_dict 2/2 ... [2023-11-14 06:35:58.370337] 2023-11-14T06:35:58.3711529Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '--shard-id=1', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:35:58.370691] 2023-11-14T06:43:44.9056694Z 2023-11-14T06:43:44.9059317Z distributed/fsdp/test_fsdp_state_dict 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_state_dict_2.2_64146294559d98d5_.log 2023-11-14T06:43:44.9167254Z Running 84 items in this shard: test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_local_state_dict_with_empty_ranks, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_sharded_load_multi_backend_pg, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_shared_module_and_shared_parameter, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_rank0_offload_save_load_flow_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_world_size_one, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_wrong_state_dict_config 2023-11-14T06:43:44.9271391Z 2023-11-14T06:43:44.9271930Z Running distributed/fsdp/test_fsdp_fx 1/1 ... [2023-11-14 06:43:44.906464] 2023-11-14T06:43:44.9274265Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:43:44.906773] 2023-11-14T06:43:49.8315266Z 2023-11-14T06:43:49.8317434Z distributed/fsdp/test_fsdp_fx 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_fx_1.1_7d0ab38dc7463a54_.log 2023-11-14T06:43:49.8319481Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_fx.py::TestSymbolicTracing::test_symbolic_tracing_outputs 2023-11-14T06:43:49.8320437Z 2023-11-14T06:43:49.8321339Z Running distributed/fsdp/test_fsdp_memory 1/1 ... [2023-11-14 06:43:49.831795] 2023-11-14T06:43:49.8325740Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:43:49.832108] 2023-11-14T06:44:09.7371590Z 2023-11-14T06:44:09.7374006Z distributed/fsdp/test_fsdp_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_memory_1.1_e4c8298cc50027df_.log 2023-11-14T06:44:09.7377305Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_memory.py::TestFSDPMemory::test_fsdp_memory_ckpt_ckpt, test/distributed/fsdp/test_fsdp_memory.py::TestFSDPMemory::test_fsdp_memory_ckpt_no_ckpt 2023-11-14T06:44:09.7379079Z 2023-11-14T06:44:09.7379687Z Running distributed/_tensor/test_dtensor_compile 1/1 ... [2023-11-14 06:44:09.737324] 2023-11-14T06:44:09.7382145Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_compile.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:44:09.737630] 2023-11-14T06:44:31.2011512Z 2023-11-14T06:44:31.2014798Z distributed/_tensor/test_dtensor_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_compile_1.1_9de133838ebb9950_.log 2023-11-14T06:44:31.2023420Z Running 9 items in this shard: test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True 2023-11-14T06:44:31.2029567Z 2023-11-14T06:44:31.2030157Z Running distributed/test_compute_comm_reordering 1/1 ... [2023-11-14 06:44:31.201295] 2023-11-14T06:44:31.2032647Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_compute_comm_reordering.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:44:31.201597] 2023-11-14T06:44:38.0303044Z 2023-11-14T06:44:38.0305529Z distributed/test_compute_comm_reordering 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_compute_comm_reordering_1.1_37b81a6c4076f356_.log 2023-11-14T06:44:38.0311507Z Running 6 items in this shard: test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_nccl_heuristics, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_raise_comms, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap_custom_runtime_estimation, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits_raise_comms 2023-11-14T06:44:38.0316582Z 2023-11-14T06:44:38.0317182Z Running distributed/_tensor/debug/test_op_coverage 1/1 ... [2023-11-14 06:44:38.030417] 2023-11-14T06:44:38.0320148Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/debug/test_op_coverage.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:44:38.030708] 2023-11-14T06:44:42.7043649Z 2023-11-14T06:44:42.7045761Z distributed/_tensor/debug/test_op_coverage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_7a86c0c4dd8f7fb1_.log 2023-11-14T06:44:42.7048010Z Running 1 items in this shard: test/distributed/_tensor/debug/test_op_coverage.py::TestOpCoverage::test_trace_with_inductor_decomp 2023-11-14T06:44:42.7049007Z 2023-11-14T06:44:42.7049561Z Running distributed/fsdp/test_fsdp_ignored_modules 1/1 ... [2023-11-14 06:44:42.704541] 2023-11-14T06:44:42.7052931Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_ignored_modules.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:44:42.704843] 2023-11-14T06:45:55.3705607Z 2023-11-14T06:45:55.3707867Z distributed/fsdp/test_fsdp_ignored_modules 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_ignored_modules_1.1_b62e351429f15278_.log 2023-11-14T06:45:55.3720779Z Running 12 items in this shard: test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_diff_ignored_modules_across_ranks, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_invalid_composable_False, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_invalid_composable_True, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_nested, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_nested_composable, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_False_composable_False, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_False_composable_True, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_True_composable_False, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_True_composable_True, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_transformer, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_transformer_composable, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_states_check 2023-11-14T06:45:55.3731047Z 2023-11-14T06:45:55.3731604Z Running distributed/_tensor/test_utils 1/1 ... [2023-11-14 06:45:55.370802] 2023-11-14T06:45:55.3733929Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_utils.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:45:55.371140] 2023-11-14T06:46:08.0679878Z 2023-11-14T06:46:08.0682862Z distributed/_tensor/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_utils_1.1_263717c7fb34bd28_.log 2023-11-14T06:46:08.0686628Z Running 3 items in this shard: test/distributed/_tensor/test_utils.py::UtilTest::test_compute_local_shape_2d_uneven, test/distributed/_tensor/test_utils.py::UtilTest::test_compute_local_shape_and_global_offset_1D, test/distributed/_tensor/test_utils.py::UtilTest::test_compute_local_shape_and_global_offset_2D 2023-11-14T06:46:08.0689094Z 2023-11-14T06:46:08.0689791Z Running distributed/checkpoint/test_file_system_checkpoint 1/1 ... [2023-11-14 06:46:08.068179] 2023-11-14T06:46:08.0692405Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:46:08.068494] 2023-11-14T06:46:42.8545508Z 2023-11-14T06:46:42.8548342Z distributed/checkpoint/test_file_system_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_1.1_8809e40e39f9984f_.log 2023-11-14T06:46:42.8557671Z Running 6 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_save_load_bytes, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor 2023-11-14T06:46:42.8563425Z 2023-11-14T06:46:42.8564203Z Running distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 ... [2023-11-14 06:46:42.854803] 2023-11-14T06:46:42.8566909Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint_cpu.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:46:42.855100] 2023-11-14T06:47:34.0769097Z 2023-11-14T06:47:34.0774376Z distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_cpu_1.1_232d5753ddc23186_.log 2023-11-14T06:47:34.0787209Z Running 12 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_save_load_bytes_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_save_load_bytes_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor_thread_count_2 2023-11-14T06:47:34.0799593Z 2023-11-14T06:47:34.0800248Z Starting test batch 'unranked_relevance' 6726.535462141037 seconds after initiating testing 2023-11-14T06:47:34.0801167Z With sharding, this batch will run 43 tests 2023-11-14T06:47:36.3208322Z Running distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 ... [2023-11-14 06:47:36.320112] 2023-11-14T06:47:36.3211136Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:47:36.320517] 2023-11-14T06:50:52.9223251Z 2023-11-14T06:50:52.9225608Z distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_1.1_48a5d3de6b20f0d7_.log 2023-11-14T06:50:52.9279077Z Running 67 items in this shard: test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorMetadata::test_serialize_and_deserialize, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestCreateTensorFromParams::test_empty, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardParameter::test_shard_parameter, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardParameter::test_shard_parameter_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardTensor::test_shard_tensor, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardTensor::test_shard_tensor_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardTensor::test_shard_tensor_with_empty_shard, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestModuleHookApi::test_collect_local_shard, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestModuleHookApi::test_reshard_output, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestLocalTensor::test_local_tensor, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestLocalTensor::test_local_tensor_error, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_cleanup, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_complete_world_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_like, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_full, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_ones, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_rand, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_zeros, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_gather_even, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_gather_uneven, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_insufficient_sharding_dims, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_invalid_pg_rpc_ranks, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_invalid_sharding, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_load_state_dict_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_multiple_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_partial_world_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_sharded_tensor_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_sharded_tensor_sizes, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_sharding_columns, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_state_dict, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_state_dict_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_state_dict_no_sharded_tensors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_create_sharded_tensor_with_ones, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_gather_even, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_gather_uneven, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_grid_sharding, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_multiple_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_partial_world_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_device, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_to_cpu, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_to_cuda, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_to_test, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_uneven_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_with_rpc_names, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalTensor::test_init_from_local_tensor, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalTensor::test_init_from_local_tensor_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_and_global_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_and_global_metadata_invalid_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_pin_memory, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_property_cross_ranks, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_shards_gaps, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_shards_overlap, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_st_base_init_from_local_shards_and_global_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorCustomOps::test_custom_op, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorCustomOps::test_custom_op_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorCustomOps::test_custom_op_override, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardMetadata::test_create_shard_with_no_placement, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardMetadata::test_shard_metadata_init, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestCreateTensorNoProcessGroupMode::test_init_from_local_shards_and_global_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestCreateTensorNoProcessGroupMode::test_non_contiguous_local_shards 2023-11-14T06:50:52.9329710Z 2023-11-14T06:50:52.9330253Z Running distributed/test_c10d_pypg 1/1 ... [2023-11-14 06:50:52.922630] 2023-11-14T06:50:52.9332532Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:50:52.922955] 2023-11-14T06:53:01.9420348Z 2023-11-14T06:53:01.9423261Z distributed/test_c10d_pypg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_pypg_1.1_084eafc5ff4ad43f_.log 2023-11-14T06:53:01.9453124Z Running 42 items in this shard: test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_dataclass_output, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_dataclass_output_unused_param, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_invoke_work_object, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_with_pypg, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_with_pypg_with_grad_views, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_invalid_powerSGD_state, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_sync_batch_norm_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_dataclass_output, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_dataclass_output_unused_param, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_invoke_work_object, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_with_pypg, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_with_pypg_with_grad_views, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_invalid_powerSGD_state, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_sync_batch_norm_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_sync_batch_norm_only_empty_input 2023-11-14T06:53:01.9481154Z 2023-11-14T06:53:01.9481850Z Running distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 ... [2023-11-14 06:53:01.942326] 2023-11-14T06:53:01.9484372Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_dtensor_state_dict.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:53:01.942763] 2023-11-14T06:54:18.0702411Z 2023-11-14T06:54:18.0707216Z distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_dtensor_state_dict_1.1_e145425cb1b6fbc9_.log 2023-11-14T06:54:18.0724414Z Running 14 items in this shard: test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_False_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_False_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_True_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_True_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_False_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_False_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_True_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_True_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_False_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_False_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_True_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_True_is_even_sharded_model_True, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_fsdp_init_with_device_mesh_is_even_sharded_model_False, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensor::test_fsdp_init_with_device_mesh_is_even_sharded_model_True 2023-11-14T06:54:18.0740339Z 2023-11-14T06:54:18.0740847Z Running distributed/test_device_mesh 1/1 ... [2023-11-14 06:54:18.070391] 2023-11-14T06:54:18.0743357Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_device_mesh.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:54:18.070689] 2023-11-14T06:55:24.7172800Z 2023-11-14T06:55:24.7174811Z distributed/test_device_mesh 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_device_mesh_1.1_94a3b8ebbf7662e6_.log 2023-11-14T06:55:24.7201053Z Running 25 items in this shard: test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_2d, test/distributed/test_device_mesh.py::DeviceMeshTest::test_fake_pg_device_mesh, test/distributed/test_device_mesh.py::DeviceMeshTest::test_init_process_group, test/distributed/test_device_mesh.py::DeviceMeshTest::test_lazy_init_device_mesh, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_nd, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_init_device_mesh, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_duplicate_mesh_dim_names, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_mesh_shape_mesh_dim_names_mismatch, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_parent_mesh, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_parent_mesh_dim_exist, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_parent_mesh_dim_not_exist, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_invalid_mesh_dim_name, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_mesh_dim_less_than_2, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_no_mesh_dim_found, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_gather_uneven, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_to_all_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_to_all_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_uneven, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_uneven 2023-11-14T06:55:24.7229982Z 2023-11-14T06:55:24.7231291Z Running distributed/fsdp/test_fsdp_freezing_weights 1/1 ... [2023-11-14 06:55:24.717412] 2023-11-14T06:55:24.7236657Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_freezing_weights.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:55:24.717714] 2023-11-14T06:56:14.8424035Z 2023-11-14T06:56:14.8429008Z distributed/fsdp/test_fsdp_freezing_weights 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_freezing_weights_1.1_8fb71272051649e6_.log 2023-11-14T06:56:14.8440481Z Running 8 items in this shard: test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True 2023-11-14T06:56:14.8450236Z 2023-11-14T06:56:14.8450767Z Running distributed/fsdp/test_fsdp_meta 1/1 ... [2023-11-14 06:56:14.842563] 2023-11-14T06:56:14.8453142Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:56:14.842877] 2023-11-14T06:57:03.3082812Z 2023-11-14T06:57:03.3085019Z distributed/fsdp/test_fsdp_meta 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_meta_1.1_86a1d625f103eca5_.log 2023-11-14T06:57:03.3097850Z Running 15 items in this shard: test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_bad_arg_meta, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_bad_arg_torchdistx, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_meta_device_with_mixed_precision, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_default_init_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_default_init_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_reset_params_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_reset_params_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_default_init_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_default_init_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_init_fn_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_init_fn_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_meta_device_default_init, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_meta_device_reset_params, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_torchdistX_default_init, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_torchdistX_init_fn 2023-11-14T06:57:03.3109684Z 2023-11-14T06:57:03.3110250Z Running distributed/_tensor/test_common_rules 1/1 ... [2023-11-14 06:57:03.308327] 2023-11-14T06:57:03.3112678Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_common_rules.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:57:03.308626] 2023-11-14T06:57:47.6734894Z 2023-11-14T06:57:47.6737558Z distributed/_tensor/test_common_rules 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_common_rules_1.1_aa449c2135cd6371_.log 2023-11-14T06:57:47.6745740Z Running 10 items in this shard: test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_basic_propagation, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_errors, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_linearity, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_merge_sharding, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_pointwise_propagation, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_enforce_sharding_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_broadcasting, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_suggestion 2023-11-14T06:57:47.6756757Z 2023-11-14T06:57:47.6758370Z Running distributed/fsdp/test_fsdp_clip_grad_norm 1/1 ... [2023-11-14 06:57:47.673631] 2023-11-14T06:57:47.6761080Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:57:47.673985] 2023-11-14T06:58:29.7702629Z 2023-11-14T06:58:29.7705491Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_clip_grad_norm_1.1_72737f60ebe253ba_.log 2023-11-14T06:58:29.7709352Z Running 4 items in this shard: test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNorm::test_ddp_parity, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNorm::test_low_precision_grads, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNorm::test_no_gradients, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNorm::test_non_root 2023-11-14T06:58:29.7712283Z 2023-11-14T06:58:29.7713397Z Running distributed/algorithms/test_join 1/1 ... [2023-11-14 06:58:29.771021] 2023-11-14T06:58:29.7718456Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/test_join.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:58:29.771349] 2023-11-14T06:59:03.6101058Z 2023-11-14T06:59:03.6106048Z distributed/algorithms/test_join 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.test_join_1.1_8c424c3203d40731_.log 2023-11-14T06:59:03.6112368Z Running 9 items in this shard: test/distributed/algorithms/test_join.py::TestJoin::test_join_kwargs, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinable_disable, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinables, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinables_throw, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_disable, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_main_hooks, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_post_hooks, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_throw 2023-11-14T06:59:03.6117612Z 2023-11-14T06:59:03.6118174Z Running distributed/test_c10d_spawn_gloo 1/1 ... [2023-11-14 06:59:03.610383] 2023-11-14T06:59:03.6120619Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '--shard-id=0', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 06:59:03.610746] 2023-11-14T07:00:12.0539094Z 2023-11-14T07:00:12.0541645Z distributed/test_c10d_spawn_gloo 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_gloo_1.1_2c51ba1f9304c00e_.log 2023-11-14T07:00:12.0545098Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cpu 2023-11-14T07:00:12.0546974Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cuda 2023-11-14T07:00:12.0549364Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_rnn 2023-11-14T07:00:12.0552049Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather 2023-11-14T07:00:12.0553658Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all 2023-11-14T07:00:12.0555920Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all_single 2023-11-14T07:00:12.0557592Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_allreduce 2023-11-14T07:00:12.0559194Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_broadcast 2023-11-14T07:00:12.0560745Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_gather 2023-11-14T07:00:12.0562321Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_reduce 2023-11-14T07:00:12.0563885Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_scatter 2023-11-14T07:00:12.0564765Z 2023-11-14T07:00:12.0565351Z Running distributed/_tensor/test_random_ops 1/1 ... [2023-11-14 07:00:12.054324] 2023-11-14T07:00:12.0567709Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_random_ops.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:00:12.054754] 2023-11-14T07:00:42.3847496Z 2023-11-14T07:00:42.3849499Z distributed/_tensor/test_random_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_random_ops_1.1_0d06beba3c43e61a_.log 2023-11-14T07:00:42.3855508Z Running 7 items in this shard: test/distributed/_tensor/test_random_ops.py::DistTensorRandomInitTest::test_init_ops, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_dropout_1d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_rand_1d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_uniform_2d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_manual_seed, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_meta_tensor_init, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_rng_tracker_init 2023-11-14T07:00:42.3859805Z 2023-11-14T07:00:42.3860506Z Running distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 ... [2023-11-14 07:00:42.384915] 2023-11-14T07:00:42.3863127Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:00:42.385218] 2023-11-14T07:01:11.5161106Z 2023-11-14T07:01:11.5163421Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks_1.1_55881abcd43aafde_.log 2023-11-14T07:01:11.5170797Z Running 6 items in this shard: test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_allreduce_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_fp16compress_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_noop_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_quantize_per_channel_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_quantize_per_tensor_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_is_last_hook 2023-11-14T07:01:11.5188922Z 2023-11-14T07:01:11.5189553Z Running distributed/fsdp/test_fsdp_fine_tune 1/1 ... [2023-11-14 07:01:11.516320] 2023-11-14T07:01:11.5192013Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fine_tune.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:01:11.516664] 2023-11-14T07:01:39.7882974Z 2023-11-14T07:01:39.7885057Z distributed/fsdp/test_fsdp_fine_tune 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_fine_tune_1.1_d0327bc14d3a0472_.log 2023-11-14T07:01:39.7889695Z Running 3 items in this shard: test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTune::test_backward_reshard_hooks, test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTune::test_hooks_multi_traversal, test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTune::test_parity_with_ddp 2023-11-14T07:01:39.7892237Z 2023-11-14T07:01:39.7892834Z Running distributed/elastic/multiprocessing/api_test 1/1 ... [2023-11-14 07:01:39.788568] 2023-11-14T07:01:39.7895443Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:01:39.788892] 2023-11-14T07:02:05.7108507Z 2023-11-14T07:02:05.7110787Z distributed/elastic/multiprocessing/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.multiprocessing.api_test_1.1_6c84287949be6a60_.log 2023-11-14T07:02:05.7138940Z Running 38 items in this shard: test/distributed/elastic/multiprocessing/api_test.py::RunProcResultsTest::test_get_failures, test/distributed/elastic/multiprocessing/api_test.py::RunProcResultsTest::test_is_failed, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_str_bad_input, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_value, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_value_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_args_env_len_mismatch, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_binary_exit, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_binary_incorrect_entrypoint, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_binary_raises, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_function_large_ret_val, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_function_raise, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_function_with_tensor, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_invalid_log_dir, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_multiprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_multiprocessing_context_poll_raises_exception, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_pcontext_wait, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_subprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_to_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_validate_full_rank, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesTest::test_void_function, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_args_env_len_mismatch, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_binary, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_binary_exit, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_binary_incorrect_entrypoint, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_binary_raises, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_binary_redirect_and_tee, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_function, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_function_large_ret_val, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_function_raise, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_function_with_tensor, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_invalid_log_dir, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_multiprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_multiprocessing_context_poll_raises_exception, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_pcontext_wait, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_subprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_to_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_validate_full_rank, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListTest::test_void_function 2023-11-14T07:02:05.7165057Z 2023-11-14T07:02:05.7165702Z Running distributed/checkpoint/test_hsdp_checkpoint 1/1 ... [2023-11-14 07:02:05.711226] 2023-11-14T07:02:05.7168237Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_hsdp_checkpoint.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:02:05.711606] 2023-11-14T07:02:28.0719957Z 2023-11-14T07:02:28.0725064Z distributed/checkpoint/test_hsdp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_hsdp_checkpoint_1.1_312759a3411eb574_.log 2023-11-14T07:02:28.0730128Z Running 4 items in this shard: test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_checkpoint_is_even_sharded_model_False, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_checkpoint_is_even_sharded_model_True, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_fsdp_checkpoint_conversion_is_even_sharded_model_False, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_fsdp_checkpoint_conversion_is_even_sharded_model_True 2023-11-14T07:02:28.0735042Z 2023-11-14T07:02:28.0736251Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 ... [2023-11-14 07:02:28.072564] 2023-11-14T07:02:28.0741463Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:02:28.072983] 2023-11-14T07:02:46.7832639Z 2023-11-14T07:02:46.7836119Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_tensor_ops_1.1_1b541a8f718ca8ed_.log 2023-11-14T07:02:46.7841120Z Running 5 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_clone, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_deep_copy, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_detach, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_inplace_copy, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_set_requires_grad 2023-11-14T07:02:46.7844428Z 2023-11-14T07:02:46.7845223Z Running distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 1/1 ... [2023-11-14 07:02:46.783405] 2023-11-14T07:02:46.7848059Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:02:46.783756] 2023-11-14T07:03:03.9334092Z 2023-11-14T07:03:03.9336736Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_optim_checkpoint_1.1_d1f070e12e517084_.log 2023-11-14T07:03:03.9340735Z Running 2 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint.py::TestOptimStateCheckpointing::test_optim_state_dict_save_load, test/distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint.py::TestOptimStateCheckpointing::test_optim_state_dict_submodule_fully_shard 2023-11-14T07:03:03.9342886Z 2023-11-14T07:03:03.9343582Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 ... [2023-11-14 07:03:03.933879] 2023-11-14T07:03:03.9346471Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:03:03.934216] 2023-11-14T07:03:19.6870692Z 2023-11-14T07:03:19.6873462Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_binary_cmp_1.1_7801904f71b0ea6b_.log 2023-11-14T07:03:19.6878701Z Running 4 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_allclose, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_allclose_tensor_specs, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_equal, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_equal_tensor_specs 2023-11-14T07:03:19.6881928Z 2023-11-14T07:03:19.6882494Z Running distributed/fsdp/test_fsdp_input 1/1 ... [2023-11-14 07:03:19.687255] 2023-11-14T07:03:19.6884853Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:03:19.687628] 2023-11-14T07:03:34.8813040Z 2023-11-14T07:03:34.8815072Z distributed/fsdp/test_fsdp_input 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_input_1.1_e27646c14da77c3d_.log 2023-11-14T07:03:34.8818020Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_input.py::TestInput::test_input_type_dict, test/distributed/fsdp/test_fsdp_input.py::TestInput::test_input_type_list 2023-11-14T07:03:34.8819352Z 2023-11-14T07:03:34.8820015Z Running distributed/_shard/sharding_plan/test_sharding_plan 1/1 ... [2023-11-14 07:03:34.881482] 2023-11-14T07:03:34.8822629Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:03:34.881821] 2023-11-14T07:03:47.5772266Z 2023-11-14T07:03:47.5774629Z distributed/_shard/sharding_plan/test_sharding_plan 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharding_plan.test_sharding_plan_1.1_3b1e74365e07fbc2_.log 2023-11-14T07:03:47.5778485Z Running 3 items in this shard: test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_custom_sharding_planner, test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_shard_module_sub_process_group, test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_sharding_plan_errors 2023-11-14T07:03:47.5780865Z 2023-11-14T07:03:47.5781412Z Running distributed/_tensor/test_view_ops 1/1 ... [2023-11-14 07:03:47.577459] 2023-11-14T07:03:47.5783736Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_view_ops.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:03:47.577816] 2023-11-14T07:04:00.0220427Z 2023-11-14T07:04:00.0222450Z distributed/_tensor/test_view_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_view_ops_1.1_94bd4b23694e9b4b_.log 2023-11-14T07:04:00.0224848Z Running 2 items in this shard: test/distributed/_tensor/test_view_ops.py::TestViewOps::test_view_groups, test/distributed/_tensor/test_view_ops.py::TestViewOps::test_view_ops 2023-11-14T07:04:00.0226088Z 2023-11-14T07:04:00.0226748Z Running distributed/_shard/sharded_tensor/ops/test_embedding 1/1 ... [2023-11-14 07:04:00.022271] 2023-11-14T07:04:00.0231496Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:04:00.022644] 2023-11-14T07:04:09.6109772Z 2023-11-14T07:04:09.6112220Z distributed/_shard/sharded_tensor/ops/test_embedding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_embedding_1.1_65c74ec125113316_.log 2023-11-14T07:04:09.6116266Z Running 2 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_embedding.py::TestShardedEmbedding::test_sharded_embedding_colwise, test/distributed/_shard/sharded_tensor/ops/test_embedding.py::TestShardedEmbedding::test_sharded_embedding_rowwise 2023-11-14T07:04:09.6118361Z 2023-11-14T07:04:09.6119373Z Running distributed/_composable/fully_shard/test_fully_shard_util 1/1 ... [2023-11-14 07:04:09.611203] 2023-11-14T07:04:09.6122051Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_util.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:04:09.611579] 2023-11-14T07:04:20.3483858Z 2023-11-14T07:04:20.3486232Z distributed/_composable/fully_shard/test_fully_shard_util 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_util_1.1_8f74ace374ca9490_.log 2023-11-14T07:04:20.3489661Z Running 1 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_util.py::TestUtils::test_get_sharded_module_tree_with_module_name_to_fqns 2023-11-14T07:04:20.3490861Z 2023-11-14T07:04:20.3491486Z Running distributed/fsdp/test_fsdp_multiple_forward 1/1 ... [2023-11-14 07:04:20.348652] 2023-11-14T07:04:20.3495277Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:04:20.349028] 2023-11-14T07:04:30.5341233Z 2023-11-14T07:04:30.5343628Z distributed/fsdp/test_fsdp_multiple_forward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_multiple_forward_1.1_e9b7c02a30988ec2_.log 2023-11-14T07:04:30.5345862Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_multiple_forward.py::TestMultiForward::test_multi_forward 2023-11-14T07:04:30.5346769Z 2023-11-14T07:04:30.5347373Z Running distributed/fsdp/test_fsdp_uneven 1/1 ... [2023-11-14 07:04:30.534345] 2023-11-14T07:04:30.5351924Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:04:30.534710] 2023-11-14T07:04:40.7199195Z 2023-11-14T07:04:40.7201288Z distributed/fsdp/test_fsdp_uneven 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_uneven_1.1_9ad6e5a414d5ca0c_.log 2023-11-14T07:04:40.7204143Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShard::test_one_iteration 2023-11-14T07:04:40.7204994Z 2023-11-14T07:04:40.7205611Z Running distributed/pipeline/sync/test_balance 1/1 ... [2023-11-14 07:04:40.720107] 2023-11-14T07:04:40.7209120Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_balance.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:04:40.720446] 2023-11-14T07:04:48.2993142Z 2023-11-14T07:04:48.2995735Z distributed/pipeline/sync/test_balance 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_balance_1.1_64de487d3ad50265_.log 2023-11-14T07:04:48.3006877Z Running 18 items in this shard: test/distributed/pipeline/sync/test_balance.py::test_blockpartition, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale, test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu], test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda], test/distributed/pipeline/sync/test_balance.py::test_not_training, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple, test/distributed/pipeline/sync/test_balance.py::test_already_has_grad 2023-11-14T07:04:48.3016677Z 2023-11-14T07:04:48.3017346Z Running distributed/tensor/parallel/test_ddp_2d_parallel 1/1 ... [2023-11-14 07:04:48.299538] 2023-11-14T07:04:48.3020169Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_ddp_2d_parallel.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:04:48.299838] 2023-11-14T07:04:56.3305085Z 2023-11-14T07:04:56.3307371Z distributed/tensor/parallel/test_ddp_2d_parallel 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_ddp_2d_parallel_1.1_6046b83c77e94463_.log 2023-11-14T07:04:56.3309992Z Running 1 items in this shard: test/distributed/tensor/parallel/test_ddp_2d_parallel.py::Test2dParallelIntegration::test_2d_ddp_integration_functionality 2023-11-14T07:04:56.3311165Z 2023-11-14T07:04:56.3311806Z Running distributed/tensor/parallel/test_tp_random_state 1/1 ... [2023-11-14 07:04:56.330633] 2023-11-14T07:04:56.3314442Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_random_state.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:04:56.330968] 2023-11-14T07:05:03.0596828Z 2023-11-14T07:05:03.0599157Z distributed/tensor/parallel/test_tp_random_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_random_state_1.1_017637baece7d195_.log 2023-11-14T07:05:03.0601752Z Running 1 items in this shard: test/distributed/tensor/parallel/test_tp_random_state.py::TensorParallelRandomStateTests::test_model_init 2023-11-14T07:05:03.0602774Z 2023-11-14T07:05:03.0603423Z Running distributed/pipeline/sync/skip/test_gpipe 1/1 ... [2023-11-14 07:05:03.059844] 2023-11-14T07:05:03.0605940Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_gpipe.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:05:03.060152] 2023-11-14T07:05:07.2838165Z 2023-11-14T07:05:07.2840617Z distributed/pipeline/sync/skip/test_gpipe 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_gpipe_1.1_6d4275244b56109a_.log 2023-11-14T07:05:07.2849513Z Running 13 items in this shard: test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip 2023-11-14T07:05:07.2856761Z 2023-11-14T07:05:07.2857244Z Running distributed/test_fake_pg 1/1 ... [2023-11-14 07:05:07.284037] 2023-11-14T07:05:07.2859505Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_fake_pg.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:05:07.284353] 2023-11-14T07:05:11.5075959Z 2023-11-14T07:05:11.5077894Z distributed/test_fake_pg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_fake_pg_1.1_7418aecdbb8be150_.log 2023-11-14T07:05:11.5084723Z Running 13 items in this shard: test/distributed/test_fake_pg.py::TestFakePG::test_all_reduce, test/distributed/test_fake_pg.py::TestFakePG::test_allgather, test/distributed/test_fake_pg.py::TestFakePG::test_alltoall, test/distributed/test_fake_pg.py::TestFakePG::test_alltoall_base, test/distributed/test_fake_pg.py::TestFakePG::test_broadcast, test/distributed/test_fake_pg.py::TestFakePG::test_construct_fsdp, test/distributed/test_fake_pg.py::TestFakePG::test_fake_pg_tracing, test/distributed/test_fake_pg.py::TestFakePG::test_fsdp_fake_e2e, test/distributed/test_fake_pg.py::TestFakePG::test_fsdp_tp_fake_e2e, test/distributed/test_fake_pg.py::TestFakePG::test_recv, test/distributed/test_fake_pg.py::TestFakePG::test_reduce_scatter, test/distributed/test_fake_pg.py::TestFakePG::test_scatter, test/distributed/test_fake_pg.py::TestFakePG::test_send 2023-11-14T07:05:11.5090148Z 2023-11-14T07:05:11.5090769Z Running distributed/pipeline/sync/test_checkpoint 1/1 ... [2023-11-14 07:05:11.507887] 2023-11-14T07:05:11.5093301Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_checkpoint.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:05:11.508188] 2023-11-14T07:05:14.5289093Z 2023-11-14T07:05:14.5291387Z distributed/pipeline/sync/test_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_checkpoint_1.1_45476bd78be5513e_.log 2023-11-14T07:05:14.5298760Z Running 9 items in this shard: test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda], test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad, test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter, test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda], test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing, test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint, test/distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output 2023-11-14T07:05:14.5304418Z 2023-11-14T07:05:14.5304891Z Running distributed/test_nccl 1/1 ... [2023-11-14 07:05:14.529118] 2023-11-14T07:05:14.5307106Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_nccl.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:05:14.529486] 2023-11-14T07:05:17.6507518Z 2023-11-14T07:05:17.6509384Z distributed/test_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_nccl_1.1_13e977476fbb9864_.log 2023-11-14T07:05:17.6517312Z Running 12 items in this shard: test/distributed/test_nccl.py::TestNCCLCUDA::test_all_gather_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_gather_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_reduce_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_reduce_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_broadcast_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_broadcast_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_collective_errors_cuda, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_scatter_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_scatter_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_unique_id_cuda 2023-11-14T07:05:17.6523070Z 2023-11-14T07:05:17.6523949Z Running distributed/fsdp/test_utils 1/1 ... [2023-11-14 07:05:17.650958] 2023-11-14T07:05:17.6526596Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:05:17.651262] 2023-11-14T07:05:20.3214257Z 2023-11-14T07:05:20.3216225Z distributed/fsdp/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_utils_1.1_a40297384aa577f7_.log 2023-11-14T07:05:20.3221474Z Running 5 items in this shard: test/distributed/fsdp/test_utils.py::TestUtils::test_apply_to_tensors_cpu_cuda, test/distributed/fsdp/test_utils.py::TestUtils::test_apply_to_tensors_devices_['cpu'], test/distributed/fsdp/test_utils.py::TestUtils::test_apply_to_tensors_devices_['cuda'], test/distributed/fsdp/test_utils.py::TestUtils::test_packed_sequence, test/distributed/fsdp/test_utils.py::TestUtils::test_replace_by_prefix 2023-11-14T07:05:20.3224266Z 2023-11-14T07:05:20.3224920Z Running distributed/elastic/utils/logging_test 1/1 ... [2023-11-14 07:05:20.321573] 2023-11-14T07:05:20.3227367Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:05:20.321882] 2023-11-14T07:05:22.9422286Z 2023-11-14T07:05:22.9424617Z distributed/elastic/utils/logging_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.logging_test_1.1_4c9426a6e1d7ccb7_.log 2023-11-14T07:05:22.9427837Z Running 2 items in this shard: test/distributed/elastic/utils/logging_test.py::LoggingTest::test_derive_module_name, test/distributed/elastic/utils/logging_test.py::LoggingTest::test_logger_name 2023-11-14T07:05:22.9429540Z 2023-11-14T07:05:22.9430139Z Running distributed/checkpoint/test_nested_dict 1/1 ... [2023-11-14 07:05:22.942356] 2023-11-14T07:05:22.9432629Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_nested_dict.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:05:22.942652] 2023-11-14T07:05:27.9673515Z 2023-11-14T07:05:27.9676085Z distributed/checkpoint/test_nested_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_nested_dict_1.1_f947723bd8dc60a1_.log 2023-11-14T07:05:27.9679090Z Running 2 items in this shard: test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_flattening_round_trip, test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_mapping 2023-11-14T07:05:27.9681019Z 2023-11-14T07:05:27.9681697Z Running distributed/_tensor/test_xla_integration 1/1 ... [2023-11-14 07:05:27.967567] 2023-11-14T07:05:27.9684277Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_xla_integration.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:05:27.967875] 2023-11-14T07:05:32.2412461Z 2023-11-14T07:05:32.2414866Z distributed/_tensor/test_xla_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_xla_integration_1.1_dba532b551f362b7_.log 2023-11-14T07:05:32.2419681Z Running 3 items in this shard: test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_replicate, test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_shard, test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_2d 2023-11-14T07:05:32.2422110Z 2023-11-14T07:05:32.2423140Z Running distributed/checkpoint/test_dedup_tensors 1/1 ... [2023-11-14 07:05:32.241376] 2023-11-14T07:05:32.2425655Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dedup_tensors.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:05:32.241680] 2023-11-14T07:05:37.0657395Z 2023-11-14T07:05:37.0659486Z distributed/checkpoint/test_dedup_tensors 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_dadb9471d9d02214_.log 2023-11-14T07:05:37.0661765Z Running 1 items in this shard: test/distributed/checkpoint/test_dedup_tensors.py::TestDedupTensor::test_dedup_shards 2023-11-14T07:05:37.0662639Z 2023-11-14T07:05:37.0663219Z Running distributed/pipeline/sync/test_dependency 1/1 ... [2023-11-14 07:05:37.065851] 2023-11-14T07:05:37.0665830Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/test_dependency.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:05:37.066156] 2023-11-14T07:05:39.4356104Z 2023-11-14T07:05:39.4358295Z distributed/pipeline/sync/test_dependency 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.test_dependency_1.1_8b89f2ec3d51eaf4_.log 2023-11-14T07:05:39.4363760Z Running 6 items in this shard: test/distributed/pipeline/sync/test_dependency.py::test_fork_join, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_leak, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad 2023-11-14T07:05:39.4367124Z 2023-11-14T07:05:39.4367742Z Running distributed/pipeline/sync/skip/test_api 1/1 ... [2023-11-14 07:05:39.435885] 2023-11-14T07:05:39.4370230Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipeline/sync/skip/test_api.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:05:39.436194] 2023-11-14T07:05:41.6552402Z 2023-11-14T07:05:41.6554772Z distributed/pipeline/sync/skip/test_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipeline.sync.skip.test_api_1.1_c5a77a4eb5146f16_.log 2023-11-14T07:05:41.6559210Z Running 3 items in this shard: test/distributed/pipeline/sync/skip/test_api.py::test_namespace_difference, test/distributed/pipeline/sync/skip/test_api.py::test_namespace_copy, test/distributed/pipeline/sync/skip/test_api.py::test_skippable_repr 2023-11-14T07:05:41.6561109Z 2023-11-14T07:05:41.6561614Z Running distributed/_shard/test_sharder 1/1 ... [2023-11-14 07:05:41.655501] 2023-11-14T07:05:41.6563955Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:05:41.655813] 2023-11-14T07:05:44.2961683Z 2023-11-14T07:05:44.2963683Z distributed/_shard/test_sharder 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.test_sharder_1.1_dc75e1c9c05b8732_.log 2023-11-14T07:05:44.2965110Z 2023-11-14T07:05:44.2965581Z Running distributed/_spmd/test_transformation 1/1 ... [2023-11-14 07:05:44.296239] 2023-11-14T07:05:44.2970406Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_spmd/test_transformation.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:05:44.296546] 2023-11-14T07:05:48.7728897Z 2023-11-14T07:05:48.7731209Z distributed/_spmd/test_transformation 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._spmd.test_transformation_1.1_f540389596243d5b_.log 2023-11-14T07:05:48.7732988Z 2023-11-14T07:05:48.7733503Z Running distributed/checkpoint/e2e/test_pipeline 1/1 ... [2023-11-14 07:05:48.772950] 2023-11-14T07:05:48.7736778Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_pipeline.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:05:48.773253] 2023-11-14T07:05:52.8991679Z 2023-11-14T07:05:52.8994046Z distributed/checkpoint/e2e/test_pipeline 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_pipeline_1.1_01184b76568bd70e_.log 2023-11-14T07:05:52.8997317Z 2023-11-14T07:05:52.8998230Z Running distributed/optim/test_named_optimizer 1/1 ... [2023-11-14 07:05:52.899249] 2023-11-14T07:05:52.9000666Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_named_optimizer.py', '--shard-id=0', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2023-11-14 07:05:52.899605] 2023-11-14T07:05:57.4197976Z 2023-11-14T07:05:57.4200379Z distributed/optim/test_named_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_named_optimizer_1.1_e10076f7486eba1c_.log 2023-11-14T07:05:57.4201828Z 2023-11-14T07:05:57.4204799Z Starting test batch 'unlikely_relevance' 7829.878350496292 seconds after initiating testing 2023-11-14T07:05:57.4205796Z With sharding, this batch will run 0 tests 2023-11-14T07:05:57.4206689Z Starting test batch 'none_relevance' 7829.878408670425 seconds after initiating testing 2023-11-14T07:05:57.4207582Z With sharding, this batch will run 0 tests 2023-11-14T07:05:58.4862968Z /var/lib/jenkins/workspace/tools/stats/upload_metrics.py:164: UserWarning: Error uploading metric td_experiment_1 to DynamoDB: Unsupported type "" for value "distributed/test_distributed_spawn" 2023-11-14T07:05:58.4865410Z warn(f"Error uploading metric {metric_name} to DynamoDB: {e}") 2023-11-14T07:05:58.5459084Z Emiting td_test_failure_stats 2023-11-14T07:05:59.9934765Z distributed/test_distributed_spawn 2/2 failed! 2023-11-14T07:06:00.3287575Z 2023-11-14T07:06:00.3288576Z real 130m35.274s 2023-11-14T07:06:00.3289129Z user 199m10.499s 2023-11-14T07:06:00.3289443Z sys 25m46.783s 2023-11-14T07:06:00.3348193Z ##[error]Process completed with exit code 1. 2023-11-14T07:06:00.3448569Z Prepare all required actions 2023-11-14T07:06:00.3449172Z Getting action download info 2023-11-14T07:06:00.5065551Z ##[group]Run ./.github/actions/pytest-cache-upload 2023-11-14T07:06:00.5066156Z with: 2023-11-14T07:06:00.5066536Z cache_dir: .pytest_cache 2023-11-14T07:06:00.5066966Z shard: 2 2023-11-14T07:06:00.5067578Z job_identifier: pull_linux-focal-cuda11.8-py3.10-gcc9_test_distributed 2023-11-14T07:06:00.5068311Z env: 2023-11-14T07:06:00.5068651Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:06:00.5069261Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:06:00.5070214Z DOCKER_CONTAINER_ID: 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:00.5071026Z ##[endgroup] 2023-11-14T07:06:00.5127316Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2023-11-14T07:06:00.5128041Z with: 2023-11-14T07:06:00.5128376Z shell: bash 2023-11-14T07:06:00.5128772Z timeout_minutes: 5 2023-11-14T07:06:00.5129302Z max_attempts: 5 2023-11-14T07:06:00.5129702Z retry_wait_seconds: 30 2023-11-14T07:06:00.5130290Z command: set -eu python3 -m pip install boto3==1.19.12 2023-11-14T07:06:00.5130950Z polling_interval_seconds: 1 2023-11-14T07:06:00.5131412Z warning_on_retry: true 2023-11-14T07:06:00.5131871Z continue_on_error: false 2023-11-14T07:06:00.5132314Z env: 2023-11-14T07:06:00.5132649Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:06:00.5133241Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:06:00.5134208Z DOCKER_CONTAINER_ID: 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:00.5135017Z ##[endgroup] 2023-11-14T07:06:00.8827151Z Defaulting to user installation because normal site-packages is not writeable 2023-11-14T07:06:00.9041011Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12) 2023-11-14T07:06:00.9111675Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2) 2023-11-14T07:06:00.9149090Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0) 2023-11-14T07:06:00.9166361Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12) 2023-11-14T07:06:00.9226633Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.18) 2023-11-14T07:06:00.9476647Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2) 2023-11-14T07:06:00.9502263Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0) 2023-11-14T07:06:01.5882423Z Command completed after 1 attempt(s). 2023-11-14T07:06:01.5946707Z ##[group]Run python3 .github/scripts/pytest_cache.py \ 2023-11-14T07:06:01.5947476Z python3 .github/scripts/pytest_cache.py \ 2023-11-14T07:06:01.5948073Z  --upload \ 2023-11-14T07:06:01.5948622Z  --cache_dir $GITHUB_WORKSPACE/$CACHE_DIR \ 2023-11-14T07:06:01.5949281Z  --pr_identifier $GITHUB_REF \ 2023-11-14T07:06:01.5949877Z  --job_identifier $JOB_IDENTIFIER \ 2023-11-14T07:06:01.5950466Z  --shard $SHARD \ 2023-11-14T07:06:01.5951199Z  --repo $REPO \ 2023-11-14T07:06:01.5951658Z  --temp_dir $RUNNER_TEMP \ 2023-11-14T07:06:01.5961395Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T07:06:01.5962050Z env: 2023-11-14T07:06:01.5962422Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:06:01.5962985Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:06:01.5964166Z DOCKER_CONTAINER_ID: 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:01.5965050Z CACHE_DIR: .pytest_cache 2023-11-14T07:06:01.5965715Z JOB_IDENTIFIER: pull_linux-focal-cuda11.8-py3.10-gcc9_test_distributed 2023-11-14T07:06:01.5966455Z SHARD: 2 2023-11-14T07:06:01.5966843Z REPO: pytorch/pytorch 2023-11-14T07:06:01.5967261Z ##[endgroup] 2023-11-14T07:06:02.0175719Z PR identifier for `refs/pull/112098/merge` is `2e5132e7254b4b60102d80422aad461b` 2023-11-14T07:06:02.0179130Z Uploading cache with args Namespace(bucket=None, cache_dir='/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache', download=False, job_identifier='pull_linux-focal-cuda11.8-py3.10-gcc9_test_distributed', pr_identifier='refs/pull/112098/merge', repo='pytorch/pytorch', shard='2', temp_dir='/home/ec2-user/actions-runner/_work/_temp', upload=True) 2023-11-14T07:06:02.0182126Z Zipping /home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache 2023-11-14T07:06:02.0184080Z to /home/ec2-user/actions-runner/_work/_temp/zip-upload/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/2 2023-11-14T07:06:02.0186759Z Uploading /home/ec2-user/actions-runner/_work/_temp/zip-upload/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/2.zip 2023-11-14T07:06:02.0189223Z to s3://gha-artifacts/pytest_cache/pytorch/pytorch/2e5132e7254b4b60102d80422aad461b/pull_linux-focal-cuda11_8-py3_10-gcc9_test_distributed/2.zip 2023-11-14T07:06:02.0634995Z ##[group]Run cat test/**/*_toprint.log || true 2023-11-14T07:06:02.0635722Z cat test/**/*_toprint.log || true 2023-11-14T07:06:02.0645001Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T07:06:02.0645659Z env: 2023-11-14T07:06:02.0646027Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:06:02.0646592Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:06:02.0647549Z DOCKER_CONTAINER_ID: 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:02.0648411Z ##[endgroup] 2023-11-14T07:06:02.0699279Z cat: test/**/*_toprint.log: No such file or directory 2023-11-14T07:06:02.0739666Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2023-11-14T07:06:02.0740310Z kill "$MONITOR_SCRIPT_PID" 2023-11-14T07:06:02.0748506Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T07:06:02.0749149Z env: 2023-11-14T07:06:02.0749485Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:06:02.0750068Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:06:02.0751054Z DOCKER_CONTAINER_ID: 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:02.0751885Z MONITOR_SCRIPT_PID: 48430 2023-11-14T07:06:02.0752341Z ##[endgroup] 2023-11-14T07:06:02.0962774Z Prepare all required actions 2023-11-14T07:06:02.0963394Z Getting action download info 2023-11-14T07:06:02.2626702Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2023-11-14T07:06:02.4564927Z ##[group]Run ./.github/actions/upload-test-artifacts 2023-11-14T07:06:02.4565517Z with: 2023-11-14T07:06:02.4566143Z file-suffix: test-distributed-2-3-linux.8xlarge.nvidia.gpu_18651430132 2023-11-14T07:06:02.4566900Z env: 2023-11-14T07:06:02.4567236Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:06:02.4567807Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:06:02.4568753Z DOCKER_CONTAINER_ID: 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:02.4569573Z ##[endgroup] 2023-11-14T07:06:02.4612156Z ##[group]Run # Remove any previous test jsons if they exist 2023-11-14T07:06:02.4612923Z # Remove any previous test jsons if they exist 2023-11-14T07:06:02.4613641Z rm -f test-jsons-*.zip 2023-11-14T07:06:02.4614291Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2023-11-14T07:06:02.4623028Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T07:06:02.4623665Z env: 2023-11-14T07:06:02.4624026Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:06:02.4624585Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:06:02.4625524Z DOCKER_CONTAINER_ID: 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:02.4626611Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_18651430132 2023-11-14T07:06:02.4627364Z ##[endgroup] 2023-11-14T07:06:02.4767382Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2023-11-14T07:06:02.4802979Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2023-11-14T07:06:02.4803883Z adding: test/minioptest_failures_dict.json (deflated 70%) 2023-11-14T07:06:02.4811309Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2023-11-14T07:06:02.4814514Z adding: test/.pytorch-slow-tests.json (deflated 78%) 2023-11-14T07:06:02.4822378Z adding: test/.pytorch-disabled-tests.json (deflated 85%) 2023-11-14T07:06:02.4857725Z ##[group]Run # Remove any previous test reports if they exist 2023-11-14T07:06:02.4858557Z # Remove any previous test reports if they exist 2023-11-14T07:06:02.4859230Z rm -f test-reports-*.zip 2023-11-14T07:06:02.4859988Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2023-11-14T07:06:02.4868485Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T07:06:02.4869125Z env: 2023-11-14T07:06:02.4869489Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:06:02.4870053Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:06:02.4871017Z DOCKER_CONTAINER_ID: 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:02.4872112Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_18651430132 2023-11-14T07:06:02.4872885Z ##[endgroup] 2023-11-14T07:06:02.5019306Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cd7fdb9f8c0f2e06.xml (deflated 38%) 2023-11-14T07:06:02.5021458Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-060ee7303a4d54cc.xml (deflated 43%) 2023-11-14T07:06:02.5023510Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ae81878f0e5c0fea.xml (deflated 58%) 2023-11-14T07:06:02.5025563Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0166ef95eebc4632.xml (deflated 44%) 2023-11-14T07:06:02.5027659Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-53df3e04fe44f6e9.xml (deflated 37%) 2023-11-14T07:06:02.5029859Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e9791be66b9171fc.xml (deflated 37%) 2023-11-14T07:06:02.5032030Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-97873986e9d52b9e.xml (deflated 37%) 2023-11-14T07:06:02.5034079Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1cc21c616fe78de7.xml (deflated 37%) 2023-11-14T07:06:02.5036346Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cacd6e10417e5b0b.xml (deflated 37%) 2023-11-14T07:06:02.5038468Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a269d9ece814e837.xml (deflated 38%) 2023-11-14T07:06:02.5040516Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-854414c4b6fc5bdc.xml (deflated 38%) 2023-11-14T07:06:02.5042768Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2bdf51a83dac44dd.xml (deflated 38%) 2023-11-14T07:06:02.5044865Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-599410bc2da0983f.xml (deflated 42%) 2023-11-14T07:06:02.5046879Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9c2c07d63a0c02bf.xml (deflated 37%) 2023-11-14T07:06:02.5048853Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bb5d0ac60d31a8a2.xml (deflated 45%) 2023-11-14T07:06:02.5050979Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-86c8ab0ffadea037.xml (deflated 45%) 2023-11-14T07:06:02.5053035Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e548d3ee42fc247a.xml (deflated 43%) 2023-11-14T07:06:02.5055019Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d0402a9b216140fe.xml (deflated 37%) 2023-11-14T07:06:02.5057107Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ebe440cc0d0631b8.xml (deflated 43%) 2023-11-14T07:06:02.5059146Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0ec6567c65bbb214.xml (deflated 43%) 2023-11-14T07:06:02.5061177Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a3ea9a25132dc21.xml (deflated 37%) 2023-11-14T07:06:02.5063155Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-22f2b06e1dd8d16b.xml (deflated 37%) 2023-11-14T07:06:02.5065238Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a8896bae29471b19.xml (deflated 38%) 2023-11-14T07:06:02.5067296Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d2a7915fae3e6991.xml (deflated 44%) 2023-11-14T07:06:02.5069345Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c5a6432ebe35965d.xml (deflated 44%) 2023-11-14T07:06:02.5071355Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a55a79345b0bba79.xml (deflated 44%) 2023-11-14T07:06:02.5073394Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-104d495c1ea4d1b5.xml (deflated 44%) 2023-11-14T07:06:02.5075620Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6e2dcc491ec8323e.xml (deflated 44%) 2023-11-14T07:06:02.5077734Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b140d019e575a1a7.xml (deflated 44%) 2023-11-14T07:06:02.5079897Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f7faba4ae64adaec.xml (deflated 43%) 2023-11-14T07:06:02.5082066Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d84c2e1d2b9ebe32.xml (deflated 43%) 2023-11-14T07:06:02.5084107Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-265b7687360e02ae.xml (deflated 43%) 2023-11-14T07:06:02.5086124Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0cf57174fab3847c.xml (deflated 43%) 2023-11-14T07:06:02.5088141Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f085a22709cd99fb.xml (deflated 45%) 2023-11-14T07:06:02.5090349Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e75200e73864320.xml (deflated 45%) 2023-11-14T07:06:02.5092392Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6a8610e50801a0c5.xml (deflated 43%) 2023-11-14T07:06:02.5094413Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13d40d984fdd4dae.xml (deflated 38%) 2023-11-14T07:06:02.5096429Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ff70070a7a9f98c4.xml (deflated 43%) 2023-11-14T07:06:02.5098481Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf37d4e4ea91cfcb.xml (deflated 38%) 2023-11-14T07:06:02.5100461Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a392ba3d8654d8e4.xml (deflated 43%) 2023-11-14T07:06:02.5102569Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-754ff6c4964eb8ea.xml (deflated 43%) 2023-11-14T07:06:02.5104583Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-27845e0f4dcbf8d6.xml (deflated 43%) 2023-11-14T07:06:02.5106624Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-31c130fec00e4620.xml (deflated 44%) 2023-11-14T07:06:02.5108651Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5baf3b2394564603.xml (deflated 44%) 2023-11-14T07:06:02.5110701Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39cebae32b7acb22.xml (deflated 44%) 2023-11-14T07:06:02.5112711Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d8bed5af9f276392.xml (deflated 38%) 2023-11-14T07:06:02.5114990Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e9cde977f38c267c.xml (deflated 44%) 2023-11-14T07:06:02.5117068Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-33d296c0b31cbf3d.xml (deflated 48%) 2023-11-14T07:06:02.5119116Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3d2d8b9a7fa2ff36.xml (deflated 38%) 2023-11-14T07:06:02.5121134Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b9a5f523ad3cd2f.xml (deflated 44%) 2023-11-14T07:06:02.5123181Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-665efddc8107d2ab.xml (deflated 49%) 2023-11-14T07:06:02.5125203Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4162206083ff4643.xml (deflated 44%) 2023-11-14T07:06:02.5127246Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30d25214f290129c.xml (deflated 49%) 2023-11-14T07:06:02.5129384Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ffe368f555f1817.xml (deflated 44%) 2023-11-14T07:06:02.5131448Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6e85f8027f37be9b.xml (deflated 41%) 2023-11-14T07:06:02.5133457Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-12d0f845b5a2affa.xml (deflated 41%) 2023-11-14T07:06:02.5135519Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d01c42b555623ccb.xml (deflated 38%) 2023-11-14T07:06:02.5137509Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e170b8f4a2da62f6.xml (deflated 38%) 2023-11-14T07:06:02.5139758Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6765ea9928a548a2.xml (deflated 48%) 2023-11-14T07:06:02.5141796Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5d0e663ec3451b3b.xml (deflated 37%) 2023-11-14T07:06:02.5143785Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4cea489298215161.xml (deflated 38%) 2023-11-14T07:06:02.5145826Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e374052ea12fe8b6.xml (deflated 37%) 2023-11-14T07:06:02.5147858Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a683c6d45948b4b6.xml (deflated 38%) 2023-11-14T07:06:02.5149883Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a4abb4662fec6865.xml (deflated 37%) 2023-11-14T07:06:02.5151921Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-10deb8a369c31800.xml (deflated 37%) 2023-11-14T07:06:02.5153951Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-75aec6f80584cb0f.xml (deflated 37%) 2023-11-14T07:06:02.5156195Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eae23d804c0d379d.xml (deflated 37%) 2023-11-14T07:06:02.5158271Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-363cfec15856d974.xml (deflated 37%) 2023-11-14T07:06:02.5160281Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7eb80984921d04c6.xml (deflated 42%) 2023-11-14T07:06:02.5162369Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f3aaf708bd416677.xml (deflated 38%) 2023-11-14T07:06:02.5164395Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e909ba3cdf8e8d37.xml (deflated 38%) 2023-11-14T07:06:02.5166436Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-616c1741d0381fec.xml (deflated 37%) 2023-11-14T07:06:02.5168469Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b132a9c8c86ff111.xml (deflated 38%) 2023-11-14T07:06:02.5170512Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-443b5715f23cdfa6.xml (deflated 37%) 2023-11-14T07:06:02.5172494Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ef2091b9d8ba7745.xml (deflated 37%) 2023-11-14T07:06:02.5174546Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-66753cc9fadb9920.xml (deflated 38%) 2023-11-14T07:06:02.5176744Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0203a1f1011cc3b7.xml (deflated 37%) 2023-11-14T07:06:02.5178776Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e09e539e66ceb772.xml (deflated 38%) 2023-11-14T07:06:02.5180822Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9d832f8f76235b27.xml (deflated 38%) 2023-11-14T07:06:02.5182919Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca6e4e3011caab2d.xml (deflated 37%) 2023-11-14T07:06:02.5184957Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-05f5c3adfcd16dbb.xml (deflated 37%) 2023-11-14T07:06:02.5186988Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-faf6b1eb15e1fd45.xml (deflated 37%) 2023-11-14T07:06:02.5189179Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6022c00986c9edd8.xml (deflated 58%) 2023-11-14T07:06:02.5191224Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bc0ebe9cbcdbb329.xml (deflated 37%) 2023-11-14T07:06:02.5193213Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af5261a4ab5981a6.xml (deflated 37%) 2023-11-14T07:06:02.5195471Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fc289028af2863fd.xml (deflated 37%) 2023-11-14T07:06:02.5197504Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ce89b88e002b7035.xml (deflated 38%) 2023-11-14T07:06:02.5199561Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2f2725b3d4a8bd36.xml (deflated 38%) 2023-11-14T07:06:02.5201608Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-95b2658d913f574a.xml (deflated 37%) 2023-11-14T07:06:02.5203636Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a62874e9c37f50d0.xml (deflated 37%) 2023-11-14T07:06:02.5205577Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-41af72a356019678.xml (deflated 38%) 2023-11-14T07:06:02.5207670Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2751d5fa6230234f.xml (deflated 43%) 2023-11-14T07:06:02.5209710Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1ebc3f718fbc0ded.xml (deflated 38%) 2023-11-14T07:06:02.5211707Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-679e951c7a59f305.xml (deflated 43%) 2023-11-14T07:06:02.5213774Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-861dc26911198536.xml (deflated 38%) 2023-11-14T07:06:02.5215768Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b0669834809292b6.xml (deflated 38%) 2023-11-14T07:06:02.5217823Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-37d1c32837ca8622.xml (deflated 37%) 2023-11-14T07:06:02.5219878Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b0f400dd2ca8abde.xml (deflated 38%) 2023-11-14T07:06:02.5221897Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-38dd9cccee12bd52.xml (deflated 37%) 2023-11-14T07:06:02.5223927Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-412246dbabdb6c8f.xml (deflated 37%) 2023-11-14T07:06:02.5226112Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-460dccc581ff5e0a.xml (deflated 43%) 2023-11-14T07:06:02.5228215Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e53470d5177864a0.xml (deflated 43%) 2023-11-14T07:06:02.5230222Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-718ed088d6e60728.xml (deflated 43%) 2023-11-14T07:06:02.5232261Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-880e7834c448e94c.xml (deflated 37%) 2023-11-14T07:06:02.5234302Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cc85c6c043c472ad.xml (deflated 37%) 2023-11-14T07:06:02.5236670Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a878ce66f4961c80.xml (deflated 44%) 2023-11-14T07:06:02.5238699Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7cae30c23078c328.xml (deflated 44%) 2023-11-14T07:06:02.5240721Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-98e2026a09403455.xml (deflated 38%) 2023-11-14T07:06:02.5242715Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-97a5be0f585b07cf.xml (deflated 38%) 2023-11-14T07:06:02.5244814Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9ebb058ed2602e20.xml (deflated 37%) 2023-11-14T07:06:02.5246823Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e380d12fd680f67.xml (deflated 38%) 2023-11-14T07:06:02.5248796Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-581a10236a4dce09.xml (deflated 43%) 2023-11-14T07:06:02.5250867Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bb04c832a2c95b05.xml (deflated 43%) 2023-11-14T07:06:02.5252948Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-56839ac91382932c.xml (deflated 44%) 2023-11-14T07:06:02.5254949Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-508cd1e5d4ad2d23.xml (deflated 37%) 2023-11-14T07:06:02.5256968Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8c47045beee452b6.xml (deflated 37%) 2023-11-14T07:06:02.5259027Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-95193ba51aaeb167.xml (deflated 37%) 2023-11-14T07:06:02.5261052Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cb4e801b0543752f.xml (deflated 47%) 2023-11-14T07:06:02.5263094Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b73a6528e3de4a23.xml (deflated 47%) 2023-11-14T07:06:02.5265135Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-22a09558f1669e9f.xml (deflated 37%) 2023-11-14T07:06:02.5267145Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b0be683482846940.xml (deflated 43%) 2023-11-14T07:06:02.5269215Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-94e5dbb92c254cdc.xml (deflated 43%) 2023-11-14T07:06:02.5271234Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-76e618f423f9b858.xml (deflated 43%) 2023-11-14T07:06:02.5273276Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3719b6619f93ead9.xml (deflated 43%) 2023-11-14T07:06:02.5275637Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13d97a46cfbfd9d8.xml (deflated 43%) 2023-11-14T07:06:02.5277642Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fd4ad010f067cdd5.xml (deflated 43%) 2023-11-14T07:06:02.5279726Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2f36ff4e5134b133.xml (deflated 37%) 2023-11-14T07:06:02.5281795Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-263a1f44f1d2213d.xml (deflated 43%) 2023-11-14T07:06:02.5283863Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ae95adad901d1f46.xml (deflated 38%) 2023-11-14T07:06:02.5286050Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-548f62cf79151c0e.xml (deflated 43%) 2023-11-14T07:06:02.5288057Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-38db4a6d51638d61.xml (deflated 37%) 2023-11-14T07:06:02.5290090Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cb63f20eed504328.xml (deflated 43%) 2023-11-14T07:06:02.5292097Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-700d22e18e80b430.xml (deflated 43%) 2023-11-14T07:06:02.5294140Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-caf53d767cacd195.xml (deflated 38%) 2023-11-14T07:06:02.5296160Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ab3e183cf7dd1b4f.xml (deflated 37%) 2023-11-14T07:06:02.5298217Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-45c9d03c52d46da1.xml (deflated 43%) 2023-11-14T07:06:02.5300266Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4ef0c518ebd570d.xml (deflated 43%) 2023-11-14T07:06:02.5302311Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3453a5f014ab16ff.xml (deflated 45%) 2023-11-14T07:06:02.5304286Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b8f546455da76027.xml (deflated 45%) 2023-11-14T07:06:02.5306350Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7afe66e44c3e3f3e.xml (deflated 37%) 2023-11-14T07:06:02.5308406Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-916adef2b5664960.xml (deflated 37%) 2023-11-14T07:06:02.5310437Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-42ca9ba78c6d0e54.xml (deflated 45%) 2023-11-14T07:06:02.5312467Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c607c425e54df3d8.xml (deflated 45%) 2023-11-14T07:06:02.5314497Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-05a1117412c66961.xml (deflated 45%) 2023-11-14T07:06:02.5316701Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a89614209161b1b0.xml (deflated 45%) 2023-11-14T07:06:02.5318803Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2ff32d525af96b9.xml (deflated 38%) 2023-11-14T07:06:02.5320839Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-44bf8f41d4682670.xml (deflated 38%) 2023-11-14T07:06:02.5323032Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-19ffa641292bd942.xml (deflated 43%) 2023-11-14T07:06:02.5325111Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8051a28cf9907a9a.xml (deflated 58%) 2023-11-14T07:06:02.5327145Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6eec9fffaad01866.xml (deflated 44%) 2023-11-14T07:06:02.5329155Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f9d7419a2d80f58e.xml (deflated 37%) 2023-11-14T07:06:02.5331173Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1d52e339fb1ef359.xml (deflated 37%) 2023-11-14T07:06:02.5333220Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d1f959b8ca800a81.xml (deflated 37%) 2023-11-14T07:06:02.5335395Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-88404a3569dc010f.xml (deflated 37%) 2023-11-14T07:06:02.5337422Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d9a5717f485e3066.xml (deflated 37%) 2023-11-14T07:06:02.5339425Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-52af8eba4f57a84b.xml (deflated 38%) 2023-11-14T07:06:02.5341503Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a72401a7ed0d27a4.xml (deflated 38%) 2023-11-14T07:06:02.5343538Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-49d2c0abd097faf3.xml (deflated 39%) 2023-11-14T07:06:02.5345630Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1ebec79c377706a9.xml (deflated 42%) 2023-11-14T07:06:02.5347712Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e7d851626ff1967.xml (deflated 37%) 2023-11-14T07:06:02.5349719Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13e69802761e5c97.xml (deflated 46%) 2023-11-14T07:06:02.5351755Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96fc1630a3b9ed56.xml (deflated 45%) 2023-11-14T07:06:02.5353788Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7fc2273751bb6fbb.xml (deflated 43%) 2023-11-14T07:06:02.5356031Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-92f81cd92ee8259e.xml (deflated 37%) 2023-11-14T07:06:02.5358029Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-efc945bfb2734d09.xml (deflated 43%) 2023-11-14T07:06:02.5360102Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ef2fcb3a41d6c836.xml (deflated 43%) 2023-11-14T07:06:02.5362141Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2149107a06e5871e.xml (deflated 37%) 2023-11-14T07:06:02.5364164Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-63cbb3f17e8365f9.xml (deflated 37%) 2023-11-14T07:06:02.5366188Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-134bac188bb5a7e6.xml (deflated 38%) 2023-11-14T07:06:02.5368232Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fcbb70fcadebd09c.xml (deflated 44%) 2023-11-14T07:06:02.5370279Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fa2e35384bedc2d4.xml (deflated 44%) 2023-11-14T07:06:02.5372459Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-52c442bceb55e8b8.xml (deflated 44%) 2023-11-14T07:06:02.5374529Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-817bb33baf5925d9.xml (deflated 44%) 2023-11-14T07:06:02.5376552Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-79b38f94ad97a840.xml (deflated 44%) 2023-11-14T07:06:02.5378583Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-426157937c92194b.xml (deflated 44%) 2023-11-14T07:06:02.5380504Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-11248026f5a3c3e8.xml (deflated 43%) 2023-11-14T07:06:02.5382811Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b31babb6476f924b.xml (deflated 42%) 2023-11-14T07:06:02.5384857Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5a046d5713ee7a6a.xml (deflated 43%) 2023-11-14T07:06:02.5386882Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5808d7ff8ca8a317.xml (deflated 43%) 2023-11-14T07:06:02.5388914Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-230dc0564d0cf487.xml (deflated 45%) 2023-11-14T07:06:02.5390919Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6621c32dbd01ec30.xml (deflated 45%) 2023-11-14T07:06:02.5392962Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b3bbfd0443f3f4b4.xml (deflated 43%) 2023-11-14T07:06:02.5395190Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-80f5e42104bbbf6d.xml (deflated 38%) 2023-11-14T07:06:02.5397247Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-60c05619cfef4f7f.xml (deflated 43%) 2023-11-14T07:06:02.5399254Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4fb774bf669e7d03.xml (deflated 38%) 2023-11-14T07:06:02.5401275Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-97860883f63d4a59.xml (deflated 43%) 2023-11-14T07:06:02.5403314Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ec57bb35a6e92474.xml (deflated 43%) 2023-11-14T07:06:02.5405341Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a655981200adcb0d.xml (deflated 43%) 2023-11-14T07:06:02.5407345Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-19f808da1dc7d4c7.xml (deflated 44%) 2023-11-14T07:06:02.5409403Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-56db543d701e215d.xml (deflated 44%) 2023-11-14T07:06:02.5411447Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e2181149ccdee895.xml (deflated 44%) 2023-11-14T07:06:02.5413461Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f08c155f99637570.xml (deflated 37%) 2023-11-14T07:06:02.5415460Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-28801aee740c8255.xml (deflated 44%) 2023-11-14T07:06:02.5417509Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0af15c41ee7a8ffc.xml (deflated 48%) 2023-11-14T07:06:02.5419453Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c01f505ed020a5e.xml (deflated 38%) 2023-11-14T07:06:02.5421702Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-da3c632ca7e02695.xml (deflated 44%) 2023-11-14T07:06:02.5423732Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8af9a73508e0c55c.xml (deflated 49%) 2023-11-14T07:06:02.5425823Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1cfbb42dd68cf893.xml (deflated 43%) 2023-11-14T07:06:02.5427843Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-131facd62ae1ccc3.xml (deflated 49%) 2023-11-14T07:06:02.5429876Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0fabd9d991ebd38d.xml (deflated 44%) 2023-11-14T07:06:02.5432084Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a1b654606298a332.xml (deflated 41%) 2023-11-14T07:06:02.5434128Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-da1144ba1b65a8e5.xml (deflated 41%) 2023-11-14T07:06:02.5436351Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9399bac6c6a07fee.xml (deflated 38%) 2023-11-14T07:06:02.5438381Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2a0d6300ee10d39.xml (deflated 38%) 2023-11-14T07:06:02.5440410Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-63ab086f43fe1847.xml (deflated 48%) 2023-11-14T07:06:02.5442380Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-26a9fb54e0617732.xml (deflated 38%) 2023-11-14T07:06:02.5444490Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0f8140fb21f690f1.xml (deflated 38%) 2023-11-14T07:06:02.5446489Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-331277589881a439.xml (deflated 37%) 2023-11-14T07:06:02.5448513Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-498e8742ff89787a.xml (deflated 38%) 2023-11-14T07:06:02.5450564Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e7fe7c41c3deb5dc.xml (deflated 37%) 2023-11-14T07:06:02.5452552Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3cc68a2843290ff9.xml (deflated 37%) 2023-11-14T07:06:02.5454605Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc83f5a6edb2ea62.xml (deflated 37%) 2023-11-14T07:06:02.5456602Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0e1235668b500c35.xml (deflated 37%) 2023-11-14T07:06:02.5458682Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-416ea1a5ccbd99ea.xml (deflated 37%) 2023-11-14T07:06:02.5460705Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2590a7ae621edf03.xml (deflated 42%) 2023-11-14T07:06:02.5462724Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0d9d96a67a205fab.xml (deflated 38%) 2023-11-14T07:06:02.5464711Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5d412cd01d5353e7.xml (deflated 38%) 2023-11-14T07:06:02.5466794Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8233f899b3a58155.xml (deflated 38%) 2023-11-14T07:06:02.5468989Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-98b0903ba38b5ffd.xml (deflated 38%) 2023-11-14T07:06:02.5471034Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6bc9eb0904ff907a.xml (deflated 37%) 2023-11-14T07:06:02.5472954Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-904fbcad1e2169c0.xml (deflated 37%) 2023-11-14T07:06:02.5475236Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-56c025f4e7ccc22e.xml (deflated 37%) 2023-11-14T07:06:02.5477342Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c2a5de6f317c70a.xml (deflated 37%) 2023-11-14T07:06:02.5479356Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-afeedc5279edb88d.xml (deflated 38%) 2023-11-14T07:06:02.5481535Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-827731b71a1740e0.xml (deflated 38%) 2023-11-14T07:06:02.5483625Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-abe6a6ab6cf4f75c.xml (deflated 37%) 2023-11-14T07:06:02.5485722Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f25ba57e0c7f995d.xml (deflated 37%) 2023-11-14T07:06:02.5487721Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34789349779fc7cf.xml (deflated 37%) 2023-11-14T07:06:02.5489734Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f9a844aea573e49d.xml (deflated 58%) 2023-11-14T07:06:02.5491796Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9982dc4f794637e8.xml (deflated 37%) 2023-11-14T07:06:02.5493858Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0b7b668dda1239f6.xml (deflated 38%) 2023-11-14T07:06:02.5495871Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bb276f3955f2a20d.xml (deflated 37%) 2023-11-14T07:06:02.5497914Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ab6b0db887aa4734.xml (deflated 37%) 2023-11-14T07:06:02.5499892Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c3d099a8a5af56b5.xml (deflated 38%) 2023-11-14T07:06:02.5501989Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e801fd9d59b1d288.xml (deflated 37%) 2023-11-14T07:06:02.5503998Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cc859826512de0d4.xml (deflated 37%) 2023-11-14T07:06:02.5506048Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e0eba9d0ea4d984.xml (deflated 37%) 2023-11-14T07:06:02.5508103Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-01ed6fe693bc6bc8.xml (deflated 43%) 2023-11-14T07:06:02.5510136Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c551f1b8f82f5b06.xml (deflated 37%) 2023-11-14T07:06:02.5512117Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3cde2392c745029.xml (deflated 43%) 2023-11-14T07:06:02.5514177Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-19293e3cee699b2c.xml (deflated 38%) 2023-11-14T07:06:02.5516457Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2daa7fe2f3b5fb64.xml (deflated 38%) 2023-11-14T07:06:02.5518691Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-63e8aca84fcc74a9.xml (deflated 38%) 2023-11-14T07:06:02.5520732Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-be7a07d2547c0128.xml (deflated 38%) 2023-11-14T07:06:02.5522766Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a5532f55621ae7c9.xml (deflated 37%) 2023-11-14T07:06:02.5524696Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7f5fd279bdd8d1df.xml (deflated 37%) 2023-11-14T07:06:02.5526793Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c2e4d25b440a5bc.xml (deflated 43%) 2023-11-14T07:06:02.5528999Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ecd2df844734e253.xml (deflated 43%) 2023-11-14T07:06:02.5531039Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f28f2962dcba214d.xml (deflated 43%) 2023-11-14T07:06:02.5533075Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-76bf8c825b4459d5.xml (deflated 38%) 2023-11-14T07:06:02.5535114Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4a2402b4756d7a5e.xml (deflated 37%) 2023-11-14T07:06:02.5537105Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-390634a096253457.xml (deflated 44%) 2023-11-14T07:06:02.5539144Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5f03ed97b35990ff.xml (deflated 44%) 2023-11-14T07:06:02.5541158Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b3244f92c0532612.xml (deflated 38%) 2023-11-14T07:06:02.5543106Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c73e52e53579496.xml (deflated 38%) 2023-11-14T07:06:02.5545186Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e96f0e1a4581e626.xml (deflated 38%) 2023-11-14T07:06:02.5547195Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-696625406e7d7986.xml (deflated 38%) 2023-11-14T07:06:02.5549290Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-111b3e2214607fd1.xml (deflated 43%) 2023-11-14T07:06:02.5551316Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-77380f697f38ab47.xml (deflated 43%) 2023-11-14T07:06:02.5553343Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c2c96cfae84d2404.xml (deflated 43%) 2023-11-14T07:06:02.5555577Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f04fa510e80ad970.xml (deflated 37%) 2023-11-14T07:06:02.5557612Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b88da5a86b45aa49.xml (deflated 37%) 2023-11-14T07:06:02.5559668Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-129be34e65b53cc2.xml (deflated 37%) 2023-11-14T07:06:02.5561681Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aec6283e280d73a8.xml (deflated 47%) 2023-11-14T07:06:02.5563731Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ab6e195afbdd1c1c.xml (deflated 47%) 2023-11-14T07:06:02.5565775Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3883d613ece035a5.xml (deflated 37%) 2023-11-14T07:06:02.5567957Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4a2688f6f50d44f7.xml (deflated 43%) 2023-11-14T07:06:02.5570000Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-175aaf03233653a9.xml (deflated 43%) 2023-11-14T07:06:02.5572038Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c785d4d3cd354646.xml (deflated 43%) 2023-11-14T07:06:02.5574074Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c72ac4b07f05d0f3.xml (deflated 43%) 2023-11-14T07:06:02.5576078Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1134e6a72bf71c4a.xml (deflated 43%) 2023-11-14T07:06:02.5578281Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca46bffaa0197f37.xml (deflated 43%) 2023-11-14T07:06:02.5580330Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9c5959b40f90f2c4.xml (deflated 37%) 2023-11-14T07:06:02.5582405Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39e8f31835911c2a.xml (deflated 43%) 2023-11-14T07:06:02.5584442Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bc60a00775021cf3.xml (deflated 38%) 2023-11-14T07:06:02.5586342Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e4af4b73f0fd8e7.xml (deflated 43%) 2023-11-14T07:06:02.5588377Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9002607c67cddd37.xml (deflated 38%) 2023-11-14T07:06:02.5590529Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d56df89c1c6a2773.xml (deflated 43%) 2023-11-14T07:06:02.5592787Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a34ea33a15644d43.xml (deflated 43%) 2023-11-14T07:06:02.5595084Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e0b5f0d90140318.xml (deflated 38%) 2023-11-14T07:06:02.5597093Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6640534141d5a1f9.xml (deflated 37%) 2023-11-14T07:06:02.5599120Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-780a9399361ad8d2.xml (deflated 43%) 2023-11-14T07:06:02.5601150Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b7f1ab5d64205005.xml (deflated 43%) 2023-11-14T07:06:02.5603162Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5b08926afbc28327.xml (deflated 45%) 2023-11-14T07:06:02.5605192Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-280336773506c5a5.xml (deflated 45%) 2023-11-14T07:06:02.5607220Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-82db110d28930ab7.xml (deflated 37%) 2023-11-14T07:06:02.5609257Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4d29828415efdb59.xml (deflated 38%) 2023-11-14T07:06:02.5611263Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-474fc6b4467eeeb9.xml (deflated 45%) 2023-11-14T07:06:02.5613302Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2faa01d1df50a7b6.xml (deflated 45%) 2023-11-14T07:06:02.5615509Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-096df655e519a773.xml (deflated 45%) 2023-11-14T07:06:02.5617618Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ed91929ff3f54e6.xml (deflated 45%) 2023-11-14T07:06:02.5619775Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e1c68edf4e710a87.xml (deflated 38%) 2023-11-14T07:06:02.5621860Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-504775bb20eb3b37.xml (deflated 38%) 2023-11-14T07:06:02.5623947Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6e2da40136798593.xml (deflated 43%) 2023-11-14T07:06:02.5625900Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-97011c5f813edeaa.xml (deflated 58%) 2023-11-14T07:06:02.5628126Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb3a98eab001f4eb.xml (deflated 39%) 2023-11-14T07:06:02.5630163Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e2416b9aa862ab3.xml (deflated 37%) 2023-11-14T07:06:02.5632170Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e4125e6349aca78e.xml (deflated 37%) 2023-11-14T07:06:02.5634213Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-576c4b9d361f090e.xml (deflated 37%) 2023-11-14T07:06:02.5636460Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8cdbf6ee296b6475.xml (deflated 38%) 2023-11-14T07:06:02.5638506Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fdde95d1d18bb8cb.xml (deflated 37%) 2023-11-14T07:06:02.5640552Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2b21dd8ffea6986a.xml (deflated 38%) 2023-11-14T07:06:02.5642595Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c1e7a9c50c3dd9e.xml (deflated 38%) 2023-11-14T07:06:02.5644635Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8ff421c8b5cee9d0.xml (deflated 39%) 2023-11-14T07:06:02.5646666Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7b403652801f7fec.xml (deflated 42%) 2023-11-14T07:06:02.5648664Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a498ea38a923eb6c.xml (deflated 37%) 2023-11-14T07:06:02.5650704Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1eab5e89c0e056b0.xml (deflated 37%) 2023-11-14T07:06:02.5652769Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fd10e508b72512c1.xml (deflated 36%) 2023-11-14T07:06:02.5654805Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-add531748b4dfa2c.xml (deflated 37%) 2023-11-14T07:06:02.5656810Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30bd179b869af655.xml (deflated 43%) 2023-11-14T07:06:02.5658844Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a983325d4597d9b5.xml (deflated 38%) 2023-11-14T07:06:02.5660875Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8ec7a5fc7170d22e.xml (deflated 49%) 2023-11-14T07:06:02.5662892Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4bbfcb68c9759f3.xml (deflated 45%) 2023-11-14T07:06:02.5665088Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2d2f64650d27a888.xml (deflated 37%) 2023-11-14T07:06:02.5667152Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51dee0dd4a88aa18.xml (deflated 44%) 2023-11-14T07:06:02.5669186Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c014d9a65f82b711.xml (deflated 38%) 2023-11-14T07:06:02.5671187Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e5ff12a2809d139.xml (deflated 37%) 2023-11-14T07:06:02.5673220Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-113b66b7917b0f1d.xml (deflated 49%) 2023-11-14T07:06:02.5675524Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e8b023e348321575.xml (deflated 38%) 2023-11-14T07:06:02.5677711Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f0549cb3b717b331.xml (deflated 38%) 2023-11-14T07:06:02.5679722Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bdbde93181036c55.xml (deflated 38%) 2023-11-14T07:06:02.5681857Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-586c16254ba73641.xml (deflated 38%) 2023-11-14T07:06:02.5683906Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb3fe0a9e2ff6b06.xml (deflated 49%) 2023-11-14T07:06:02.5685878Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-56091b807efd8598.xml (deflated 49%) 2023-11-14T07:06:02.5687897Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3f2b843826f7d53b.xml (deflated 38%) 2023-11-14T07:06:02.5689938Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5f9f441da3ea77ca.xml (deflated 37%) 2023-11-14T07:06:02.5691974Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ac8dc59b1805052.xml (deflated 37%) 2023-11-14T07:06:02.5694007Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b3fa2a0cf405b83a.xml (deflated 38%) 2023-11-14T07:06:02.5696016Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-72bfca61089b0504.xml (deflated 38%) 2023-11-14T07:06:02.5698047Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-584714f9fc9a72c5.xml (deflated 38%) 2023-11-14T07:06:02.5700135Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0adf1aab22feb1c2.xml (deflated 38%) 2023-11-14T07:06:02.5702151Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-42842533df8b252f.xml (deflated 43%) 2023-11-14T07:06:02.5704158Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c4120e466f4e713.xml (deflated 43%) 2023-11-14T07:06:02.5706182Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b365cf4e8961d93.xml (deflated 43%) 2023-11-14T07:06:02.5708220Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b3f3afa0f570c718.xml (deflated 44%) 2023-11-14T07:06:02.5710253Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-773af5119130b2a2.xml (deflated 44%) 2023-11-14T07:06:02.5712270Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f3aded032445a5a.xml (deflated 44%) 2023-11-14T07:06:02.5714429Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3710ed8320dbe006.xml (deflated 44%) 2023-11-14T07:06:02.5716687Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-64de7f16b808587e.xml (deflated 44%) 2023-11-14T07:06:02.5718727Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13fa052b8a05419a.xml (deflated 44%) 2023-11-14T07:06:02.5720729Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c3ee94082f625a8c.xml (deflated 38%) 2023-11-14T07:06:02.5722756Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-78718b57f1dc0d99.xml (deflated 37%) 2023-11-14T07:06:02.5724963Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ef633029817cc5b.xml (deflated 49%) 2023-11-14T07:06:02.5726996Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-69699f97e0448663.xml (deflated 49%) 2023-11-14T07:06:02.5729009Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a8dc7bb5c9ccf400.xml (deflated 49%) 2023-11-14T07:06:02.5731046Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-57967a7a01c6de2f.xml (deflated 49%) 2023-11-14T07:06:02.5733086Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8af2b6da1bf0e86a.xml (deflated 37%) 2023-11-14T07:06:02.5735129Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15efff3a1d0272f1.xml (deflated 37%) 2023-11-14T07:06:02.5737152Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ea0c5f0b2bfdb0c.xml (deflated 42%) 2023-11-14T07:06:02.5739199Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1fd204ab22b27a7f.xml (deflated 42%) 2023-11-14T07:06:02.5741227Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9199eda87445e0b1.xml (deflated 42%) 2023-11-14T07:06:02.5743253Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c56409f0b30ce821.xml (deflated 42%) 2023-11-14T07:06:02.5745269Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e8f5add6ec72509.xml (deflated 38%) 2023-11-14T07:06:02.5747306Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d760b56f723bba08.xml (deflated 37%) 2023-11-14T07:06:02.5749346Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a67403e21a0e67d2.xml (deflated 37%) 2023-11-14T07:06:02.5751392Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cae12df9d44ec4fe.xml (deflated 37%) 2023-11-14T07:06:02.5753410Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e1b2c0e952cc14c.xml (deflated 37%) 2023-11-14T07:06:02.5755700Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-008a112b0c5906b8.xml (deflated 37%) 2023-11-14T07:06:02.5757679Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2563a37ec75fac00.xml (deflated 37%) 2023-11-14T07:06:02.5759759Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bc15b85b75a4b4d5.xml (deflated 37%) 2023-11-14T07:06:02.5761994Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4702bdc83785d1e8.xml (deflated 38%) 2023-11-14T07:06:02.5764120Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5440508529da0bf2.xml (deflated 38%) 2023-11-14T07:06:02.5766192Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3f5581195dc4f129.xml (deflated 38%) 2023-11-14T07:06:02.5768115Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-700392a528786283.xml (deflated 78%) 2023-11-14T07:06:02.5770146Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e91ba7ddccec34c8.xml (deflated 78%) 2023-11-14T07:06:02.5772156Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9613591347de16a.xml (deflated 37%) 2023-11-14T07:06:02.5774369Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-52defb573393a1f1.xml (deflated 37%) 2023-11-14T07:06:02.5776407Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d3d5430b0a950244.xml (deflated 37%) 2023-11-14T07:06:02.5778441Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9ceeb35cb9e28942.xml (deflated 37%) 2023-11-14T07:06:02.5780597Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9cca654cac4f9f35.xml (deflated 37%) 2023-11-14T07:06:02.5782702Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c8cb5e62554ad8e.xml (deflated 38%) 2023-11-14T07:06:02.5784736Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc19f69085cd5d60.xml (deflated 38%) 2023-11-14T07:06:02.5786786Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-58fefe7a75eb8d83.xml (deflated 37%) 2023-11-14T07:06:02.5788789Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b23db701a815e67f.xml (deflated 37%) 2023-11-14T07:06:02.5790830Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6052f85bf4f36bd9.xml (deflated 37%) 2023-11-14T07:06:02.5792865Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dfef39b05cf903c6.xml (deflated 58%) 2023-11-14T07:06:02.5795122Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-392bb019e04a525b.xml (deflated 37%) 2023-11-14T07:06:02.5797097Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a92d5866ed468410.xml (deflated 37%) 2023-11-14T07:06:02.5799153Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-89c8eb122c442fd2.xml (deflated 37%) 2023-11-14T07:06:02.5801187Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a29961c4787a509.xml (deflated 37%) 2023-11-14T07:06:02.5803209Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2284649f6ce1fd6b.xml (deflated 37%) 2023-11-14T07:06:02.5805211Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0216a5c1e9673a99.xml (deflated 37%) 2023-11-14T07:06:02.5807234Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e690c97262b9f1a.xml (deflated 37%) 2023-11-14T07:06:02.5809280Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d1cd7d2afbcc0beb.xml (deflated 37%) 2023-11-14T07:06:02.5811527Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7d11243343d2612e.xml (deflated 38%) 2023-11-14T07:06:02.5813573Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b13d84c867fe99ab.xml (deflated 43%) 2023-11-14T07:06:02.5815605Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5da298f07d8641f6.xml (deflated 37%) 2023-11-14T07:06:02.5817644Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32ba4a8b8a071bce.xml (deflated 37%) 2023-11-14T07:06:02.5819678Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70f6fcfa34947147.xml (deflated 38%) 2023-11-14T07:06:02.5821681Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-695df6a87d1ca124.xml (deflated 38%) 2023-11-14T07:06:02.5823881Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1b7f35dfb2ff66f.xml (deflated 38%) 2023-11-14T07:06:02.5825925Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34bc5819fb526dcc.xml (deflated 36%) 2023-11-14T07:06:02.5827961Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e9170ff2b6efaeae.xml (deflated 37%) 2023-11-14T07:06:02.5829975Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2bc13a1ae54b17e4.xml (deflated 38%) 2023-11-14T07:06:02.5832017Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4d54ca8c3597dd27.xml (deflated 38%) 2023-11-14T07:06:02.5834064Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ea9e7533e641e6f.xml (deflated 37%) 2023-11-14T07:06:02.5836357Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c8e49f57f9738e72.xml (deflated 37%) 2023-11-14T07:06:02.5838367Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b4029a89c195b80c.xml (deflated 37%) 2023-11-14T07:06:02.5840403Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dce97195ad54772d.xml (deflated 37%) 2023-11-14T07:06:02.5842443Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-016c4a267886d17b.xml (deflated 37%) 2023-11-14T07:06:02.5844474Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3408f4e0db3157c1.xml (deflated 45%) 2023-11-14T07:06:02.5846503Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6a68b2b4eecc0c38.xml (deflated 45%) 2023-11-14T07:06:02.5848553Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-691c3f0b0bf33453.xml (deflated 45%) 2023-11-14T07:06:02.5850584Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3257467972824c7.xml (deflated 43%) 2023-11-14T07:06:02.5852575Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-75143866348aa99a.xml (deflated 42%) 2023-11-14T07:06:02.5854591Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6712135cd5b769b4.xml (deflated 43%) 2023-11-14T07:06:02.5856624Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f4bea4e91a1a7d84.xml (deflated 43%) 2023-11-14T07:06:02.5858672Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-066b58d1b312a291.xml (deflated 37%) 2023-11-14T07:06:02.5860825Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af55d825bec77c78.xml (deflated 38%) 2023-11-14T07:06:02.5862889Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d31389e48a81cae1.xml (deflated 37%) 2023-11-14T07:06:02.5864934Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-79b628499dec5c4a.xml (deflated 47%) 2023-11-14T07:06:02.5866972Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-14e6c982cc5fbc72.xml (deflated 47%) 2023-11-14T07:06:02.5868974Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-054a7e4687e3a111.xml (deflated 37%) 2023-11-14T07:06:02.5871147Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-263aa259b77e97a5.xml (deflated 37%) 2023-11-14T07:06:02.5873192Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32cb87c54d935331.xml (deflated 37%) 2023-11-14T07:06:02.5875514Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c0793b30ce0e2fd.xml (deflated 49%) 2023-11-14T07:06:02.5877498Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2e55e0043dcf126f.xml (deflated 49%) 2023-11-14T07:06:02.5879531Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-560ff1bcd8020f97.xml (deflated 49%) 2023-11-14T07:06:02.5881597Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-884269dd150ded62.xml (deflated 38%) 2023-11-14T07:06:02.5883657Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c63d4dd5c837ed8f.xml (deflated 45%) 2023-11-14T07:06:02.5885670Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-997d52848e6f94f4.xml (deflated 38%) 2023-11-14T07:06:02.5887690Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-64a9352591e67483.xml (deflated 44%) 2023-11-14T07:06:02.5889711Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-04a0409456687347.xml (deflated 38%) 2023-11-14T07:06:02.5891744Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f34ff544b4c04c0c.xml (deflated 42%) 2023-11-14T07:06:02.5893755Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-877844c626dccdad.xml (deflated 38%) 2023-11-14T07:06:02.5895799Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-27b712d4ae912e86.xml (deflated 38%) 2023-11-14T07:06:02.5897843Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f2ec2b9bb9299300.xml (deflated 42%) 2023-11-14T07:06:02.5899951Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f7a9233114a957f.xml (deflated 42%) 2023-11-14T07:06:02.5901960Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-010ac736bd74e081.xml (deflated 38%) 2023-11-14T07:06:02.5904002Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e398410b713cf020.xml (deflated 49%) 2023-11-14T07:06:02.5906212Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a5303858a1646c9f.xml (deflated 38%) 2023-11-14T07:06:02.5908308Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f3f026ed1bc50aaa.xml (deflated 37%) 2023-11-14T07:06:02.5910373Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-020d0ed4d835eaa5.xml (deflated 41%) 2023-11-14T07:06:02.5912379Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-42e62a6fa656f400.xml (deflated 42%) 2023-11-14T07:06:02.5914407Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-54819b4865d23f05.xml (deflated 37%) 2023-11-14T07:06:02.5916634Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8d505b21c2e6332c.xml (deflated 38%) 2023-11-14T07:06:02.5918719Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b55fee1eeb8885cc.xml (deflated 37%) 2023-11-14T07:06:02.5920923Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f635cdfd203c24d6.xml (deflated 38%) 2023-11-14T07:06:02.5922968Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1b3811cd57ad63e8.xml (deflated 38%) 2023-11-14T07:06:02.5925017Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-8c272ede8b5a3be4.xml (deflated 82%) 2023-11-14T07:06:02.5927363Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_fsdp_2d_parallel/distributed.tensor.parallel.test_fsdp_2d_parallel-d2e28d127e574b59.xml (deflated 89%) 2023-11-14T07:06:02.5929545Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-e93e78e9e1e97a63.xml (deflated 37%) 2023-11-14T07:06:02.5931408Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-daf6b98e668710a0.xml (deflated 39%) 2023-11-14T07:06:02.5933276Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-3ed5cef8518672dd.xml (deflated 36%) 2023-11-14T07:06:02.5935171Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-dc9a0d9084f49f4c.xml (deflated 37%) 2023-11-14T07:06:02.5936945Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-f60bf6dc43d92c6c.xml (deflated 44%) 2023-11-14T07:06:02.5938801Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-351b12338e20486c.xml (deflated 37%) 2023-11-14T07:06:02.5940634Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-4274ba8968504761.xml (deflated 37%) 2023-11-14T07:06:02.5942507Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-d91a7143b37600c2.xml (deflated 37%) 2023-11-14T07:06:02.5944279Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-df50c20d9f6dea96.xml (deflated 38%) 2023-11-14T07:06:02.5946196Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-d922bc8bc241605b.xml (deflated 39%) 2023-11-14T07:06:02.5947998Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-b6a595ee8e667057.xml (deflated 39%) 2023-11-14T07:06:02.5949811Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-b313cb0674edf263.xml (deflated 38%) 2023-11-14T07:06:02.5951645Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-59716bacc85e79b7.xml (deflated 38%) 2023-11-14T07:06:02.5953503Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-21e548927eb1ea15.xml (deflated 37%) 2023-11-14T07:06:02.5955547Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-efd267c6102b030f.xml (deflated 37%) 2023-11-14T07:06:02.5957546Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-36c69b40bc8fd692.xml (deflated 38%) 2023-11-14T07:06:02.5959423Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-e8be40a1640b964c.xml (deflated 37%) 2023-11-14T07:06:02.5961273Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-2496f51dbd4eac8d.xml (deflated 37%) 2023-11-14T07:06:02.5963124Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-a83a731512a1aab4.xml (deflated 37%) 2023-11-14T07:06:02.5964935Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-415a64f45a7a5d64.xml (deflated 37%) 2023-11-14T07:06:02.5966785Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-79e47fa99db7a614.xml (deflated 38%) 2023-11-14T07:06:02.5968786Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-8ac1b0ecfef1e7eb.xml (deflated 38%) 2023-11-14T07:06:02.5970642Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-4c8ebfacdbe63135.xml (deflated 38%) 2023-11-14T07:06:02.5972466Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-eb8429f9b0f98fbf.xml (deflated 37%) 2023-11-14T07:06:02.5974316Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-b6a727b84672cbea.xml (deflated 38%) 2023-11-14T07:06:02.5976164Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-3ac97a2e397e862f.xml (deflated 38%) 2023-11-14T07:06:02.5978008Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-8ba5da81cdb4102e.xml (deflated 38%) 2023-11-14T07:06:02.5979832Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-d77873c2704b1860.xml (deflated 45%) 2023-11-14T07:06:02.5981739Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-280eb0c800f53bca.xml (deflated 38%) 2023-11-14T07:06:02.5983584Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-010553f06598d48c.xml (deflated 38%) 2023-11-14T07:06:02.5985419Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-5b206fcdd1c4afbe.xml (deflated 44%) 2023-11-14T07:06:02.5987239Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-ebdf8fb193cef8b2.xml (deflated 42%) 2023-11-14T07:06:02.5989094Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-7c01ddabad98a7a1.xml (deflated 42%) 2023-11-14T07:06:02.5990943Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-dfe5efab9f30624d.xml (deflated 42%) 2023-11-14T07:06:02.5992774Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-c51aeb7b4da81118.xml (deflated 38%) 2023-11-14T07:06:02.5994841Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-ca71c5dc04b754b8.xml (deflated 38%) 2023-11-14T07:06:02.5996690Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-df5d34ec54eed475.xml (deflated 38%) 2023-11-14T07:06:02.5998547Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-531b6e4ac71bef5c.xml (deflated 38%) 2023-11-14T07:06:02.6000360Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-165246afb5dc0c12.xml (deflated 45%) 2023-11-14T07:06:02.6002188Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-487a412f7cf9ae93.xml (deflated 45%) 2023-11-14T07:06:02.6004052Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-93d13126f46cc532.xml (deflated 38%) 2023-11-14T07:06:02.6006010Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-c3a4e1f05d355bcc.xml (deflated 38%) 2023-11-14T07:06:02.6007882Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-9683f66aab8158a9.xml (deflated 38%) 2023-11-14T07:06:02.6009757Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-01e4c430bd882b36.xml (deflated 38%) 2023-11-14T07:06:02.6011555Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-78c0df9e09c532b4.xml (deflated 47%) 2023-11-14T07:06:02.6013448Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-b5b1e53cf1b69152.xml (deflated 47%) 2023-11-14T07:06:02.6015210Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-67fd20d62eaa5976.xml (deflated 38%) 2023-11-14T07:06:02.6017201Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-00ff508ebe4a656f.xml (deflated 38%) 2023-11-14T07:06:02.6019056Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-dff5a36bfd34f033.xml (deflated 38%) 2023-11-14T07:06:02.6020905Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-55db7ace36016fc7.xml (deflated 49%) 2023-11-14T07:06:02.6022713Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-824837f133029051.xml (deflated 38%) 2023-11-14T07:06:02.6024561Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-758d6a44ae4fa8a9.xml (deflated 37%) 2023-11-14T07:06:02.6026410Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-5e2e79f8f47e0838.xml (deflated 37%) 2023-11-14T07:06:02.6028236Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-a7048d5a0e0c64c7.xml (deflated 37%) 2023-11-14T07:06:02.6030079Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-57bc78c69f34a81e.xml (deflated 42%) 2023-11-14T07:06:02.6031930Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-9b31443697e31257.xml (deflated 37%) 2023-11-14T07:06:02.6033767Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-0aca0a8c5d5114e5.xml (deflated 42%) 2023-11-14T07:06:02.6035794Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-ce4176eb8c52d281.xml (deflated 37%) 2023-11-14T07:06:02.6037640Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-a4abc180a788f976.xml (deflated 36%) 2023-11-14T07:06:02.6039500Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-cfd013cfadd50b5a.xml (deflated 37%) 2023-11-14T07:06:02.6041366Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-5840232d67cf898f.xml (deflated 37%) 2023-11-14T07:06:02.6043175Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-0394969af9ae7809.xml (deflated 37%) 2023-11-14T07:06:02.6045018Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-cfe03e8a335dedd5.xml (deflated 37%) 2023-11-14T07:06:02.6046870Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-c28a4b40187ceea6.xml (deflated 37%) 2023-11-14T07:06:02.6048710Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-e4e3a85888200a17.xml (deflated 37%) 2023-11-14T07:06:02.6050516Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-487982faed6c08ce.xml (deflated 37%) 2023-11-14T07:06:02.6052378Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-79b66a0bb2e5f0ef.xml (deflated 37%) 2023-11-14T07:06:02.6054317Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-227934a80022d476.xml (deflated 40%) 2023-11-14T07:06:02.6056180Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-7c521409066e11f2.xml (deflated 41%) 2023-11-14T07:06:02.6057991Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-3e9ae94d348d2f14.xml (deflated 37%) 2023-11-14T07:06:02.6059840Z adding: test/test-reports/python-pytest/distributed.test_c10d_ucc/distributed.test_c10d_ucc-30812c869c9d3961.xml (deflated 37%) 2023-11-14T07:06:02.6061932Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_unshard_params/distributed.fsdp.test_fsdp_unshard_params-8c5238691f76a312.xml (deflated 80%) 2023-11-14T07:06:02.6064150Z adding: test/test-reports/python-pytest/distributed.test_launcher/distributed.test_launcher-f225b48a808eba78.xml (deflated 39%) 2023-11-14T07:06:02.6066292Z adding: test/test-reports/python-pytest/distributed.test_inductor_collectives/distributed.test_inductor_collectives-a922dcbba25319e0.xml (deflated 87%) 2023-11-14T07:06:02.6068584Z adding: test/test-reports/python-pytest/distributed._shard.sharded_optim.test_sharded_optim/distributed._shard.sharded_optim.test_sharded_optim-42179303872d8b77.xml (deflated 66%) 2023-11-14T07:06:02.6070978Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_ops/distributed._tensor.test_dtensor_ops-b3e33712246c0a8d.xml (deflated 28%) 2023-11-14T07:06:02.6073236Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ea8f578dbf736ae1.xml (deflated 38%) 2023-11-14T07:06:02.6075781Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-60e401517f41eccf.xml (deflated 37%) 2023-11-14T07:06:02.6078189Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-6e97298417dd7d83.xml (deflated 37%) 2023-11-14T07:06:02.6080571Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-dfd19115ebc37334.xml (deflated 38%) 2023-11-14T07:06:02.6082974Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-892068f0533768ef.xml (deflated 37%) 2023-11-14T07:06:02.6085326Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a525ff546355eeae.xml (deflated 37%) 2023-11-14T07:06:02.6087700Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-08bb0dfb96fe8b0d.xml (deflated 47%) 2023-11-14T07:06:02.6090089Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d923fce1cbe81822.xml (deflated 46%) 2023-11-14T07:06:02.6092469Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-43937c568b7f80ec.xml (deflated 46%) 2023-11-14T07:06:02.6094819Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1c2b741795e89765.xml (deflated 46%) 2023-11-14T07:06:02.6097189Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9fdbdeaad12bd3d0.xml (deflated 46%) 2023-11-14T07:06:02.6099571Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-bb7ae2a2a0b40b80.xml (deflated 38%) 2023-11-14T07:06:02.6101935Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-8c9d925f8008234e.xml (deflated 39%) 2023-11-14T07:06:02.6104395Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b09a8eb3d935810f.xml (deflated 39%) 2023-11-14T07:06:02.6106797Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-f86009a65e32bfda.xml (deflated 39%) 2023-11-14T07:06:02.6109171Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-62ccc175a0fff87b.xml (deflated 39%) 2023-11-14T07:06:02.6111539Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-69ed4bc0eb459320.xml (deflated 39%) 2023-11-14T07:06:02.6113907Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2769537426f02947.xml (deflated 38%) 2023-11-14T07:06:02.6116586Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-289b5ca6ea0c6a8f.xml (deflated 38%) 2023-11-14T07:06:02.6118971Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ec3fb33cabbc4873.xml (deflated 38%) 2023-11-14T07:06:02.6121345Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-22cc98cf8f599fca.xml (deflated 38%) 2023-11-14T07:06:02.6123708Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-35cbd7869b5f7309.xml (deflated 38%) 2023-11-14T07:06:02.6126055Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-16d7532b6f7f9ff5.xml (deflated 38%) 2023-11-14T07:06:02.6152417Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-466f61fd5a20bccc.xml (deflated 38%) 2023-11-14T07:06:02.6155450Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-fd1ac1be23491e28.xml (deflated 38%) 2023-11-14T07:06:02.6157801Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-f7c079c76fa64e3c.xml (deflated 38%) 2023-11-14T07:06:02.6160182Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-7b6ccbdd775c4ebf.xml (deflated 38%) 2023-11-14T07:06:02.6162572Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4e35c9e0fb7f5c30.xml (deflated 38%) 2023-11-14T07:06:02.6164945Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-30415279e17a41a0.xml (deflated 38%) 2023-11-14T07:06:02.6167286Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-dbeb9fe87be1d04a.xml (deflated 38%) 2023-11-14T07:06:02.6169653Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-621c1c8b89f6dd0a.xml (deflated 38%) 2023-11-14T07:06:02.6172001Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-7173ee0e37b54f67.xml (deflated 38%) 2023-11-14T07:06:02.6174357Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b4da34bd240996da.xml (deflated 38%) 2023-11-14T07:06:02.6176712Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-569fc01d5eff68e9.xml (deflated 38%) 2023-11-14T07:06:02.6179235Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ab14359602f47e73.xml (deflated 38%) 2023-11-14T07:06:02.6181685Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d096b7d162f6beba.xml (deflated 39%) 2023-11-14T07:06:02.6184033Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-8819675c6e7fe6aa.xml (deflated 38%) 2023-11-14T07:06:02.6186379Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-dcc2618f1130024c.xml (deflated 37%) 2023-11-14T07:06:02.6188708Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-485a15e3a1e5d190.xml (deflated 38%) 2023-11-14T07:06:02.6191286Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-daee9b8d264b38e6.xml (deflated 47%) 2023-11-14T07:06:02.6193622Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-04ebcc07afae4362.xml (deflated 47%) 2023-11-14T07:06:02.6196355Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a194f3913469ba4e.xml (deflated 38%) 2023-11-14T07:06:02.6198687Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5438fda56af45748.xml (deflated 38%) 2023-11-14T07:06:02.6201048Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-043e01aea1f1edd6.xml (deflated 38%) 2023-11-14T07:06:02.6203491Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1574e7c3f6ea50ea.xml (deflated 38%) 2023-11-14T07:06:02.6205909Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-726b988f25016b0b.xml (deflated 38%) 2023-11-14T07:06:02.6208378Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-dd472f5e5a0df9c1.xml (deflated 37%) 2023-11-14T07:06:02.6210592Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4690d976a6eb2fb9.xml (deflated 38%) 2023-11-14T07:06:02.6212942Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-15b726b1c08f450f.xml (deflated 38%) 2023-11-14T07:06:02.6215288Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2fbf27e768031cd7.xml (deflated 48%) 2023-11-14T07:06:02.6217654Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5546b4902cd8c233.xml (deflated 48%) 2023-11-14T07:06:02.6220286Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_runtime/distributed._composable.fully_shard.test_fully_shard_runtime-28c49fe118784898.xml (deflated 51%) 2023-11-14T07:06:02.6222720Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_state_dict/distributed.fsdp.test_fsdp_state_dict-14dca701841883be.xml (deflated 95%) 2023-11-14T07:06:02.6224807Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_fx/distributed.fsdp.test_fsdp_fx-f556765ea869bad9.xml (deflated 36%) 2023-11-14T07:06:02.6226857Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_memory/distributed.fsdp.test_fsdp_memory-3c7841275d52b167.xml (deflated 50%) 2023-11-14T07:06:02.6229174Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_compile/distributed._tensor.test_dtensor_compile-d0290b79a7f978e0.xml (deflated 78%) 2023-11-14T07:06:02.6231432Z adding: test/test-reports/python-pytest/distributed.test_compute_comm_reordering/distributed.test_compute_comm_reordering-0f863061c779a455.xml (deflated 81%) 2023-11-14T07:06:02.6233721Z adding: test/test-reports/python-pytest/distributed._tensor.debug.test_op_coverage/distributed._tensor.debug.test_op_coverage-8921b3e57476fd96.xml (deflated 35%) 2023-11-14T07:06:02.6236502Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_ignored_modules/distributed.fsdp.test_fsdp_ignored_modules-8f5505cf01c434e3.xml (deflated 82%) 2023-11-14T07:06:02.6238644Z adding: test/test-reports/python-pytest/distributed._tensor.test_utils/distributed._tensor.test_utils-d18a67b3203e183a.xml (deflated 69%) 2023-11-14T07:06:02.6240907Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_file_system_checkpoint/distributed.checkpoint.test_file_system_checkpoint-6235bf9872c72a68.xml (deflated 69%) 2023-11-14T07:06:02.6243695Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_file_system_checkpoint_cpu/distributed.checkpoint.test_file_system_checkpoint_cpu-6a44ff0255c399d7.xml (deflated 82%) 2023-11-14T07:06:02.6246309Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_sharded_tensor/distributed._shard.sharded_tensor.test_sharded_tensor-7e1c5a6aca7f5831.xml (deflated 94%) 2023-11-14T07:06:02.6248528Z adding: test/test-reports/python-pytest/distributed.test_c10d_pypg/distributed.test_c10d_pypg-8102217e3d1a4c4a.xml (deflated 91%) 2023-11-14T07:06:02.6250649Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_dtensor_state_dict/distributed.fsdp.test_fsdp_dtensor_state_dict-74920dc22de49585.xml (deflated 87%) 2023-11-14T07:06:02.6252800Z adding: test/test-reports/python-pytest/distributed.test_device_mesh/distributed.test_device_mesh-5b29c5704b8d64db.xml (deflated 91%) 2023-11-14T07:06:02.6254972Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_freezing_weights/distributed.fsdp.test_fsdp_freezing_weights-7544dba2a4b92e40.xml (deflated 83%) 2023-11-14T07:06:02.6257170Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_meta/distributed.fsdp.test_fsdp_meta-d5430695402fb2fa.xml (deflated 88%) 2023-11-14T07:06:02.6259255Z adding: test/test-reports/python-pytest/distributed._tensor.test_common_rules/distributed._tensor.test_common_rules-81d19e8921ae5a99.xml (deflated 86%) 2023-11-14T07:06:02.6261474Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-dc8f241f32c35e6a.xml (deflated 61%) 2023-11-14T07:06:02.6263678Z adding: test/test-reports/python-pytest/distributed.algorithms.test_join/distributed.algorithms.test_join-b49f1e2efcaf8855.xml (deflated 77%) 2023-11-14T07:06:02.6265769Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-6ec8a7d0d217a598.xml (deflated 36%) 2023-11-14T07:06:02.6267784Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-011f739086763dbc.xml (deflated 36%) 2023-11-14T07:06:02.6269815Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-6cfe6fe20be8b873.xml (deflated 36%) 2023-11-14T07:06:02.6271847Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-17dedc74d1eef3c0.xml (deflated 35%) 2023-11-14T07:06:02.6273878Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-e12e813dc8bc436a.xml (deflated 35%) 2023-11-14T07:06:02.6276384Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-a17bc8a8cf1830c3.xml (deflated 35%) 2023-11-14T07:06:02.6278435Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-27f3bf2f3dbd399a.xml (deflated 35%) 2023-11-14T07:06:02.6280596Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-3c69a147b954e581.xml (deflated 35%) 2023-11-14T07:06:02.6282703Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-d7b671f160a847e0.xml (deflated 34%) 2023-11-14T07:06:02.6284701Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-e3182bbe9088689e.xml (deflated 35%) 2023-11-14T07:06:02.6286828Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-d47fbfe3c4f613b5.xml (deflated 35%) 2023-11-14T07:06:02.6288863Z adding: test/test-reports/python-pytest/distributed._tensor.test_random_ops/distributed._tensor.test_random_ops-56e413f1beaf7f2c.xml (deflated 73%) 2023-11-14T07:06:02.6291437Z adding: test/test-reports/python-pytest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks-49d74ee36c40f15d.xml (deflated 74%) 2023-11-14T07:06:02.6293924Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_fine_tune/distributed.fsdp.test_fsdp_fine_tune-224ea6533a57e624.xml (deflated 54%) 2023-11-14T07:06:02.6296241Z adding: test/test-reports/python-pytest/distributed.elastic.multiprocessing.api_test/distributed.elastic.multiprocessing.api_test-6ec765371b9cdb3f.xml (deflated 89%) 2023-11-14T07:06:02.6298696Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_hsdp_checkpoint/distributed.checkpoint.test_hsdp_checkpoint-221af87e06859808.xml (deflated 78%) 2023-11-14T07:06:02.6301195Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_tensor_ops/distributed._shard.sharded_tensor.ops.test_tensor_ops-93a5983ed3196cc3.xml (deflated 80%) 2023-11-14T07:06:02.6304007Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_optim_checkpoint/distributed._composable.fully_shard.test_fully_shard_optim_checkpoint-07622b6039491a15.xml (deflated 54%) 2023-11-14T07:06:02.6306775Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_binary_cmp/distributed._shard.sharded_tensor.ops.test_binary_cmp-5081f6b4930dfa94.xml (deflated 78%) 2023-11-14T07:06:02.6309089Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-161d91d934442167.xml (deflated 50%) 2023-11-14T07:06:02.6311401Z adding: test/test-reports/python-pytest/distributed._shard.sharding_plan.test_sharding_plan/distributed._shard.sharding_plan.test_sharding_plan-298c3c625977365c.xml (deflated 73%) 2023-11-14T07:06:02.6313662Z adding: test/test-reports/python-pytest/distributed._tensor.test_view_ops/distributed._tensor.test_view_ops-911120dc11061cd7.xml (deflated 51%) 2023-11-14T07:06:02.6316212Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_embedding/distributed._shard.sharded_tensor.ops.test_embedding-0944558aafbddb34.xml (deflated 66%) 2023-11-14T07:06:02.6318821Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_util/distributed._composable.fully_shard.test_fully_shard_util-b5240174202cd8fd.xml (deflated 39%) 2023-11-14T07:06:02.6321330Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_multiple_forward/distributed.fsdp.test_fsdp_multiple_forward-58718bfd12be0c3e.xml (deflated 40%) 2023-11-14T07:06:02.6323555Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-b36cd9153ca488fd.xml (deflated 37%) 2023-11-14T07:06:02.6325765Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_balance/distributed.pipeline.sync.test_balance-174cfd88a2f53ea3.xml (deflated 82%) 2023-11-14T07:06:02.6328190Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_ddp_2d_parallel/distributed.tensor.parallel.test_ddp_2d_parallel-7a2ac1b4e8a7ad4c.xml (deflated 51%) 2023-11-14T07:06:02.6330885Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_random_state/distributed.tensor.parallel.test_tp_random_state-53f9d1e1fd1a8d95.xml (deflated 51%) 2023-11-14T07:06:02.6333392Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_gpipe/distributed.pipeline.sync.skip.test_gpipe-4c7a4367c3387e29.xml (deflated 80%) 2023-11-14T07:06:02.6335500Z adding: test/test-reports/python-pytest/distributed.test_fake_pg/distributed.test_fake_pg-4574126c63e70b77.xml (deflated 79%) 2023-11-14T07:06:02.6337599Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_checkpoint/distributed.pipeline.sync.test_checkpoint-dbdb6245a228417c.xml (deflated 78%) 2023-11-14T07:06:02.6339659Z adding: test/test-reports/python-pytest/distributed.test_nccl/distributed.test_nccl-406f2a9865f34674.xml (deflated 81%) 2023-11-14T07:06:02.6341681Z adding: test/test-reports/python-pytest/distributed.fsdp.test_utils/distributed.fsdp.test_utils-aeacc0e229eb0570.xml (deflated 67%) 2023-11-14T07:06:02.6343828Z adding: test/test-reports/python-pytest/distributed.elastic.utils.logging_test/distributed.elastic.utils.logging_test-78367b786c3a3a83.xml (deflated 51%) 2023-11-14T07:06:02.6346090Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_nested_dict/distributed.checkpoint.test_nested_dict-888c910161bbfbce.xml (deflated 51%) 2023-11-14T07:06:02.6348360Z adding: test/test-reports/python-pytest/distributed._tensor.test_xla_integration/distributed._tensor.test_xla_integration-0dd5a6bd09327984.xml (deflated 74%) 2023-11-14T07:06:02.6350653Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dedup_tensors/distributed.checkpoint.test_dedup_tensors-ab34cbb37c55a7cd.xml (deflated 38%) 2023-11-14T07:06:02.6353036Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.test_dependency/distributed.pipeline.sync.test_dependency-de358db8e2d05113.xml (deflated 73%) 2023-11-14T07:06:02.6355596Z adding: test/test-reports/python-pytest/distributed.pipeline.sync.skip.test_api/distributed.pipeline.sync.skip.test_api-87dcfcc797093a9f.xml (deflated 60%) 2023-11-14T07:06:02.6406236Z ##[group]Run # Remove any previous test reports if they exist 2023-11-14T07:06:02.6407110Z # Remove any previous test reports if they exist 2023-11-14T07:06:02.6407782Z rm -f logs-*.zip 2023-11-14T07:06:02.6408718Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2023-11-14T07:06:02.6409660Z # so check to see if the file exists first 2023-11-14T07:06:02.6410251Z if [ -f 'usage_log.txt' ]; then 2023-11-14T07:06:02.6410923Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2023-11-14T07:06:02.6411540Z fi 2023-11-14T07:06:02.6411981Z if ls test/**/*.log 1> /dev/null 2>&1; then 2023-11-14T07:06:02.6412723Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2023-11-14T07:06:02.6413353Z fi 2023-11-14T07:06:02.6422053Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T07:06:02.6422662Z env: 2023-11-14T07:06:02.6423011Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:06:02.6423574Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:06:02.6424484Z DOCKER_CONTAINER_ID: 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:02.6425561Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_18651430132 2023-11-14T07:06:02.6426314Z ##[endgroup] 2023-11-14T07:06:02.7604669Z adding: usage_log.txt (deflated 96%) 2023-11-14T07:06:02.7738369Z adding: test/test-reports/distributed.fsdp.test_fsdp_uneven_1.1_9ad6e5a414d5ca0c_.log (deflated 53%) 2023-11-14T07:06:02.7739919Z adding: test/test-reports/distributed.test_distributed_spawn_2.2_525f18ae3bfe5473_.log (deflated 13%) 2023-11-14T07:06:02.7741316Z adding: test/test-reports/distributed.test_fake_pg_1.1_7418aecdbb8be150_.log (deflated 72%) 2023-11-14T07:06:02.7742959Z adding: test/test-reports/distributed.test_distributed_spawn_2.2_321c95eedf29904e_.log (deflated 12%) 2023-11-14T07:06:02.7744381Z adding: test/test-reports/distributed.test_nccl_1.1_13e977476fbb9864_.log (deflated 74%) 2023-11-14T07:06:02.7770443Z adding: test/test-reports/distributed.test_distributed_spawn_2.2_059d461e6a272799_.log (deflated 95%) 2023-11-14T07:06:02.7771880Z adding: test/test-reports/distributed.fsdp.test_utils_1.1_a40297384aa577f7_.log (deflated 65%) 2023-11-14T07:06:02.7799345Z adding: test/test-reports/distributed.test_distributed_spawn_2.2_ad3bea52ae63b2cd_.log (deflated 95%) 2023-11-14T07:06:02.7800779Z adding: test/test-reports/distributed._shard.test_sharder_1.1_dc75e1c9c05b8732_.log (stored 0%) 2023-11-14T07:06:02.7873902Z adding: test/test-reports/distributed.fsdp.test_fsdp_core_1.2_b2f383fff00019b6_.log (deflated 93%) 2023-11-14T07:06:02.7875915Z adding: test/test-reports/distributed.test_device_mesh_1.1_94a3b8ebbf7662e6_.log (deflated 84%) 2023-11-14T07:06:02.7877478Z adding: test/test-reports/distributed.tensor.parallel.test_fsdp_2d_parallel_1.1_dcae9ad6d8b7f765_.log (deflated 81%) 2023-11-14T07:06:02.7890205Z adding: test/test-reports/distributed.test_c10d_ucc_1.1_43a9d5815ac9ab5f_.log (deflated 95%) 2023-11-14T07:06:02.7891617Z adding: test/test-reports/distributed.fsdp.test_fsdp_meta_1.1_86a1d625f103eca5_.log (deflated 83%) 2023-11-14T07:06:02.7896394Z adding: test/test-reports/distributed.fsdp.test_fsdp_unshard_params_1.1_dfdb81cd23df9d6f_.log (deflated 95%) 2023-11-14T07:06:02.7897843Z adding: test/test-reports/distributed.test_launcher_1.1_2c56c5528fbe9ba1_.log (deflated 58%) 2023-11-14T07:06:02.7899261Z adding: test/test-reports/distributed._tensor.test_common_rules_1.1_aa449c2135cd6371_.log (deflated 77%) 2023-11-14T07:06:02.7900762Z adding: test/test-reports/distributed.test_inductor_collectives_1.1_dc424b0b5fd53bbd_.log (deflated 86%) 2023-11-14T07:06:02.7903567Z adding: test/test-reports/distributed.fsdp.test_fsdp_dtensor_state_dict_1.1_e145425cb1b6fbc9_.log (deflated 84%) 2023-11-14T07:06:02.7905221Z adding: test/test-reports/distributed._shard.sharded_optim.test_sharded_optim_1.1_e4069bc29a1e9457_.log (deflated 61%) 2023-11-14T07:06:02.7913854Z adding: test/test-reports/distributed.fsdp.test_fsdp_clip_grad_norm_1.1_72737f60ebe253ba_.log (deflated 98%) 2023-11-14T07:06:02.7915691Z adding: test/test-reports/distributed._tensor.test_dtensor_ops_1.1_d3ae1cbf6d08bb8d_.log (deflated 52%) 2023-11-14T07:06:02.7917167Z adding: test/test-reports/distributed.algorithms.test_join_1.1_8c424c3203d40731_.log (deflated 72%) 2023-11-14T07:06:02.7923548Z adding: test/test-reports/distributed.rpc.cuda.test_tensorpipe_agent_1.2_52f2d26eb667a09e_.log (deflated 95%) 2023-11-14T07:06:02.7926790Z adding: test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_1.1_48a5d3de6b20f0d7_.log (deflated 90%) 2023-11-14T07:06:02.7928594Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_runtime_1.1_1706520139d64dec_.log (deflated 62%) 2023-11-14T07:06:02.7930207Z adding: test/test-reports/distributed.test_c10d_spawn_gloo_1.1_2c51ba1f9304c00e_.log (deflated 93%) 2023-11-14T07:06:02.7943710Z adding: test/test-reports/distributed.fsdp.test_fsdp_state_dict_2.2_64146294559d98d5_.log (deflated 90%) 2023-11-14T07:06:02.7945163Z adding: test/test-reports/distributed.fsdp.test_fsdp_fx_1.1_7d0ab38dc7463a54_.log (deflated 53%) 2023-11-14T07:06:02.7946676Z adding: test/test-reports/distributed.fsdp.test_fsdp_memory_1.1_e4c8298cc50027df_.log (deflated 59%) 2023-11-14T07:06:02.7948160Z adding: test/test-reports/distributed._tensor.test_random_ops_1.1_0d06beba3c43e61a_.log (deflated 69%) 2023-11-14T07:06:02.7949638Z adding: test/test-reports/distributed._tensor.test_dtensor_compile_1.1_9de133838ebb9950_.log (deflated 72%) 2023-11-14T07:06:02.7951137Z adding: test/test-reports/distributed.fsdp.test_fsdp_fine_tune_1.1_d0327bc14d3a0472_.log (deflated 95%) 2023-11-14T07:06:02.7952591Z adding: test/test-reports/distributed.test_compute_comm_reordering_1.1_37b81a6c4076f356_.log (deflated 73%) 2023-11-14T07:06:02.7954281Z adding: test/test-reports/distributed.fsdp.test_fsdp_input_1.1_e27646c14da77c3d_.log (deflated 57%) 2023-11-14T07:06:02.7956063Z adding: test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_7a86c0c4dd8f7fb1_.log (deflated 54%) 2023-11-14T07:06:02.7957654Z adding: test/test-reports/distributed._tensor.test_view_ops_1.1_94bd4b23694e9b4b_.log (deflated 55%) 2023-11-14T07:06:02.7959114Z adding: test/test-reports/distributed.fsdp.test_fsdp_ignored_modules_1.1_b62e351429f15278_.log (deflated 87%) 2023-11-14T07:06:02.7960602Z adding: test/test-reports/distributed._tensor.test_utils_1.1_263717c7fb34bd28_.log (deflated 59%) 2023-11-14T07:06:02.7962163Z adding: test/test-reports/distributed.checkpoint.test_file_system_checkpoint_1.1_8809e40e39f9984f_.log (deflated 85%) 2023-11-14T07:06:02.7963987Z adding: test/test-reports/distributed.checkpoint.test_file_system_checkpoint_cpu_1.1_232d5753ddc23186_.log (deflated 90%) 2023-11-14T07:06:02.7965619Z adding: test/test-reports/distributed.test_c10d_pypg_1.1_084eafc5ff4ad43f_.log (deflated 90%) 2023-11-14T07:06:02.7967079Z adding: test/test-reports/distributed.fsdp.test_fsdp_freezing_weights_1.1_8fb71272051649e6_.log (deflated 82%) 2023-11-14T07:06:02.7968666Z adding: test/test-reports/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks_1.1_55881abcd43aafde_.log (deflated 74%) 2023-11-14T07:06:02.7970352Z adding: test/test-reports/distributed.elastic.multiprocessing.api_test_1.1_6c84287949be6a60_.log (deflated 89%) 2023-11-14T07:06:02.7971988Z adding: test/test-reports/distributed.checkpoint.test_hsdp_checkpoint_1.1_312759a3411eb574_.log (deflated 70%) 2023-11-14T07:06:02.7973720Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_tensor_ops_1.1_1b541a8f718ca8ed_.log (deflated 70%) 2023-11-14T07:06:02.7975525Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_optim_checkpoint_1.1_d1f070e12e517084_.log (deflated 92%) 2023-11-14T07:06:02.7977302Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_binary_cmp_1.1_7801904f71b0ea6b_.log (deflated 69%) 2023-11-14T07:06:02.7979076Z adding: test/test-reports/distributed._shard.sharding_plan.test_sharding_plan_1.1_3b1e74365e07fbc2_.log (deflated 65%) 2023-11-14T07:06:02.7980765Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_embedding_1.1_65c74ec125113316_.log (deflated 61%) 2023-11-14T07:06:02.7982563Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_util_1.1_8f74ace374ca9490_.log (deflated 57%) 2023-11-14T07:06:02.7984239Z adding: test/test-reports/distributed.fsdp.test_fsdp_multiple_forward_1.1_e9b7c02a30988ec2_.log (deflated 55%) 2023-11-14T07:06:02.7985823Z adding: test/test-reports/distributed.pipeline.sync.test_balance_1.1_64de487d3ad50265_.log (deflated 77%) 2023-11-14T07:06:02.7987395Z adding: test/test-reports/distributed.tensor.parallel.test_ddp_2d_parallel_1.1_6046b83c77e94463_.log (deflated 55%) 2023-11-14T07:06:02.7989062Z adding: test/test-reports/distributed.tensor.parallel.test_tp_random_state_1.1_017637baece7d195_.log (deflated 55%) 2023-11-14T07:06:02.7990763Z adding: test/test-reports/distributed.pipeline.sync.skip.test_gpipe_1.1_6d4275244b56109a_.log (deflated 75%) 2023-11-14T07:06:02.7992441Z adding: test/test-reports/distributed.pipeline.sync.test_checkpoint_1.1_45476bd78be5513e_.log (deflated 72%) 2023-11-14T07:06:02.7993964Z adding: test/test-reports/distributed.elastic.utils.logging_test_1.1_4c9426a6e1d7ccb7_.log (deflated 58%) 2023-11-14T07:06:02.7995778Z adding: test/test-reports/distributed.checkpoint.test_nested_dict_1.1_f947723bd8dc60a1_.log (deflated 59%) 2023-11-14T07:06:02.7997329Z adding: test/test-reports/distributed._tensor.test_xla_integration_1.1_dba532b551f362b7_.log (deflated 66%) 2023-11-14T07:06:02.7998881Z adding: test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_dadb9471d9d02214_.log (deflated 52%) 2023-11-14T07:06:02.8000620Z adding: test/test-reports/distributed.pipeline.sync.test_dependency_1.1_8b89f2ec3d51eaf4_.log (deflated 68%) 2023-11-14T07:06:02.8002252Z adding: test/test-reports/distributed.pipeline.sync.skip.test_api_1.1_c5a77a4eb5146f16_.log (deflated 60%) 2023-11-14T07:06:02.8003771Z adding: test/test-reports/distributed._spmd.test_transformation_1.1_f540389596243d5b_.log (stored 0%) 2023-11-14T07:06:02.8005245Z adding: test/test-reports/distributed.checkpoint.e2e.test_pipeline_1.1_01184b76568bd70e_.log (stored 0%) 2023-11-14T07:06:02.8006754Z adding: test/test-reports/distributed.optim.test_named_optimizer_1.1_e10076f7486eba1c_.log (stored 0%) 2023-11-14T07:06:02.8077814Z ##[group]Run seemethere/upload-artifact-s3@v5 2023-11-14T07:06:02.8078354Z with: 2023-11-14T07:06:02.8078793Z s3-prefix: pytorch/pytorch/6859165800/1/artifact 2023-11-14T07:06:02.8079593Z retention-days: 14 2023-11-14T07:06:02.8080001Z if-no-files-found: warn 2023-11-14T07:06:02.8080461Z path: test-jsons-*.zip 2023-11-14T07:06:02.8080892Z name: artifact 2023-11-14T07:06:02.8081277Z s3-bucket: gha-artifacts 2023-11-14T07:06:02.8081780Z region: us-east-1 2023-11-14T07:06:02.8082158Z env: 2023-11-14T07:06:02.8082482Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:06:02.8083056Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:06:02.8083990Z DOCKER_CONTAINER_ID: 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:02.8084791Z ##[endgroup] 2023-11-14T07:06:03.2910723Z NOTE: s3-prefix specified, ignoring name parameter 2023-11-14T07:06:03.2911559Z With the provided path, there will be 1 file uploaded 2023-11-14T07:06:03.2912355Z Uploading to s3 prefix: pytorch/pytorch/6859165800/1/artifact 2023-11-14T07:06:03.2923672Z Starting upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_18651430132.zip 2023-11-14T07:06:03.4813745Z Finished upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_18651430132.zip 2023-11-14T07:06:03.5006974Z ##[group]Run seemethere/upload-artifact-s3@v5 2023-11-14T07:06:03.5007518Z with: 2023-11-14T07:06:03.5007989Z s3-prefix: pytorch/pytorch/6859165800/1/artifact 2023-11-14T07:06:03.5008592Z retention-days: 14 2023-11-14T07:06:03.5008997Z if-no-files-found: error 2023-11-14T07:06:03.5009466Z path: test-reports-*.zip 2023-11-14T07:06:03.5009911Z name: artifact 2023-11-14T07:06:03.5010287Z s3-bucket: gha-artifacts 2023-11-14T07:06:03.5010733Z region: us-east-1 2023-11-14T07:06:03.5011113Z env: 2023-11-14T07:06:03.5011438Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:06:03.5012014Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:06:03.5012952Z DOCKER_CONTAINER_ID: 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:03.5013749Z ##[endgroup] 2023-11-14T07:06:03.9792452Z NOTE: s3-prefix specified, ignoring name parameter 2023-11-14T07:06:03.9793318Z With the provided path, there will be 1 file uploaded 2023-11-14T07:06:03.9794081Z Uploading to s3 prefix: pytorch/pytorch/6859165800/1/artifact 2023-11-14T07:06:03.9805563Z Starting upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_18651430132.zip 2023-11-14T07:06:04.1744624Z Finished upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_18651430132.zip 2023-11-14T07:06:04.1936391Z ##[group]Run seemethere/upload-artifact-s3@v5 2023-11-14T07:06:04.1936933Z with: 2023-11-14T07:06:04.1937377Z s3-prefix: pytorch/pytorch/6859165800/1/artifact 2023-11-14T07:06:04.1937983Z retention-days: 14 2023-11-14T07:06:04.1938393Z if-no-files-found: ignore 2023-11-14T07:06:04.1938846Z path: logs-*.zip 2023-11-14T07:06:04.1939246Z name: artifact 2023-11-14T07:06:04.1939620Z s3-bucket: gha-artifacts 2023-11-14T07:06:04.1940063Z region: us-east-1 2023-11-14T07:06:04.1940441Z env: 2023-11-14T07:06:04.1940769Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:06:04.1941365Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:06:04.1942310Z DOCKER_CONTAINER_ID: 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:04.1943103Z ##[endgroup] 2023-11-14T07:06:04.6705913Z NOTE: s3-prefix specified, ignoring name parameter 2023-11-14T07:06:04.6706747Z With the provided path, there will be 1 file uploaded 2023-11-14T07:06:04.6707919Z Uploading to s3 prefix: pytorch/pytorch/6859165800/1/artifact 2023-11-14T07:06:04.6718795Z Starting upload of logs-test-distributed-2-3-linux.8xlarge.nvidia.gpu_18651430132.zip 2023-11-14T07:06:04.8815590Z Finished upload of logs-test-distributed-2-3-linux.8xlarge.nvidia.gpu_18651430132.zip 2023-11-14T07:06:04.9004898Z ##[group]Run # shellcheck disable=SC2156 2023-11-14T07:06:04.9005560Z # shellcheck disable=SC2156 2023-11-14T07:06:04.9006608Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2023-11-14T07:06:04.9016437Z shell: /usr/bin/bash -e {0} 2023-11-14T07:06:04.9016894Z env: 2023-11-14T07:06:04.9017259Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:06:04.9017820Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:06:04.9018769Z DOCKER_CONTAINER_ID: 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:04.9019594Z ##[endgroup] 2023-11-14T07:06:05.1943288Z GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2 2023-11-14T07:06:05.1944010Z Copyright (C) 2020 Free Software Foundation, Inc. 2023-11-14T07:06:05.1944880Z License GPLv3+: GNU GPL version 3 or later 2023-11-14T07:06:05.1945986Z This is free software: you are free to change and redistribute it. 2023-11-14T07:06:05.1946800Z There is NO WARRANTY, to the extent permitted by law. 2023-11-14T07:06:05.1947533Z Type "show copying" and "show warranty" for details. 2023-11-14T07:06:05.1948322Z This GDB was configured as "x86_64-linux-gnu". 2023-11-14T07:06:05.1949014Z Type "show configuration" for configuration details. 2023-11-14T07:06:05.1949707Z For bug reporting instructions, please see: 2023-11-14T07:06:05.1950327Z . 2023-11-14T07:06:05.1951066Z Find the GDB manual and other documentation resources online at: 2023-11-14T07:06:05.1951853Z . 2023-11-14T07:06:05.1952335Z 2023-11-14T07:06:05.1952502Z For help, type "help". 2023-11-14T07:06:05.1953124Z Type "apropos word" to search for commands related to "word"... 2023-11-14T07:06:05.2725863Z Reading symbols from python... 2023-11-14T07:06:05.9971722Z 2023-11-14T07:06:05.9972838Z warning: core file may not match specified executable file. 2023-11-14T07:06:06.0268564Z [New LWP 1157] 2023-11-14T07:06:06.0268952Z [New LWP 1161] 2023-11-14T07:06:06.0269537Z [New LWP 1154] 2023-11-14T07:06:06.0270409Z [New LWP 1162] 2023-11-14T07:06:06.0271023Z [New LWP 1165] 2023-11-14T07:06:06.0271382Z [New LWP 1168] 2023-11-14T07:06:06.0271761Z [New LWP 1152] 2023-11-14T07:06:06.0272180Z [New LWP 1169] 2023-11-14T07:06:06.0272530Z [New LWP 1173] 2023-11-14T07:06:06.0272906Z [New LWP 1174] 2023-11-14T07:06:06.0303753Z [Thread debugging using libthread_db enabled] 2023-11-14T07:06:06.0305441Z Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1". 2023-11-14T07:06:13.1061199Z 50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. 2023-11-14T07:06:13.1063084Z warning: File "/var/lib/jenkins/workspace/.gdbinit" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load". 2023-11-14T07:06:13.1064925Z Core was generated by `/opt/conda/envs/py_3.10/bin/python -c from multiprocessing.spawn import spawn_m'. 2023-11-14T07:06:13.1066043Z Program terminated with signal SIGABRT, Aborted. 2023-11-14T07:06:13.1066856Z #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 2023-11-14T07:06:13.1067686Z [Current thread is 1 (Thread 0x7f1f4ca0c700 (LWP 1157))] 2023-11-14T07:06:13.1145208Z To enable execution of this file add 2023-11-14T07:06:13.1146941Z add-auto-load-safe-path /var/lib/jenkins/workspace/.gdbinit 2023-11-14T07:06:13.1147834Z line to your configuration file "/var/lib/jenkins/.gdbinit". 2023-11-14T07:06:13.1148585Z To completely disable this security protection add 2023-11-14T07:06:13.1149273Z set auto-load safe-path / 2023-11-14T07:06:13.1149898Z line to your configuration file "/var/lib/jenkins/.gdbinit". 2023-11-14T07:06:13.1150696Z For more information about this security protection see the 2023-11-14T07:06:13.1151711Z "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: 2023-11-14T07:06:13.1152580Z info "(gdb)Auto-loading safe path" 2023-11-14T07:06:13.1153691Z #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 2023-11-14T07:06:13.1154952Z #1 0x00007f1fe443c859 in __GI_abort () at abort.c:79 2023-11-14T07:06:13.1305011Z #2 0x00007f1fdc8ee35a in __cxxabiv1::__terminate (handler=) 2023-11-14T07:06:13.1306806Z at /opt/conda/conda-bld/gcc-compiler_1654084175708/work/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:48 2023-11-14T07:06:13.1308124Z #3 0x00007f1fdc8ee3c5 in std::terminate () 2023-11-14T07:06:13.1309222Z at /opt/conda/conda-bld/gcc-compiler_1654084175708/work/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:58 2023-11-14T07:06:13.2327290Z #4 0x00007f1fdc8ee34f in std::rethrow_exception (ep=...) 2023-11-14T07:06:13.2329369Z at /opt/conda/conda-bld/gcc-compiler_1654084175708/work/gcc/libstdc++-v3/libsupc++/eh_ptr.cc:217 2023-11-14T07:06:13.2330618Z #5 0x00007f1fc1a1c3b3 in c10d::ProcessGroupNCCL::ncclCommWatchdog() [clone .cold] () 2023-11-14T07:06:13.2331883Z from /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so 2023-11-14T07:06:13.2427146Z #6 0x00007f1fdc918bf4 in std::execute_native_thread_routine (__p=0x71b77f0) 2023-11-14T07:06:13.2429565Z at /opt/conda/conda-bld/gcc-compiler_1654084175708/work/gcc/libstdc++-v3/src/c++11/thread.cc:82 2023-11-14T07:06:13.2430994Z #7 0x00007f1fe476e609 in start_thread (arg=) 2023-11-14T07:06:13.2431670Z at pthread_create.c:477 2023-11-14T07:06:13.2432154Z #8 0x00007f1fe4539133 in clone () 2023-11-14T07:06:13.2432728Z at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 2023-11-14T07:06:13.4369546Z GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2 2023-11-14T07:06:13.4370783Z Copyright (C) 2020 Free Software Foundation, Inc. 2023-11-14T07:06:13.4371688Z License GPLv3+: GNU GPL version 3 or later 2023-11-14T07:06:13.4372662Z This is free software: you are free to change and redistribute it. 2023-11-14T07:06:13.4373524Z There is NO WARRANTY, to the extent permitted by law. 2023-11-14T07:06:13.4374310Z Type "show copying" and "show warranty" for details. 2023-11-14T07:06:13.4375075Z This GDB was configured as "x86_64-linux-gnu". 2023-11-14T07:06:13.4375770Z Type "show configuration" for configuration details. 2023-11-14T07:06:13.4376459Z For bug reporting instructions, please see: 2023-11-14T07:06:13.4377111Z . 2023-11-14T07:06:13.4377828Z Find the GDB manual and other documentation resources online at: 2023-11-14T07:06:13.4378639Z . 2023-11-14T07:06:13.4379097Z 2023-11-14T07:06:13.4379283Z For help, type "help". 2023-11-14T07:06:13.4379871Z Type "apropos word" to search for commands related to "word"... 2023-11-14T07:06:13.5152042Z Reading symbols from python... 2023-11-14T07:06:14.2295211Z 2023-11-14T07:06:14.2296696Z warning: core file may not match specified executable file. 2023-11-14T07:06:14.2589407Z [New LWP 3513] 2023-11-14T07:06:14.2590179Z [New LWP 3518] 2023-11-14T07:06:14.2590929Z [New LWP 3517] 2023-11-14T07:06:14.2591625Z [New LWP 3510] 2023-11-14T07:06:14.2592012Z [New LWP 3524] 2023-11-14T07:06:14.2592363Z [New LWP 3521] 2023-11-14T07:06:14.2592739Z [New LWP 3525] 2023-11-14T07:06:14.2593114Z [New LWP 3508] 2023-11-14T07:06:14.2593471Z [New LWP 3529] 2023-11-14T07:06:14.2593849Z [New LWP 3530] 2023-11-14T07:06:14.2624496Z [Thread debugging using libthread_db enabled] 2023-11-14T07:06:14.2626128Z Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1". 2023-11-14T07:06:21.3985518Z 50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. 2023-11-14T07:06:21.3987144Z Core was generated by `/opt/conda/envs/py_3.10/bin/python -c from multiprocessing.spawn import spawn_m'. 2023-11-14T07:06:21.3988234Z Program terminated with signal SIGABRT, Aborted. 2023-11-14T07:06:21.3989054Z #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 2023-11-14T07:06:21.3989876Z [Current thread is 1 (Thread 0x7fd990da4700 (LWP 3513))] 2023-11-14T07:06:21.3991902Z warning: File "/var/lib/jenkins/workspace/.gdbinit" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load". 2023-11-14T07:06:21.4053138Z To enable execution of this file add 2023-11-14T07:06:21.4054805Z add-auto-load-safe-path /var/lib/jenkins/workspace/.gdbinit 2023-11-14T07:06:21.4056055Z line to your configuration file "/var/lib/jenkins/.gdbinit". 2023-11-14T07:06:21.4056815Z To completely disable this security protection add 2023-11-14T07:06:21.4057505Z set auto-load safe-path / 2023-11-14T07:06:21.4058097Z line to your configuration file "/var/lib/jenkins/.gdbinit". 2023-11-14T07:06:21.4058908Z For more information about this security protection see the 2023-11-14T07:06:21.4059931Z "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: 2023-11-14T07:06:21.4060775Z info "(gdb)Auto-loading safe path" 2023-11-14T07:06:21.4061506Z #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 2023-11-14T07:06:21.4062333Z #1 0x00007fda1c80a859 in __GI_abort () at abort.c:79 2023-11-14T07:06:21.4213800Z #2 0x00007fda14cbc35a in __cxxabiv1::__terminate (handler=) 2023-11-14T07:06:21.4215654Z at /opt/conda/conda-bld/gcc-compiler_1654084175708/work/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:48 2023-11-14T07:06:21.4216668Z #3 0x00007fda14cbc3c5 in std::terminate () 2023-11-14T07:06:21.4217760Z at /opt/conda/conda-bld/gcc-compiler_1654084175708/work/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:58 2023-11-14T07:06:21.5242668Z #4 0x00007fda14cbc34f in std::rethrow_exception (ep=...) 2023-11-14T07:06:21.5244218Z at /opt/conda/conda-bld/gcc-compiler_1654084175708/work/gcc/libstdc++-v3/libsupc++/eh_ptr.cc:217 2023-11-14T07:06:21.5245456Z #5 0x00007fd9f9dea3b3 in c10d::ProcessGroupNCCL::ncclCommWatchdog() [clone .cold] () 2023-11-14T07:06:21.5246686Z from /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so 2023-11-14T07:06:21.5343032Z #6 0x00007fda14ce6bf4 in std::execute_native_thread_routine (__p=0x785ac80) 2023-11-14T07:06:21.5345421Z at /opt/conda/conda-bld/gcc-compiler_1654084175708/work/gcc/libstdc++-v3/src/c++11/thread.cc:82 2023-11-14T07:06:21.5346604Z #7 0x00007fda1cb3c609 in start_thread (arg=) 2023-11-14T07:06:21.5347246Z at pthread_create.c:477 2023-11-14T07:06:21.5347758Z #8 0x00007fda1c907133 in clone () 2023-11-14T07:06:21.5348341Z at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 2023-11-14T07:06:21.7718348Z ##[group]Run seemethere/upload-artifact-s3@v5 2023-11-14T07:06:21.7718989Z with: 2023-11-14T07:06:21.7719498Z name: coredumps-distributed-2-3-linux.8xlarge.nvidia.gpu 2023-11-14T07:06:21.7720318Z retention-days: 14 2023-11-14T07:06:21.7721004Z if-no-files-found: ignore 2023-11-14T07:06:21.7721522Z path: ./**/core.[1-9]* 2023-11-14T07:06:21.7722034Z s3-bucket: gha-artifacts 2023-11-14T07:06:21.7722669Z region: us-east-1 2023-11-14T07:06:21.7723078Z env: 2023-11-14T07:06:21.7723492Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:06:21.7724476Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:06:21.7725523Z DOCKER_CONTAINER_ID: 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:21.7726408Z ##[endgroup] 2023-11-14T07:06:32.9902584Z With the provided path, there will be 2 files uploaded 2023-11-14T07:06:32.9903952Z Uploading to s3 prefix: pytorch/pytorch/6859165800/coredumps-distributed-2-3-linux.8xlarge.nvidia.gpu 2023-11-14T07:06:32.9914537Z Starting upload of test/core.1152 2023-11-14T07:06:43.8856529Z Finished upload of test/core.1152 2023-11-14T07:06:43.8865504Z Starting upload of test/core.3508 2023-11-14T07:06:50.5473657Z Finished upload of test/core.3508 2023-11-14T07:06:50.5739492Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2023-11-14T07:06:50.5740213Z with: 2023-11-14T07:06:50.5740543Z env: 2023-11-14T07:06:50.5740907Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:06:50.5741461Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:06:50.5742394Z DOCKER_CONTAINER_ID: 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:50.5743209Z ##[endgroup] 2023-11-14T07:06:50.5771072Z ##[group]Run set -eou pipefail 2023-11-14T07:06:50.5771604Z set -eou pipefail 2023-11-14T07:06:50.5772055Z  2023-11-14T07:06:50.5772704Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2023-11-14T07:06:50.5773775Z for _ in $(seq 1440); do 2023-11-14T07:06:50.5774373Z  # Break if no ssh session exists anymore 2023-11-14T07:06:50.5775005Z  if [ "$(who)" = "" ]; then 2023-11-14T07:06:50.5775498Z  break 2023-11-14T07:06:50.5775974Z  fi 2023-11-14T07:06:50.5776343Z  echo "." 2023-11-14T07:06:50.5776766Z  sleep 5 2023-11-14T07:06:50.5777172Z done 2023-11-14T07:06:50.5786437Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T07:06:50.5787080Z env: 2023-11-14T07:06:50.5787442Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:06:50.5788003Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:06:50.5788941Z DOCKER_CONTAINER_ID: 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:50.5789770Z ##[endgroup] 2023-11-14T07:06:50.5820532Z Holding runner for 2 hours until all ssh sessions have logged out 2023-11-14T07:06:50.5881178Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2023-11-14T07:06:50.5882159Z # ignore expansion of "docker ps -q" since it could be empty 2023-11-14T07:06:50.5882975Z # shellcheck disable=SC2046 2023-11-14T07:06:50.5883588Z docker stop $(docker ps -q) || true 2023-11-14T07:06:50.5884190Z # Prune all of the docker images 2023-11-14T07:06:50.5884774Z docker system prune -af 2023-11-14T07:06:50.5893377Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T07:06:50.5894016Z env: 2023-11-14T07:06:50.5894384Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:06:50.5894947Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:06:50.5895926Z DOCKER_CONTAINER_ID: 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:50.5896873Z ##[endgroup] 2023-11-14T07:06:50.9626749Z 390ec749fb05 2023-11-14T07:06:51.5157771Z Deleted Containers: 2023-11-14T07:06:51.5159144Z 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:51.5159730Z 2023-11-14T07:06:58.0767906Z Deleted Images: 2023-11-14T07:06:58.0769877Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9:3cbebab0ba3af5b92c112f499347ddfdd0f631c8 2023-11-14T07:06:58.0772917Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn8-py3-gcc9@sha256:155cc70f81c46492d7369e444f8774cfbc3ff3ec6c7d0217deb03d9c42253389 2023-11-14T07:06:58.0775237Z deleted: sha256:2e9cd9f64120afd8fb911e301ea6adc44cad71e8bfde2785095ab22e8b7f1f32 2023-11-14T07:06:58.0777541Z deleted: sha256:512838e73d4f3d028fbbe7052c9d1f8206ef2659c60359ab52b34e4cc35c3992 2023-11-14T07:06:58.0779163Z deleted: sha256:a9d8f9d40a750124a6f8093111d44b5278d65dd1a2f791289f48f8724dbb459e 2023-11-14T07:06:58.0780215Z deleted: sha256:7be8ab7c9adc25b2f1ba0f4c4ca11050fd97f8678c8bbc4074e9532c54e0d7f9 2023-11-14T07:06:58.0781380Z deleted: sha256:83a338be1d118af6b97d3695eb7f03a4a6f3b727fd1515e133df986281bc45aa 2023-11-14T07:06:58.0782393Z deleted: sha256:853b0173ff3fe3deb14968f3ea4a3159098a702b026f1813e4777ab0c1bd1b15 2023-11-14T07:06:58.0784008Z deleted: sha256:146b0656459f08e735bf0326b9c46dcb5add40d800a3f0fa7f753d73fba2bc48 2023-11-14T07:06:58.0786435Z deleted: sha256:c5c9d92208787809f5e8e4e2c199d194a1c6d9cc1e625b1a4c4681be00e25f08 2023-11-14T07:06:58.0787597Z deleted: sha256:506b175b96c4889f411b3d98ba1bcc3a695d189d9f75859b7c53d1d7c6c84c62 2023-11-14T07:06:58.0788726Z deleted: sha256:bd30c7b38cc8f59afc412d106e97b45cfdc9f076e11a734c19b045153eea3dbc 2023-11-14T07:06:58.0790370Z deleted: sha256:f86fe52700a5bdaadba6e76e1d72209470b879c254d5c2f80a931b7de263a640 2023-11-14T07:06:58.0791432Z deleted: sha256:128374ff7ddd2bb50e4dd80e5a8029dc3c6498f66efed461a5db0a38eb830df4 2023-11-14T07:06:58.0792587Z deleted: sha256:4140552c990c771f35f0bfffa628800ad4caffebd171f152bd3f6909a33afefa 2023-11-14T07:06:58.0793568Z deleted: sha256:f0484d0bf7d841b21262a867fd30ff4a6cc1fd974036859e4100ec576c9117d2 2023-11-14T07:06:58.0795513Z deleted: sha256:fb583a2e1b7d9574e950b005fdc068b75673b4a719b41972da5f09c4287903f7 2023-11-14T07:06:58.0796574Z deleted: sha256:a5d7b2a2a66e710c2b300ffd3208b143f26caaaebbe936f70d48d748d888d970 2023-11-14T07:06:58.0797651Z deleted: sha256:814a71ffc7f1e74711aded1614eafb2e2d475994244b16bcf5aee091e8505aa7 2023-11-14T07:06:58.0798738Z deleted: sha256:a595f3bb7d57f50c1c0023fe87259e1fb2f04073eafbe45a4f41df19a137c840 2023-11-14T07:06:58.0799801Z deleted: sha256:f00286388f7c580b213816f52863390af7e56967b92b4e4c8421167eda1e2fd9 2023-11-14T07:06:58.0800817Z deleted: sha256:79652a8cad146a70172780b29b4d868800098f6dfa17da14f851143254a9786e 2023-11-14T07:06:58.0801883Z deleted: sha256:dbb5eb27c0b50d5693a5a24d3b114fa312270d8d08ea0bd7e89f346437979a50 2023-11-14T07:06:58.0802956Z deleted: sha256:1d75aa6c5012f2747eb72b6861274c2f46f89675170aa4c7000b295a487b7bfd 2023-11-14T07:06:58.0803991Z deleted: sha256:c85b89d91368af1bc72c26119c221e7daa8a195b18fde34a88b7c9037208c401 2023-11-14T07:06:58.0805117Z deleted: sha256:04e30a2271f62fb8c83dabac48f382c43155c877ba30fae77dbe53acb902eb82 2023-11-14T07:06:58.0806252Z deleted: sha256:559cad94e52b67a628fb575fe5d0a1835cccc9d5a9f65d73a201003c6e7092cf 2023-11-14T07:06:58.0807542Z deleted: sha256:b4ee4f97dcc06aa686efba9fa49bb8ed5a5f9ee5e555dfd1854c4f617ccdde42 2023-11-14T07:06:58.0808549Z deleted: sha256:9c80008875379b87b4eb54e0a202471743e15020597b6ce1081a47ce3e243657 2023-11-14T07:06:58.0809653Z deleted: sha256:478b4d9242bf51f25afaee8a17668e62354eaf0f037498ef7974207f09dd9fc0 2023-11-14T07:06:58.0811400Z deleted: sha256:fd3dce86797122ae34ab925851b8018328f36db5add7ce0a6c59ec42782a9253 2023-11-14T07:06:58.0813666Z deleted: sha256:07d43fc41ab24cf9a60d85cedc219939a6c0e13e5da54b2b3b1ba04ae335f8a0 2023-11-14T07:06:58.0815205Z deleted: sha256:3b0bf5a23b0af98b94fde5c294d6cdb7179f52d5bfaea66c96fcdef51bc17494 2023-11-14T07:06:58.0816322Z deleted: sha256:7b22f709c4f3b882fe0f7b19c7d2722f1beffe00fdc21601d8725b2ec758cb2d 2023-11-14T07:06:58.0817443Z deleted: sha256:0398ace2cfc2753c45b24e112fd3e2ed823c2a3cbf69bf70c7ea7bf5d90a4d5d 2023-11-14T07:06:58.0818506Z deleted: sha256:91a560086e3b6ef014ab3c6b1f20bfa2a33267f4e338a54566c717c11542966b 2023-11-14T07:06:58.0819604Z deleted: sha256:c8300e5affc1abe3fde52a5fbdb868af96a8c8ac4293ea5fa0861dea94b086bf 2023-11-14T07:06:58.0820707Z deleted: sha256:6fe25eaec572795438d50d127b23a949366874beec0c20e82b69f6909b09a3a4 2023-11-14T07:06:58.0821763Z deleted: sha256:b676deccd8574b11ab6baeabc60868f51d5745f73a5df8293ae6caccd82d5757 2023-11-14T07:06:58.0822843Z deleted: sha256:7f0fa62b771f33c2743916a00800c0b7ef42397b25107e6bfd5d2ae7d05d3661 2023-11-14T07:06:58.0823881Z deleted: sha256:97b766680dda9399a9703808320b8142ab92653b8a66b7356a81818e031b1c31 2023-11-14T07:06:58.0824967Z deleted: sha256:a63c13b07c37c9db4762fb6ded0a310801775ab21694d7357f5adc547589f152 2023-11-14T07:06:58.0826016Z deleted: sha256:716f2784f05e703b3e8e997219994fff81f33da47346da6fe92b3b5dd52afe90 2023-11-14T07:06:58.0827071Z deleted: sha256:218d7338e785bbec77a628e85f5c0042d627ba7ef2785407740f960744e9c031 2023-11-14T07:06:58.0828126Z deleted: sha256:fc92863fa41959f0178a6cc629583b23677afcf0179eb394ad4301345ad86b7a 2023-11-14T07:06:58.0829150Z deleted: sha256:28efec4552900781978c637546ebf0199d26c1e56ee4e17fe392059c932ddac2 2023-11-14T07:06:58.0830455Z deleted: sha256:3bdcccc186266e4aeb82fe136daf065d3edee2ad2d0f49e29aa0bf2aa0d8f343 2023-11-14T07:06:58.0831606Z deleted: sha256:54e5f78fa13bc8fabfeb36f48b88e7a84380a00d12ccbc184a16513a115076a9 2023-11-14T07:06:58.0832644Z deleted: sha256:59e468e5df4e579b108d9298d26dca1701012533337f3448f04156a24c6cd076 2023-11-14T07:06:58.0833740Z deleted: sha256:c8debf67eb97b09bc5582acd540435d55afbcfe3e074f5a3c117e37db387de29 2023-11-14T07:06:58.0835332Z deleted: sha256:4ef0affb5a7229892f3928318556d8d9b688f567daf1da4225b0e0950cba8fcb 2023-11-14T07:06:58.0836416Z deleted: sha256:902999e13a0732020040c9fec1b3d5f731bf15c96b759b3a81053afc0f421840 2023-11-14T07:06:58.0837444Z deleted: sha256:70467a5ca7e57d18e5748ae93aa7050fe8860d0da0a558b21d3deafcc2d0b636 2023-11-14T07:06:58.0838704Z deleted: sha256:90479113476838afd72a83b5b42cdf6e75574f00af3f820fe34977b5f3fd2548 2023-11-14T07:06:58.0839767Z deleted: sha256:ab01c4657ada85c5bff93428e0493e9bffb26fe7604cf1c84919ff2c0289029f 2023-11-14T07:06:58.0840848Z deleted: sha256:864065ffcddbc3e8d476f5bf00831656acefbdf4e0e7ddfa5deddbcea116ab74 2023-11-14T07:06:58.0841934Z deleted: sha256:5f277e57fd48ab88775573de37d122ca48dae81f4e8d988ca27b876523f76598 2023-11-14T07:06:58.0842983Z deleted: sha256:606f9756a15c988a2bdda76a0d691866d94632ad66845486b599d33ebf3f1960 2023-11-14T07:06:58.0844019Z deleted: sha256:1a185cc315e63cd72c51b03ef6d959c8f6ad0e31b165fe28deefb04e81e23fbf 2023-11-14T07:06:58.0845100Z deleted: sha256:11106898dfd6aa5a82a26bb1adfa3b96cfd17b46004e6f798408b8a4ff90a20e 2023-11-14T07:06:58.0846170Z deleted: sha256:d5b52f1c2127024e85dec1b5313c886e9856efaf4beb5d1625861b0a45417784 2023-11-14T07:06:58.0847234Z deleted: sha256:350f24064adcc53c861785ed453c62fa78c810eb6272144307367af2237523f8 2023-11-14T07:06:58.0848300Z deleted: sha256:d5e1eb7e991eb872cbeda851811d29f073f51604aa7aeaeaeb11a404be874b83 2023-11-14T07:06:58.0849378Z deleted: sha256:a0f524642395fd3247e6fe0f53b91a520b1a6b132521ed9ab5d7481a867707e6 2023-11-14T07:06:58.0850428Z deleted: sha256:b7d7a9df07683914a58a4a4415b7c341536391af96904c6ae7f5c1f31dc1ee7e 2023-11-14T07:06:58.0851462Z deleted: sha256:37deeab68757221d4740d97062f42d85962ba2a179e895b0234d73681c140219 2023-11-14T07:06:58.0852510Z deleted: sha256:347b7d34ff056fa73466bd5774158bd466e7392c2b95c5cb6ed383c0344296f3 2023-11-14T07:06:58.0853582Z deleted: sha256:ec66d8cea54a2f4dfbbd8342ce082503bf8541e996a800c0d724b8dd2fea7f6a 2023-11-14T07:06:58.0854234Z 2023-11-14T07:06:58.0854412Z Total reclaimed space: 25.61GB 2023-11-14T07:06:58.0903077Z ##[group]Run set +e 2023-11-14T07:06:58.0903675Z set +e 2023-11-14T07:06:58.0904042Z set -x 2023-11-14T07:06:58.0904427Z  2023-11-14T07:06:58.0904799Z nvidia-smi 2023-11-14T07:06:58.0905592Z # NB: Surprisingly, nvidia-smi command returns successfully with return code 0 even in 2023-11-14T07:06:58.0906876Z # the case where the driver has already crashed as it still can get the driver version 2023-11-14T07:06:58.0908119Z # and some basic information like the bus ID. However, the rest of the information 2023-11-14T07:06:58.0909063Z # would be missing (ERR!), for example: 2023-11-14T07:06:58.0909610Z # 2023-11-14T07:06:58.0910156Z # +-----------------------------------------------------------------------------+ 2023-11-14T07:06:58.0911139Z # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | 2023-11-14T07:06:58.0912154Z # |-------------------------------+----------------------+----------------------+ 2023-11-14T07:06:58.0913105Z # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | 2023-11-14T07:06:58.0914200Z # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | 2023-11-14T07:06:58.0915740Z # | | | MIG M. | 2023-11-14T07:06:58.0916504Z # |===============================+======================+======================| 2023-11-14T07:06:58.0917306Z # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | 2023-11-14T07:06:58.0918261Z # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | 2023-11-14T07:06:58.0919147Z # | | | ERR! | 2023-11-14T07:06:58.0919960Z # +-------------------------------+----------------------+----------------------+ 2023-11-14T07:06:58.0920601Z # 2023-11-14T07:06:58.0921129Z # +-----------------------------------------------------------------------------+ 2023-11-14T07:06:58.0921979Z # | Processes: | 2023-11-14T07:06:58.0923112Z # | GPU GI CI PID Type Process name GPU Memory | 2023-11-14T07:06:58.0924015Z # | ID ID Usage | 2023-11-14T07:06:58.0924790Z # |=============================================================================| 2023-11-14T07:06:58.0925583Z # +-----------------------------------------------------------------------------+ 2023-11-14T07:06:58.0926219Z # 2023-11-14T07:06:58.0926922Z # This should be reported as a failure instead as it will guarantee to fail when 2023-11-14T07:06:58.0927842Z # Docker tries to run with --gpus all 2023-11-14T07:06:58.0928380Z # 2023-11-14T07:06:58.0929066Z # So, the correct check here is to query one of the missing piece of info like 2023-11-14T07:06:58.0930050Z # GPU name, so that the command can fail accordingly 2023-11-14T07:06:58.0930944Z nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2023-11-14T07:06:58.0931654Z NVIDIA_SMI_STATUS=$? 2023-11-14T07:06:58.0932121Z  2023-11-14T07:06:58.0932902Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2023-11-14T07:06:58.0934051Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2023-11-14T07:06:58.0935108Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2023-11-14T07:06:58.0936011Z  .github/scripts/stop_runner_service.sh 2023-11-14T07:06:58.0936573Z fi 2023-11-14T07:06:58.0936931Z  2023-11-14T07:06:58.0937822Z # For runner with multiple GPUs, we also want to confirm that the number of GPUs are the 2023-11-14T07:06:58.0939078Z # power of 2, i.e. 1, 2, 4, or 8. This is to avoid flaky test issue when one GPU fails 2023-11-14T07:06:58.0940066Z # https://github.com/pytorch/test-infra/issues/4000 2023-11-14T07:06:58.0940849Z GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) 2023-11-14T07:06:58.0941468Z NVIDIA_SMI_STATUS=$? 2023-11-14T07:06:58.0941910Z  2023-11-14T07:06:58.0942695Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2023-11-14T07:06:58.0943857Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2023-11-14T07:06:58.0944891Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2023-11-14T07:06:58.0945803Z  .github/scripts/stop_runner_service.sh 2023-11-14T07:06:58.0946387Z fi 2023-11-14T07:06:58.0946725Z  2023-11-14T07:06:58.0947184Z # Check the GPU count to be a power of 2 2023-11-14T07:06:58.0948258Z if [ "$GPU_COUNT" -le 8 ] && [ "$GPU_COUNT" -ne 1 ] && [ "$GPU_COUNT" -ne 2 ] && [ "$GPU_COUNT" -ne 4 ] && [ "$GPU_COUNT" -ne 8 ]; then 2023-11-14T07:06:58.0949681Z  echo "NVIDIA driver detects $GPU_COUNT GPUs. The runner has a broken GPU, shutting it down..." 2023-11-14T07:06:58.0950667Z  .github/scripts/stop_runner_service.sh 2023-11-14T07:06:58.0951246Z fi 2023-11-14T07:06:58.0960817Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2023-11-14T07:06:58.0961462Z env: 2023-11-14T07:06:58.0961799Z GIT_DEFAULT_BRANCH: main 2023-11-14T07:06:58.0962380Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2023-11-14T07:06:58.0963336Z DOCKER_CONTAINER_ID: 390ec749fb05b8f360493e0bc55b038535a50db59a1fe040f1f3cb1ebd4fc9b8 2023-11-14T07:06:58.0964313Z RUNNER_WORKSPACE: /home/ec2-user/actions-runner/_work/pytorch 2023-11-14T07:06:58.0964987Z ##[endgroup] 2023-11-14T07:06:58.0994475Z + nvidia-smi 2023-11-14T07:06:59.7050564Z Tue Nov 14 07:06:59 2023 2023-11-14T07:06:59.7052016Z +---------------------------------------------------------------------------------------+ 2023-11-14T07:06:59.7053516Z | NVIDIA-SMI 535.54.03 Driver Version: 535.54.03 CUDA Version: 12.2 | 2023-11-14T07:06:59.7054528Z |-----------------------------------------+----------------------+----------------------+ 2023-11-14T07:06:59.7055547Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2023-11-14T07:06:59.7056651Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2023-11-14T07:06:59.7057547Z | | | MIG M. | 2023-11-14T07:06:59.7058254Z |=========================================+======================+======================| 2023-11-14T07:06:59.7098109Z | 0 Tesla M60 Off | 00000000:00:1D.0 Off | 0 | 2023-11-14T07:06:59.7099547Z | N/A 31C P0 37W / 150W | 0MiB / 7680MiB | 98% Default | 2023-11-14T07:06:59.7100369Z | | | N/A | 2023-11-14T07:06:59.7101254Z +-----------------------------------------+----------------------+----------------------+ 2023-11-14T07:06:59.7146224Z | 1 Tesla M60 Off | 00000000:00:1E.0 Off | 0 | 2023-11-14T07:06:59.7147835Z | N/A 25C P0 38W / 150W | 0MiB / 7680MiB | 3% Default | 2023-11-14T07:06:59.7148649Z | | | N/A | 2023-11-14T07:06:59.7149521Z +-----------------------------------------+----------------------+----------------------+ 2023-11-14T07:06:59.7150358Z 2023-11-14T07:06:59.7152455Z +---------------------------------------------------------------------------------------+ 2023-11-14T07:06:59.7153518Z | Processes: | 2023-11-14T07:06:59.7154376Z | GPU GI CI PID Type Process name GPU Memory | 2023-11-14T07:06:59.7155767Z | ID ID Usage | 2023-11-14T07:06:59.7156509Z |=======================================================================================| 2023-11-14T07:06:59.7157405Z | No running processes found | 2023-11-14T07:06:59.7158343Z +---------------------------------------------------------------------------------------+ 2023-11-14T07:07:00.6486514Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2023-11-14T07:07:02.2446918Z Tesla M60 2023-11-14T07:07:03.1317496Z + NVIDIA_SMI_STATUS=0 2023-11-14T07:07:03.1318729Z + '[' 0 -ne 0 ']' 2023-11-14T07:07:03.1324440Z ++ nvidia-smi --list-gpus 2023-11-14T07:07:03.1325430Z ++ wc -l 2023-11-14T07:07:05.9212840Z + GPU_COUNT=2 2023-11-14T07:07:05.9213749Z + NVIDIA_SMI_STATUS=0 2023-11-14T07:07:05.9214790Z + '[' 0 -ne 0 ']' 2023-11-14T07:07:05.9215269Z + '[' 2 -le 8 ']' 2023-11-14T07:07:05.9215727Z + '[' 2 -ne 1 ']' 2023-11-14T07:07:05.9216115Z + '[' 2 -ne 2 ']' 2023-11-14T07:07:05.9295287Z Post job cleanup. 2023-11-14T07:07:05.9364006Z Post job cleanup. 2023-11-14T07:07:06.0560041Z [command]/usr/bin/git version 2023-11-14T07:07:06.0616170Z git version 2.40.1 2023-11-14T07:07:06.0681665Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/7de95a9b-901a-49f7-bfde-d3f4dfe48c62' before making global git config changes 2023-11-14T07:07:06.0683792Z Adding repository directory to the temporary git global config as a safe directory 2023-11-14T07:07:06.0689238Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2023-11-14T07:07:06.0735391Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2023-11-14T07:07:06.0774591Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2023-11-14T07:07:06.1112013Z Entering 'android/libs/fbjni' 2023-11-14T07:07:06.1169591Z Entering 'third_party/FP16' 2023-11-14T07:07:06.1226314Z Entering 'third_party/FXdiv' 2023-11-14T07:07:06.1281807Z Entering 'third_party/NNPACK' 2023-11-14T07:07:06.1337128Z Entering 'third_party/QNNPACK' 2023-11-14T07:07:06.1394342Z Entering 'third_party/VulkanMemoryAllocator' 2023-11-14T07:07:06.1450057Z Entering 'third_party/XNNPACK' 2023-11-14T07:07:06.1523213Z Entering 'third_party/benchmark' 2023-11-14T07:07:06.1578402Z Entering 'third_party/cpuinfo' 2023-11-14T07:07:06.1636667Z Entering 'third_party/cub' 2023-11-14T07:07:06.1692090Z Entering 'third_party/cudnn_frontend' 2023-11-14T07:07:06.1753752Z Entering 'third_party/cutlass' 2023-11-14T07:07:06.1819200Z Entering 'third_party/eigen' 2023-11-14T07:07:06.1878368Z Entering 'third_party/fbgemm' 2023-11-14T07:07:06.1936105Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-11-14T07:07:06.1991596Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-11-14T07:07:06.2047359Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-11-14T07:07:06.2111199Z Entering 'third_party/fbgemm/third_party/googletest' 2023-11-14T07:07:06.2167107Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-11-14T07:07:06.2222571Z Entering 'third_party/flatbuffers' 2023-11-14T07:07:06.2281569Z Entering 'third_party/fmt' 2023-11-14T07:07:06.2339355Z Entering 'third_party/foxi' 2023-11-14T07:07:06.2394069Z Entering 'third_party/gemmlowp/gemmlowp' 2023-11-14T07:07:06.2452535Z Entering 'third_party/gloo' 2023-11-14T07:07:06.2509652Z Entering 'third_party/googletest' 2023-11-14T07:07:06.2567988Z Entering 'third_party/ideep' 2023-11-14T07:07:06.2622561Z Entering 'third_party/ideep/mkl-dnn' 2023-11-14T07:07:06.2685867Z Entering 'third_party/ios-cmake' 2023-11-14T07:07:06.2741529Z Entering 'third_party/ittapi' 2023-11-14T07:07:06.2798992Z Entering 'third_party/kineto' 2023-11-14T07:07:06.2856989Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-11-14T07:07:06.2913804Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-11-14T07:07:06.2974111Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-11-14T07:07:06.3031060Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-11-14T07:07:06.3088136Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-11-14T07:07:06.3142829Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-11-14T07:07:06.3202913Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-11-14T07:07:06.3259411Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-11-14T07:07:06.3315824Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-11-14T07:07:06.3373605Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-11-14T07:07:06.3431205Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-11-14T07:07:06.3488944Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-11-14T07:07:06.3546721Z Entering 'third_party/mimalloc' 2023-11-14T07:07:06.3603641Z Entering 'third_party/nccl/nccl' 2023-11-14T07:07:06.3660625Z Entering 'third_party/neon2sse' 2023-11-14T07:07:06.3715855Z Entering 'third_party/nlohmann' 2023-11-14T07:07:06.3773509Z Entering 'third_party/onnx' 2023-11-14T07:07:06.3851849Z Entering 'third_party/onnx/third_party/benchmark' 2023-11-14T07:07:06.3908462Z Entering 'third_party/onnx/third_party/pybind11' 2023-11-14T07:07:06.3969388Z Entering 'third_party/onnx-tensorrt' 2023-11-14T07:07:06.4025954Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-11-14T07:07:06.4087184Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-11-14T07:07:06.4142679Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-11-14T07:07:06.4199546Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-11-14T07:07:06.4259987Z Entering 'third_party/pocketfft' 2023-11-14T07:07:06.4317291Z Entering 'third_party/protobuf' 2023-11-14T07:07:06.4377533Z Entering 'third_party/protobuf/third_party/benchmark' 2023-11-14T07:07:06.4432921Z Entering 'third_party/protobuf/third_party/googletest' 2023-11-14T07:07:06.4490084Z Entering 'third_party/psimd' 2023-11-14T07:07:06.4545513Z Entering 'third_party/pthreadpool' 2023-11-14T07:07:06.4602327Z Entering 'third_party/pybind11' 2023-11-14T07:07:06.4657637Z Entering 'third_party/python-peachpy' 2023-11-14T07:07:06.4712659Z Entering 'third_party/sleef' 2023-11-14T07:07:06.4769674Z Entering 'third_party/tbb' 2023-11-14T07:07:06.4827765Z Entering 'third_party/tensorpipe' 2023-11-14T07:07:06.4882962Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-11-14T07:07:06.4937949Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-11-14T07:07:06.4992268Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-11-14T07:07:06.5047220Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-11-14T07:07:06.5101793Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-11-14T07:07:06.5159449Z Entering 'third_party/zstd' 2023-11-14T07:07:06.5238949Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2023-11-14T07:07:06.5267038Z http.https://github.com/.extraheader 2023-11-14T07:07:06.5280137Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2023-11-14T07:07:06.5320555Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2023-11-14T07:07:06.5643487Z Entering 'android/libs/fbjni' 2023-11-14T07:07:06.5680341Z http.https://github.com/.extraheader 2023-11-14T07:07:06.5716470Z Entering 'third_party/FP16' 2023-11-14T07:07:06.5753231Z http.https://github.com/.extraheader 2023-11-14T07:07:06.5789129Z Entering 'third_party/FXdiv' 2023-11-14T07:07:06.5825892Z http.https://github.com/.extraheader 2023-11-14T07:07:06.5861476Z Entering 'third_party/NNPACK' 2023-11-14T07:07:06.5898392Z http.https://github.com/.extraheader 2023-11-14T07:07:06.5933801Z Entering 'third_party/QNNPACK' 2023-11-14T07:07:06.5970060Z http.https://github.com/.extraheader 2023-11-14T07:07:06.6005704Z Entering 'third_party/VulkanMemoryAllocator' 2023-11-14T07:07:06.6042126Z http.https://github.com/.extraheader 2023-11-14T07:07:06.6078057Z Entering 'third_party/XNNPACK' 2023-11-14T07:07:06.6114078Z http.https://github.com/.extraheader 2023-11-14T07:07:06.6164407Z Entering 'third_party/benchmark' 2023-11-14T07:07:06.6201372Z http.https://github.com/.extraheader 2023-11-14T07:07:06.6237196Z Entering 'third_party/cpuinfo' 2023-11-14T07:07:06.6273261Z http.https://github.com/.extraheader 2023-11-14T07:07:06.6308488Z Entering 'third_party/cub' 2023-11-14T07:07:06.6346285Z http.https://github.com/.extraheader 2023-11-14T07:07:06.6382240Z Entering 'third_party/cudnn_frontend' 2023-11-14T07:07:06.6418687Z http.https://github.com/.extraheader 2023-11-14T07:07:06.6460950Z Entering 'third_party/cutlass' 2023-11-14T07:07:06.6497081Z http.https://github.com/.extraheader 2023-11-14T07:07:06.6540243Z Entering 'third_party/eigen' 2023-11-14T07:07:06.6576671Z http.https://github.com/.extraheader 2023-11-14T07:07:06.6614076Z Entering 'third_party/fbgemm' 2023-11-14T07:07:06.6651387Z http.https://github.com/.extraheader 2023-11-14T07:07:06.6687817Z Entering 'third_party/fbgemm/third_party/asmjit' 2023-11-14T07:07:06.6725190Z http.https://github.com/.extraheader 2023-11-14T07:07:06.6760481Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2023-11-14T07:07:06.6796745Z http.https://github.com/.extraheader 2023-11-14T07:07:06.6831989Z Entering 'third_party/fbgemm/third_party/cutlass' 2023-11-14T07:07:06.6869774Z http.https://github.com/.extraheader 2023-11-14T07:07:06.6913117Z Entering 'third_party/fbgemm/third_party/googletest' 2023-11-14T07:07:06.6948888Z http.https://github.com/.extraheader 2023-11-14T07:07:06.6984302Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2023-11-14T07:07:06.7019913Z http.https://github.com/.extraheader 2023-11-14T07:07:06.7056502Z Entering 'third_party/flatbuffers' 2023-11-14T07:07:06.7093491Z http.https://github.com/.extraheader 2023-11-14T07:07:06.7130847Z Entering 'third_party/fmt' 2023-11-14T07:07:06.7167714Z http.https://github.com/.extraheader 2023-11-14T07:07:06.7203000Z Entering 'third_party/foxi' 2023-11-14T07:07:06.7240131Z http.https://github.com/.extraheader 2023-11-14T07:07:06.7274444Z Entering 'third_party/gemmlowp/gemmlowp' 2023-11-14T07:07:06.7311778Z http.https://github.com/.extraheader 2023-11-14T07:07:06.7347185Z Entering 'third_party/gloo' 2023-11-14T07:07:06.7384297Z http.https://github.com/.extraheader 2023-11-14T07:07:06.7420179Z Entering 'third_party/googletest' 2023-11-14T07:07:06.7458305Z http.https://github.com/.extraheader 2023-11-14T07:07:06.7493704Z Entering 'third_party/ideep' 2023-11-14T07:07:06.7530787Z http.https://github.com/.extraheader 2023-11-14T07:07:06.7565399Z Entering 'third_party/ideep/mkl-dnn' 2023-11-14T07:07:06.7601996Z http.https://github.com/.extraheader 2023-11-14T07:07:06.7644191Z Entering 'third_party/ios-cmake' 2023-11-14T07:07:06.7680890Z http.https://github.com/.extraheader 2023-11-14T07:07:06.7715353Z Entering 'third_party/ittapi' 2023-11-14T07:07:06.7751819Z http.https://github.com/.extraheader 2023-11-14T07:07:06.7788078Z Entering 'third_party/kineto' 2023-11-14T07:07:06.7824764Z http.https://github.com/.extraheader 2023-11-14T07:07:06.7860349Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2023-11-14T07:07:06.7897026Z http.https://github.com/.extraheader 2023-11-14T07:07:06.7935046Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2023-11-14T07:07:06.7971292Z http.https://github.com/.extraheader 2023-11-14T07:07:06.8009229Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2023-11-14T07:07:06.8045429Z http.https://github.com/.extraheader 2023-11-14T07:07:06.8082607Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2023-11-14T07:07:06.8118724Z http.https://github.com/.extraheader 2023-11-14T07:07:06.8154459Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2023-11-14T07:07:06.8191581Z http.https://github.com/.extraheader 2023-11-14T07:07:06.8226992Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2023-11-14T07:07:06.8263286Z http.https://github.com/.extraheader 2023-11-14T07:07:06.8301197Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2023-11-14T07:07:06.8337754Z http.https://github.com/.extraheader 2023-11-14T07:07:06.8373949Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2023-11-14T07:07:06.8409653Z http.https://github.com/.extraheader 2023-11-14T07:07:06.8446451Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2023-11-14T07:07:06.8482407Z http.https://github.com/.extraheader 2023-11-14T07:07:06.8519625Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2023-11-14T07:07:06.8554780Z http.https://github.com/.extraheader 2023-11-14T07:07:06.8592436Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2023-11-14T07:07:06.8628889Z http.https://github.com/.extraheader 2023-11-14T07:07:06.8664357Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2023-11-14T07:07:06.8700906Z http.https://github.com/.extraheader 2023-11-14T07:07:06.8737762Z Entering 'third_party/mimalloc' 2023-11-14T07:07:06.8775370Z http.https://github.com/.extraheader 2023-11-14T07:07:06.8810918Z Entering 'third_party/nccl/nccl' 2023-11-14T07:07:06.8848120Z http.https://github.com/.extraheader 2023-11-14T07:07:06.8884377Z Entering 'third_party/neon2sse' 2023-11-14T07:07:06.8922216Z http.https://github.com/.extraheader 2023-11-14T07:07:06.8956878Z Entering 'third_party/nlohmann' 2023-11-14T07:07:06.8993427Z http.https://github.com/.extraheader 2023-11-14T07:07:06.9030957Z Entering 'third_party/onnx' 2023-11-14T07:07:06.9068566Z http.https://github.com/.extraheader 2023-11-14T07:07:06.9124894Z Entering 'third_party/onnx/third_party/benchmark' 2023-11-14T07:07:06.9163225Z http.https://github.com/.extraheader 2023-11-14T07:07:06.9199112Z Entering 'third_party/onnx/third_party/pybind11' 2023-11-14T07:07:06.9235307Z http.https://github.com/.extraheader 2023-11-14T07:07:06.9273725Z Entering 'third_party/onnx-tensorrt' 2023-11-14T07:07:06.9310396Z http.https://github.com/.extraheader 2023-11-14T07:07:06.9345083Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2023-11-14T07:07:06.9382470Z http.https://github.com/.extraheader 2023-11-14T07:07:06.9424361Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2023-11-14T07:07:06.9461900Z http.https://github.com/.extraheader 2023-11-14T07:07:06.9499086Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2023-11-14T07:07:06.9535955Z http.https://github.com/.extraheader 2023-11-14T07:07:06.9571852Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2023-11-14T07:07:06.9608189Z http.https://github.com/.extraheader 2023-11-14T07:07:06.9650361Z Entering 'third_party/pocketfft' 2023-11-14T07:07:06.9686879Z http.https://github.com/.extraheader 2023-11-14T07:07:06.9722457Z Entering 'third_party/protobuf' 2023-11-14T07:07:06.9760613Z http.https://github.com/.extraheader 2023-11-14T07:07:06.9800691Z Entering 'third_party/protobuf/third_party/benchmark' 2023-11-14T07:07:06.9837292Z http.https://github.com/.extraheader 2023-11-14T07:07:06.9871590Z Entering 'third_party/protobuf/third_party/googletest' 2023-11-14T07:07:06.9908528Z http.https://github.com/.extraheader 2023-11-14T07:07:06.9945637Z Entering 'third_party/psimd' 2023-11-14T07:07:06.9982862Z http.https://github.com/.extraheader 2023-11-14T07:07:07.0018060Z Entering 'third_party/pthreadpool' 2023-11-14T07:07:07.0055415Z http.https://github.com/.extraheader 2023-11-14T07:07:07.0090737Z Entering 'third_party/pybind11' 2023-11-14T07:07:07.0127431Z http.https://github.com/.extraheader 2023-11-14T07:07:07.0163030Z Entering 'third_party/python-peachpy' 2023-11-14T07:07:07.0200340Z http.https://github.com/.extraheader 2023-11-14T07:07:07.0234820Z Entering 'third_party/sleef' 2023-11-14T07:07:07.0271960Z http.https://github.com/.extraheader 2023-11-14T07:07:07.0307585Z Entering 'third_party/tbb' 2023-11-14T07:07:07.0344555Z http.https://github.com/.extraheader 2023-11-14T07:07:07.0384053Z Entering 'third_party/tensorpipe' 2023-11-14T07:07:07.0421144Z http.https://github.com/.extraheader 2023-11-14T07:07:07.0456801Z Entering 'third_party/tensorpipe/third_party/googletest' 2023-11-14T07:07:07.0494576Z http.https://github.com/.extraheader 2023-11-14T07:07:07.0530499Z Entering 'third_party/tensorpipe/third_party/libnop' 2023-11-14T07:07:07.0566656Z http.https://github.com/.extraheader 2023-11-14T07:07:07.0602624Z Entering 'third_party/tensorpipe/third_party/libuv' 2023-11-14T07:07:07.0639174Z http.https://github.com/.extraheader 2023-11-14T07:07:07.0674920Z Entering 'third_party/tensorpipe/third_party/pybind11' 2023-11-14T07:07:07.0711075Z http.https://github.com/.extraheader 2023-11-14T07:07:07.0746905Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2023-11-14T07:07:07.0784057Z http.https://github.com/.extraheader 2023-11-14T07:07:07.0822248Z Entering 'third_party/zstd' 2023-11-14T07:07:07.0858630Z http.https://github.com/.extraheader 2023-11-14T07:07:07.1237824Z Cleaning up orphan processes